Job Description:
- We are looking for a skilled Operations Engineer to support our Azure-based Data Engineering platform powered by Databricks.
- This role blends DevOps practices, system reliability, and security auditing to ensure our data infrastructure is robust, secure, and scalable.
- You’ll be instrumental in automating workflows, monitoring system health, and enforcing security best practices across our data ecosystem.
Key Responsibilities:
- Maintain and monitor Databricks-based data pipelines and Azure data services (e.g., Data Lake, Synapse, Key Vault).
- Implement CI/CD pipelines using Azure DevOps for data engineering workflows
- Conduct regular system health checks and performance tuning of clusters and jobs.
- Automate infrastructure provisioning using tools like Terraform or ARM templates.
- Use monitoring tools (e.g., Azure Monitor, Log Analytics, Prometheus) to track system metrics and alert on anomalies.
- Perform security audits using tools like Databricks Security Analysis Tool (SAT) to ensure compliance with best practices.
- Manage secrets and credentials securely using Azure Key Vault and Databricks secret scopes.
- Collaborate with security teams to enforce network security, access control, and data encryption policies.
- Maintain operational runbooks and incident response documentation.
Qualifications:
Bachelor’s degree in computer science, Engineering, or related field.
- • 5+ years of experience in DevOps, cloud operations, or data platform support.
- • Proficiency in Azure services, Databricks, and scripting languages (Python, Bash).
- • Experience with CI/CD tools (Azure DevOps, GitHub Actions) and infrastructure-as-code.
- • Familiarity with security frameworks and auditing tools in cloud environments.
- • Strong troubleshooting skills and a proactive approach to system reliability. Preferred Skills:
- • Experience with Databricks SAT for security posture assessments.
- • Knowledge of Unity Catalog, RBAC, and data governance in Azure Databricks.
- • Exposure to MLOps and operationalizing machine learning pipelines.
- • Understanding of cost optimization strategies in Azure.