We are seeking a DevOps Engineer to set up, configure, and operationalize a new Databricks environment with a primary focus on business intelligence (BI), analytics, and data engineering workflows.
Working closely with our ML Ops Engineers, you will ensure Databricks is fully prepared for both traditional BI/data processing use cases and AI workloads - including secure access for data analysts, seamless integration with downstream AI and BI tools, and optimized data pipelines.
Key Responsibilities
Environment Setup & Configuration
Deploy and configure Databricks workspace(s) for multi-team usage.
Manage shared clusters, automated job clusters, and interactive clusters.
Configure role-based permissions aligned with governance policies.
Data Integration Enablement
Establish secure connections to on-prem and cloud data sources (SQL Server, Data Lake, APIs).
Build shared ingestion pipelines for BI and analytics teams.
Automate daily/weekly data refresh schedules.
Connectivity for BI Tools
Integrate Databricks with BI platforms (e.g., Power BI).
Optimize query connectors and JDBC/ODBC configurations.
Operational Excellence
Implement monitoring and logging for jobs and pipelines.
Define backup and disaster recovery processes.
Apply cost tracking and optimization practices for cluster usage.
Automation & CI/CD
Set up CI/CD pipelines for data engineering code.
Manage deployment workflows for notebooks, SQL queries, and data models.
Collaboration
Partner with ML Ops Engineers to align infrastructure for ML and BI use cases.
Work with Data Engineers to maintain central data sources.
Collaborate with security teams to enforce access controls.
Governance
Enforce GDPR and internal compliance rules.
Maintain workspace auditing and logging.
Document environment setup and operational procedures.