We are looking for an experienced Senior Azure Databricks Engineer with strong hands-on expertise in Python, SQL, and Apache Spark to design, build, and optimize scalable data pipelines and analytics solutions on the Azure cloud platform. The ideal candidate should have experience working with large datasets, distributed data processing, analytics use cases, and modern data engineering practices.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks
- Implement ETL/ELT workflows using PySpark, Spark SQL, and Python
- Implement pipelines for data ingestion using Azure Data Factory
- Optimize Spark jobs for performance, cost, and scalability
- Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
- Build and manage Delta Lake tables (ACID, time travel, schema evolution)
- Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)
- Develop complex queries and transformations using SQL
- Collaborate with Data Science teams to prepare data for modelling use cases,
ensuring appropriate transformations, feature generation, and storage.
- Follow best practices for security, access control, and governance in Azure
- Ensure data quality, validation, and monitoring using testing tools
- Deployment of solutions to Production environments