Anticipated Contract End Date/Length: December 31, 2026
Work set up: Hybrid
Our client in the Information Technology and Services industry is looking for a Lead Azure Databricks Engineer to drive the design, build, optimisation, and governance of enterprise data platforms within a highly regulated insurance environment. This is a 100% hands-on role requiring deep technical expertise combined with strong leadership capability.
The successful candidate will operate within complex, fast-paced settings and bring proven delivery experience aligned to Lloyd鈥檚 of London regulatory and operational requirements. This role will play a pivotal part in shaping cloud data strategy, enabling advanced analytics, and delivering secure, scalable, and high-performing solutions across the organisation.
What you will do:
- Lead the development and optimisation of large-scale data platforms using Azure services including ADF, Data Lake, Key Vault, and Azure Functions, alongside Databricks technologies such as Delta Lake, PySpark, and Unity Catalog
- Collaborate with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance Solvency II, and risk teams to deliver compliant and auditable data solutions
- Translate business and regulatory requirements into scalable, secure cloud data architectures
- Own and resolve complex technical challenges with clarity, speed, and accountability
- Champion best-practice design patterns, Medallion architecture, Lakehouse principles, and data lifecycle management
- Implement CI/CD pipelines using Azure DevOps, including branching and merging strategies for data solutions
- Drive automation and cloud-native engineering principles across the platform
- Provide technical leadership and mentorship, setting engineering standards and uplifting team capability
- Identify opportunities to improve performance, reduce cost, and enhance resilience through continuous improvement initiatives
- Oversee end-to-end data engineering delivery, ensuring reliability, observability, data quality, and performance optimisation