What is this position about?
We are looking for a Data Quality Engineer with strong experience in Azure and Databricks to ensure data quality, reliability, and consistency across modern data platforms. This role focuses on validating data pipelines, implementing automated quality checks, and collaborating closely with Data Engineering and business teams to guarantee accurate and production-ready data assets.
- Design and implement a data quality framework across Bronze, Silver, and Gold layers — defining validation rules, threshold tolerances, and alerting standards
- Build and maintain automated data quality checks within Databricks pipelines — row counts, null checks, referential integrity, schema validation, and business rule assertions
- Own reconciliation between source systems and Databricks layers — ensuring source data lands accurately and transformations produce expected outputs
- Validate identity resolution outputs in the Silver layer — reviewing match rates, investigating false positives and false negatives, and ensuring enterprise identifiers are being assigned correctly across source populations
- Perform end-to-end pipeline testing — validating that data flows correctly from ingestion through to the Gold layer and that downstream reporting outputs reflect accurate data
- Partner with Data Engineers to define acceptance criteria for each sprint’s pipeline and data model deliverables before they are promoted to production
- Support UAT with client business stakeholders — helping them validate that Gold layer outputs meet their reporting requirements
- Document all QA processes, test results, and data quality findings in a format that can be handed off to the client team at engagement close
- Monitor pipeline health post-deployment — investigating and triaging data quality incidents and working with engineers to resolve root causes quickly