Data Quality Engineer

Job Overview

The Data Quality Engineer will ensure the accuracy, consistency, and reliability of data across the Microsoft Fabric Data Warehouse, its integrations, and connected services. The role focuses on testing and validating data pipelines, event-driven processes, and Azure-based integrations such as Logic Apps and Service Bus. The engineer will collaborate closely with data engineers, architects, and business units to maintain high-quality, trustworthy data for analytics and reporting.

Key Responsibilities

1. Design, develop, and execute data quality and validation tests for Microsoft Fabric Data Warehouse objects (tables, views, semantic models).
2. Validate ETL/ELT pipelines, including Fabric Dataflows and integrated source systems, for accuracy and completeness.
3. Test and monitor event-driven data processes, including Azure Service Bus messages, queues, and topics.
4. Validate Azure Logic Apps workflows and integrations that move or transform data between systems.
5. Perform end-to-end testing of data ingestion, transformations, and delivery to downstream consumers or BI systems.
6. Develop and maintain automated data quality checks, reconciliation scripts, and test frameworks.
7. Identify, analyze, and report data quality issues, including root cause analysis and impact assessment.
8. Collaborate with Data Engineers, BI teams, and business units to define data quality rules, metrics, and acceptance criteria.
9. Monitor data quality KPIs, implement controls to prevent defects, and ensure reliability in production pipelines.
10. Document test cases, workflows, data quality rules, and testing outcomes clearly for technical and business stakeholders.
11. Familiarity with unit testing, automated data validation, and integration testing tools such as dbt, Great Expectations, tSQLt, pytest, Azure Logic Apps testing, and Service Bus testing scripts.
12. Perform unit testing and validation of Azure Logic Apps workflows and Service Bus message flows.
13. Develop and maintain automated data quality checks and integration tests using tools such as dbt, Great Expectations, or custom scripts.
14. Validate and test API endpoints and integration workflows using tools such as Postman to ensure data is accurately transmitted between systems.
15. Perform end-to-end testing of event-driven processes and service integrations using Postman and automated scripts.