About the Role
We are seeking a skilled and forward-thinking Data Quality Engineer to advance the data trust, governance, and certification framework for our enterprise Data Lakehouse platform built on Databricks, Apache Iceberg, AWS (Glue, Glue Catalog, SageMaker Studio), Dremio, Atlan, and Power BI.
This role is critical in ensuring that data across Bronze (raw), Silver (curated), and Gold (business-ready) layers is certified, discoverable, and AI/BI-ready. You will design data quality pipelines, semantic layers, and governance workflows, enabling both Power BI dashboards and Conversational Analytics leveraging LLMs (Large Language Models).
Your work will ensure that all 9 dimensions of data quality (accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity, conformity, reliability) are continuously met, so both humans and AI systems can trust and use the data effectively.
ESSENTIAL DUTIES AND RESPONSIBILITIES
Data Quality & Reliability
Build and maintain automated validation frameworks across Bronze → Silver → Gold pipelines.
Develop tests for schema drift, anomalies, reconciliation, timeliness, and referential integrity.
Integrate validation into Databricks (Delta Lake, Delta Live Tables, Unity Catalog) and Iceberg-based pipelines.
Data Certification & Governance
Define data certification workflows ensuring only trusted data is promoted for BI/AI consumption.
Leverage Atlan and AWS Glue Catalog for metadata management, lineage, glossary, and access control.
Utilize Iceberg’s schema evolution & time travel to ensure reproducibility and auditability.
Semantic Layer & Business Consumption
Build a governed semantic layer on gold data to support BI and AI-driven consumption.
Enable Power BI dashboards and self-service reporting with certified KPIs and metrics.
Partner with data stewards to align semantic models with business glossaries in Atlan.
Conversational Analytics & LLM Enablement
Prepare and certify datasets that fuel conversational analytics experiences.
Collaborate with AI/ML teams to integrate LLM-based query interfaces (e.g., natural language to SQL) with Dremio, Databricks SQL, and Power BI.
Ensure LLM responses are grounded on high-quality, certified datasets, reducing hallucinations and maintaining trust.
ML Readiness & SageMaker Studio
Provide certified, feature-ready datasets for ML training and inference in SageMaker Studio.
Collaborate with ML engineers to ensure input data meets all 9 quality dimensions.
Establish monitoring for data drift and model reliability.
Holistic Data Quality Dimensions
Continuously enforce all 9 dimensions of data quality:
Accuracy, Completeness, Consistency, Timeliness, Validity, Uniqueness, Integrity, Conformity, Reliability.
Western Digital
https://careers.smartrecruiters.com/westerndigital