Role Overview:
The Microsoft Fabric Data Engineer is responsible for designing, building, optimizing, and operating enterprise-scale data platforms on Microsoft Fabric. This role focuses on data ingestion, transformation, storage, governance, automation, and platform reliability using OneLake, Lakehouse, Data Warehouse, Data Pipelines, and Spark technologies.
The position is heavily focused on Data Engineering and Platform Engineering rather than reporting and dashboard development. The successful candidate will build scalable, governed, and high-performance data solutions that support analytics, AI, operational reporting, and business intelligence initiatives across the organization.
This role is primarily focused on Data Engineering, Data Platform Development, and Microsoft Fabric architecture. Candidates whose experience is primarily centered around Power BI report development, dashboard creation, or data visualization without substantial Data Engineering experience may not be a fit for this position.
Key Responsibilities
1. Data Engineering & Platform Development
- Design and implement enterprise-scale Lakehouse architectures using OneLake and Delta Lake.
- Build and maintain robust batch, incremental, CDC, and near real-time data ingestion pipelines.
- Develop scalable ETL/ELT solutions using Fabric Data Pipelines, Dataflows Gen2, PySpark, and SQL.
- Implement and manage Medallion Architecture (Bronze, Silver, Gold).
- Develop reusable and metadata-driven ingestion and transformation frameworks.
- Integrate data from ERP systems, SAP, REST APIs, SQL Server, Dataverse, and other enterprise applications.
- Design and maintain enterprise data models supporting analytical and operational workloads.
2. Fabric Engineering & Optimization
- Develop and optimize Fabric Lakehouses and Data Warehouses.
- Build advanced Notebook-based transformations utilizing PySpark and Spark SQL.
- Implement Delta Lake capabilities including:
- Merge/Upsert
- Change Data Feed (CDF)
- Time Travel
- Schema Evolution
- Optimize
- Vacuum
- Design scalable storage, partitioning, and file management strategies within OneLake.
- Optimize Spark workloads and Data Warehouse performance.
3. Performance, Reliability & Monitoring
- Tune large-scale Spark and SQL workloads.
- Implement monitoring, alerting, and operational dashboards using Fabric Monitoring Hub and Metrics.
- Conduct performance testing and scalability assessments.
- Troubleshoot pipeline failures and platform performance bottlenecks.
- Define and monitor SLAs for critical data assets and pipelines.
- Ensure high availability and operational excellence across data workloads.
4. DevOps & Platform Automation
- Implement CI/CD using Fabric Git Integration and Deployment Pipelines.
- Build automated deployment processes across Development, Test, and Production environments.
- Develop automated testing frameworks for data quality, schema validation, and regression testing.
- Manage environment configurations, secrets, and deployment parameters.
- Utilize Azure DevOps or GitHub Actions to support release automation and governance.
- Define rollback, release management, and change control processes.
5. Data Quality, Governance & Security
- Implement automated data quality validation frameworks.
- Develop reconciliation, completeness, and consistency checks.
- Manage data lineage, metadata, and documentation standards.
- Implement sensitivity labels, access controls, auditing, and compliance requirements.
- Ensure adherence to organizational standards for governance, security, retention, and PII handling.
- Manage workspace permissions, Managed Identities, RLS, and CLS where required.
6. Technical Leadership & Collaboration
- Translate business requirements into scalable technical solutions.
- Collaborate with Data Architects, Data Analysts, Integration Engineers, and business stakeholders.
- Lead engineering best practices and platform standards.
- Mentor junior engineers and support knowledge-sharing initiatives.
- Produce architecture documentation and technical design specifications.
Drive continuous improvement of the data platform and engineering practices.