Position Summary
The Azure Platform & Database Administrator is responsible for the administration, reliability, security, and operational excellence of CAQH’s Azure data platforms, including Databricks, Azure SQL, the Enterprise Data Warehouse (EDW), and Azure Data Factory. This role will design and manage the technical foundations that support data engineering, analytics, and product delivery ensuring our platforms are stable, scalable, and aligned with CAQH’s modernization strategy.
The ideal candidate brings deep expertise in Azure cloud infrastructure, Databricks platform administration, SQL database management, and automated DevOps practices. This individual will own platform governance, manage schema/catalog structures, oversee data access and security, optimize performance, and partner closely with data engineers and product teams to enable smooth development and deployment workflows.
This is a full-time, remote, exempt position reporting to the Sr. Director, Data Engineering & Architecture.
Specific Responsibilities
Databricks Platform Administration
- Manage and maintain all Databricks workspaces across development, test, and production environments.
- Administer Unity Catalog / Hive Metastore, including schema design, permissions, external locations, mounts, and data governance configurations.
- Manage clusters, cluster policies, job clusters, cluster performance, auto-scaling, and cost optimization.
- Oversee Databricks Jobs, workflows, task orchestration, alerting, and SLAs.
- Configure and maintain external data sources and connectors (ADLS Gen2, Delta tables, SQL MI, Snowflake, APIs).
- Implement and maintain CI/CD pipelines for Databricks notebooks, SQL, and infrastructure using Azure DevOps/GitHub Actions.
- Develop, maintain, and enforce platform security standards including RBAC, service principals, secrets, key vault integration, and audit logging.
Azure Platform & SQL Database Administration
- Administer Azure SQL Managed Instance and other SQL environments supporting the CAQH EDW.
- Perform database performance tuning, indexing, partitioning, storage management, and query optimization.
- Oversee backup/restore operations, failover strategies, and high availability configurations.
- Manage schema changes, deployments, and environment synchronization across dev, test, and prod.
- Partner with the Data Engineering team on efficient ingestion and transformation patterns to SQL-based systems.
- Monitor capacity, costs, and system health for all SQL and storage resources in Azure.
ADF & Integration Platform Operations
- Administer Azure Data Factory including linked services, integration runtimes, pipelines, triggers, and parameterized workflows.
- Implement enterprise standards for logging, error handling, retries, alerting, and operational monitoring.
- Ensure secure configuration and compliance for all data movement operations, including PII/PHI safeguards.
Platform Governance & Standards
- Establish and enforce standards for data access, identity management, secrets rotation, key management, and auditability.