Key Responsibilities:
- Engaging in the planning and design process.
- Validate ETL and data workflows running on Azure Databricks for accuracy, performance, and reliability.
- Ensuring code reliability and robustness along with function specificity
- Design, develop, and maintain automated test frameworks for data pipelines, APIs, and containerized services.
- Build and execute integration, performance, and regression test suites for Flask-based APIs.
- Implement CI/CD quality gates in containerized deployments (Kubernetes).
- Collaborate with data engineers, software developers, and DevOps teams to identify test gaps and drive quality improvements.
- Monitor and troubleshoot test failures, ensuring timely resolution of defects.
- Contribute to best practices for testing in cloud-native and data-intensive environments.
Required Skills & Experience:
- Strong programming skills in Python, with proven experience writing test automation frameworks.
- Adept with white box testing.
- Hands-on experience testing data platforms (preferably Azure Databricks, Spark, or other big data frameworks).
- Experience with API testing (REST/Flask) using frameworks like PyTest, unittest, or similar.
- Proficiency in Kubernetes, Docker, and containerized environments.
- Familiarity with CI/CD tools (e.g., Azure DevOps, GitHub Actions, Jenkins).
- Strong knowledge of software testing principles including functional, regression, integration, and performance testing.
- Experience with SQL and validating large datasets.
Preferred Qualifications:
- Experience with cloud platforms, especially Azure (Databricks, Data Lake, Kubernetes Service).
- Exposure to data quality frameworks or libraries (e.g., Great Expectations).
- Knowledge of monitoring tools (Prometheus, Grafana, or similar) for validating system health and performance.
- Strong analytical and problem-solving skills with attention to detail.