Main objective of the job:
- Develop Data Pipelines: Design and operate automated ETL/ELT workflows using Azure tools to process and deliver data.
- Manage the Data Platform: Provision, secure, and optimize the underlying Azure infrastructure for our data solutions.
Main attributions and responsibilities:
- Data Pipelines: Assist in designing and maintaining data pipelines using Azure Data Factory and Azure Databricks.
- ETL Development: Build and optimize data transformation workflows in Python, focusing on data quality and performance.
- Azure Storage: Manage data within Azure Blob Storage and Data Lake Storage, including basic partitioning and access tier management.
- Security & Access: Gain hands-on experience in securing data via Azure Key Vault and implementing RBAC (Role-Based Access Control) strategies.
- Orchestration: Support the automation of data workflows using tools like Airflow or Kafka.
- Cloud Awareness: Learn and apply best practices for Azure cost management and resource automation.
- Collaboration: Work within an international Agile team, documenting processes to ensure operational excellence.
- Fulfils any other tasks related to area of responsibility, as requested by direct superior.