As a Data Engineer, you will be responsible for:
- Building and maintaining processes for collecting data from various sources into the Data Lake.
- Designing, developing, and optimizing complex data pipelines to ensure reliable data flow.
- Creating, developing, and maintaining frameworks that facilitate the construction of data pipelines.
- Implementing comprehensive testing frameworks for data pipelines.
- Collaborating with analysts and data scientists to ensure the delivery of high-quality data.
- Ensuring robust data management, security, and compliance practices.
- Researching and implementing new technologies to improve data pipeline performance.
- Leveraging and integrating data from various types of source systems, including Kafka, MQ, SFTP, databases, APIs, and file shares.