Objective:
The objectives of a Data Engineer typically revolve around designing, implementing, and managing robust data architectures and infrastructures. The Data Engineer having 7+ years of exp will play a key role in driving the organization's data strategy, ensuring the efficient flow and storage of data, and supporting data-driven decision-making processes.
Duties and Responsibilities:
Data Model Design:
- Develop and maintain a scalable and efficient data models and schemas that support the storage, processing, and retrieval of data.
Data Integration:
- Implement robust ELT (Extract, Load, Transform) processes to integrate data from various sources into a unified and consistent format.
- Ensure seamless data flow between different systems, databases, and applications.
Data Quality:
- Implement data validation and cleansing processes to ensure the accuracy and reliability of data.
Emerging Technologies:
- Stay abreast of emerging technologies and trends in the field of data engineering.
Database Management:
- Manage and optimize database systems, including relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
Performance Optimization:
- Identify and address performance bottlenecks in data processing and storage systems.
- Optimize queries, indexing strategies, and data partitioning for improved performance.
Scalability and Reliability:
- Ensure that data systems are scalable to handle growing volumes of data.
- Implement redundancy and fault-tolerance measures to enhance data system reliability.
Mandatory Skills Required:
- Strong hands-on expertise with AWS services such as S3, Glue, Lambda, Athena, Redshift, Step Functions, Streaming tools like Apache Kafka \ Amazon Kinesis.
- Proficiency in Python, PySpark, experience with ELT and orchestration tools
- Strong hands-on expertise on Apache Hudi \ Apache Iceberg
- Strong hands-on expertise In SQL
- Experience using GitHub for version control and collaboration
- Develop, optimize, and maintain ELT processes for data integration.
- Strong communication skills to interface with technical and business stakeholders.
Secondary Skills Required:
- Knowledge of cloud infrastructure automation (e.g., Terraform, CloudFormation).
- Experience with DevOps CICD pipelines and source code management.
- AWS Certification, experience in platform automation and monitoring frameworks.