Ever wondered what it feels like to be a Data Engineer (Senior-Lead) in Netcompany ?
Joining us as a Data Engineer (Senior-Lead), you will have strong experience working with relational databases, SQL, big data tools such as PySpark and Spark SQL, and orchestration tools like Apache Airflow. This role also requires experience in using Notebooks (e.g., Jupyter, Databricks) for development and collaboration. Depending on your experience level, you may contribute to or lead data engineering projects, mentor team members, and drive key architectural decisions.
As a Data Engineer (Senior-Lead) you will:
- Design, develop, and maintain scalable data pipelines and ETL/ELT processes to support business requirements.
- Write and maintain complex SQL queries to extract, transform, and load data into various systems.
- Utilize PySpark and Spark SQL in Notebooks (e.g., Jupyter, Databricks) to process and analyze large datasets efficiently in distributed computing environments.
- Use of orchestration tools such as Apache Ariflow to schedule, monitor, and optimize data workflows and pipeline execution.
- Collaborate with business analysts, data architects, and other stakeholders to ensure seamless data integration across systems.
- Ensure the quality, accuracy, and consistency of data by implementing data validation and monitoring mechanisms.
- Troubleshoot and resolve issues related to data processing performance, data integrity, and system scalability.
- Contribute to the development of data models, architectures, and workflows to support analytics and reporting needs.
- Lead data architecture and design decisions, mentor junior team members, and drive best practices within the data engineering team. Take ownership of large-scale data engineering projects and provide strategic leadership in data initiatives.