About the role
As a Data Engineer, you will be at the heart of our AI-driven evolution, implementing scalable pipelines and high-quality data governance that fuel our powerful IR Ops Platform. We鈥檙e looking for a technical trailblazer who operates with integrity and is ready to work cross-functionally to deliver remarkable outcomes for our global client base.
What you'll do
- Responsible for implementing and maintaining data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams.
- Extend support to implement governance across data landscape and ensure data ingress/egress follows the set hygiene guidelines and sanity rules.
- Work with stakeholders including the executive, product, data and design teams to assist with data acquisition, data-related technical issues and other analytics needs.
- Work cross-functionally to explore and propose solutions to business problems that can be addressed using insights from data.
- Responsible for various types of documentation including business requirements, functional/technical specifications, process flows, unit test plans, user acceptance plans etc.
Tasks
- Build high quality, scalable, optimized and maintainable data pipelines based on Q4 best practices.
- Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
- Contribute to definition and implementation of standards and best practices for Data Analytics and Data Governance.
- Maintain and troubleshoot the infrastructure built for optimal extraction, transformation, and loading of data from a wide variety of data sources.
- Identify, design, and implement process improvements: automate manual processes, optimize data delivery, improve data reliability, efficiency, and quality, etc
- Build analytics tools that utilize the data pipeline to provide actionable insights into student learning, customer acquisition, operational efficiency, and other key metrics.
Qualifications
- Minimum of 4+ years of professional experience in a Data Engineer or related role.
- Working familiarity with a variety of different storage mechanisms including SQL & No-SQL databases, Data Warehouses, and Data Lakes.
- Experience working with AWS Cloud platforms and related systems.
- Experience building and optimizing data pipelines, architectures, and data sets.
- Experience with big data tools: Databricks, Spark, Kafka, etc.
- Experience with data pipeline and workflow management tools, such as Airflow, Snowpipe.
- Experience with real-time data processing and stream-processing systems: Kinesis, Spark-Streaming, etc.
- Experience in requirements analysis, design, implementation, and testing of software solutions, especially data related, using Python, Scala, and/or other programming languages.
- A successful history of manipulating, processing and extracting value from large disconnected datasets.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Strong project management, organizational and communication skills.