Natixis in Portugal is currently seeking Google Cloud Platform (GCP) Data Engineers to join its BPCE SI (BPCE Solutions Informatiques) team, within the IT (Information Technologies) department, to work in a hybrid model in the Porto or Lisbon office.
Responsibilities
Data and Database Migration:
- Actively participate in the design and execution of the data and ETL process migration strategy from SQL Server to GCP.
- Develop and optimize scripts and pipelines for efficiently transferring and transforming data into BigQuery.
Design and Development of Data Pipelines on GCP:
- Design, develop, and maintain robust and high-performing ETL/ELT pipelines on GCP, replacing or rebuilding existing flows.
- Orchestrate and schedule these pipelines using Google Cloud Composer (or Apache Airflow) to ensure reliable migration and operationalization.
Database Management and Optimization on GCP:
- Develop and optimize complex SQL queries for BigQuery, taking into account the specificities of migrating from SQL Server to BigQuery (e.g., optimizing partitioning, clustering).
- Contribute to the design and implementation of optimized data schemas for BigQuery, adapted to new cloud architectures.
Development and Automation:
- Use Python to develop scripts for automating migration tasks, data transformation, and integration with other GCP services.
- Implement processes to ensure data quality and integrity during and after migration.
Collaboration and Support:
- Collaborate closely with current teams (SQL Server administrators, analysts) to understand the existing environment and facilitate the transition.
- Exchange with end-users (business lines, data scientists) to understand their needs and ensure the new platform meets their expectations.
- Participate in code reviews, development best practices, and testing strategies for the new platform.
- Ensure monitoring, debugging, and incident resolution related to data pipelines during the transition phase and upon production deployment.
Technological Watch and Optimization:
- Stay informed about best practices for migration and the latest developments on Google Cloud Platform.
- Propose improvements and optimizations for the new Data platform on GCP.