• Govern data acquisition and ingestion architecture for enterprise-wide data & analytics integration.
• Establish reusable, scalable, and governed interfaces to all source systems across business domains.
• Define standards and ensure engineering quality across ingestion pipelines and upstream data flows.
• Act as escalation and design authority for ingestion, unstructured data, and advanced analytics integration points.
• Oversees ingestion and integration design across 100+ data sources.
• Supports 25+ data engineering and data science professionals with ingestion architecture and troubleshooting.
• Interfaces with source system experts, data scientists, platform architects, and functional IT stakeholders.
Main Tasks
• Define ingestion frameworks for batch, streaming, and APIs.
• Establish standards for JSON, CSV, XML, and unstructured data.
• Govern manual uploads and exception workflows.
• Collaborate with system owners to define interface specifications.
• Manage schema evolution and API reliability.
• Align ingestion architecture with platform and business requirements.
• Collaborate on ingestion for data science and ML pipelines
• Ensure robust API and event-driven integration for AI/LLM use cases.
• Support complex acquisition scenarios with scalability requirements.
• Review high-risk ingestion pipelines for performance and security.
• Support data engineers on error handling, retries, and load orchestration.
• Ensure ingestion aligns with governance and architecture.
• Maintain ingestion libraries and templates.
• Promote reuse and automation across engineering teams.
• Lead best practices in TDD, code reviews, and pipeline testing.
• Maintain architectural blueprints, templates, and best practices.
• Publish design guidelines and coding standards.
• Create re-usable architecture patterns for lakehouse environments.
• Monitor usage and implement auto-scaling policies.
• Analyze and optimize cluster configurations for cost-efficiency.
• Provide cost transparency and usage reporting to stakeholders.
continental