What You Will Be Doing
As a Principal Observability Engineer, you will lead the vision, strategy, and engineering implementation of Observability at scale across Experian's hybrid infrastructure and global product ecosystem. Responsibilities include:
Observability Platform Engineering
- Architect, design, and evolve Observability frameworks, instrumentation standards, and reusable libraries across compute, microservices, APIs, and data platforms.
- Lead the transition to Observability as Code, enabling Splunk, Cribl, Dynatrace, and future tool configurations to be fully versionâcontrolled, automated, and consumed via Terraform and CI/CD pipelines.
- Build advanced distributed tracing solutions using OpenTelemetry, ensuring deep visibility across upstream and downstream systems.
Coding, APIs & Automation
- Build highâquality APIs, SDKs, and automation frameworks that standardize Observability onboarding for engineering teams.
- Develop and maintain an APIâdriven orchestration layer that abstracts underlying Observability tooling.
- Enable engineering teams to instrument applications through reusable code modules, templates, and policyâdriven automation.
Tooling & Standards Deployment
- Evaluate, adopt, and deploy new Observability tooling, dashboards/tiles (tiling exposure), and visualization standards using infrastructureâasâcode, pipelines, and programmatic configuration.
- Lead tooling modernization initiatives for synthetic monitoring, browser instrumentation, APM, logs, and telemetry ingestion.
CrossâFunctional Collaboration
- Work with platform, SRE, DevOps and product engineering teams to define, build, and mature endâtoâend monitoring standards.
- Influence architecture decisions to ensure Observability considerations are embedded early across services.
- Partner with Cloud Engineering & Operations, leveraging practices from infrastructure selfâservice programs
Leadership & Governance
- Define the Observability roadmap, KPIs, and success criteria for global adoption.
- Mentor senior engineers, drive knowledge sharing, and build a culture of proactive Observability excellence.
- Contribute to engineering governance through patterns, blueprints, guardrails, and audit/compliance alignment.
What Your Background Looks Like
Technical Expertise
- 10â15+ years in largeâscale engineering organizations with deep Observability ownership.
- Advanced proficiency in at least one coding language (Python, Go, Java preferred).
- Proven experience building APIs and API frameworks that enable selfâservice consumption.
- Strong handsâon in distributed tracing, OpenTelemetry, logs, metrics, events, and telemetry flow management.
- Expertise with tools such as Dynatrace, Splunk, Cribl, Prometheus, Grafana, Datadog, etc.
- Strong IaC skills using Terraform, Ansible; comfortable deploying full tooling ecosystems through CI/CD.
- 5+ years of experience managing Observability in ecosystems with multiple upstream/downstream service dependencies.
- Knowledge of cloud platforms (AWS, GCP, Azure) and hybrid architectures.
Architecture & Design
- Deep understanding of microservices, event-driven patterns, API gateways, middleware, and message queues.
- Experience designing Observability for complex, globally distributed systems.
Soft Skills
- Communicate technical strategy to senior stakeholders.
- Passion for simplifying complex problems through automation.
- Lead initiatives and drive largeâscale adoption.
Nice to Have
- Experience implementing templated dashboards, tiles, and standardized visualization frameworks.
- Exposure to cost optimization, data ingestion pipelines, or AIâdriven monitoring approaches.