Company:
Oliver Wyman
Description:
The Role
We are looking for an experienced Lead Quality Engineer – AI Delivery to lead Quality Engineering across multiple AI delivery pods building enterprise-scale AI solutions. This role will be based in Cluj and has a requirement of working at least three days a week in the office.
Our engineering teams already operate within an established shift-left Secure SDLC, with core engineering, security, code quality, and pipeline controls embedded into delivery. This role builds on that foundation, ensuring each AI solution has an appropriate risk-based quality strategy, with particular focus on GenAI evaluation, end-to-end quality, UAT readiness, and release confidence.
You will work closely with Product Managers, Product Owners, Engineering Leads, Architects, and developers, combining strategic quality leadership with hands-on technical capability.
We will count on you to:
- Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.
- Partner with Product and Engineering to define test strategy, acceptance criteria, UAT approach, quality risks, and release-readiness criteria.
- Ensure appropriate coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.
- Define testing and evaluation approaches for LLM, RAG, agentic, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behaviour, and non-deterministic outputs.
- Establish appropriate AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.
- Guide and, where needed, implement reusable automation across web, APIs, backend services, integrations, asynchronous workflows, and AI components.
- Ensure products are ready for UAT, with appropriate test evidence, environments, test data, defect status, entry/exit criteria, and known risks.
- Provide Pod Leads and programme leadership with clear quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release-readiness reporting.
- Conduct quality reviews across pods and identify gaps in testing, testability, observability, or quality evidence.
- Use production incidents, escaped defects, telemetry, and user feedback to continuously improve quality approaches.
- Coach engineers and delivery teams to strengthen shared ownership of quality across the lifecycle.
What you need to have
- Proven experience leading Quality Engineering for enterprise-scale AI / GenAI solutions in production.
- Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.
- Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.
- Experience working across multiple delivery teams or pods in complex enterprise environments.
- Strong software engineering and automation skills in TypeScript / JavaScript and/or Python.
- Experience with modern automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.
- Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.
- Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.
- Good understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.
- Strong stakeholder management skills and the ability to translate quality evidence into clear risks, metrics, and recommendations for leadership.
What makes you stand out
- Experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.
- Experience testing complex RAG, multi-agent, or tool-calling solutions.
- Experience with performance, resilience, accessibility, or security testing.
- Experience using production telemetry and incidents to improve test and evaluation strategies.
- Ability to balance strong quality controls with pragmatic, fast-moving AI delivery.
Technology context
Typical technologies include:
React, TypeScript, Node.js / NestJS, Python / FastAPI, REST APIs, async workflows, AWS / Azure, GitHub Actions / GitLab CI / Azure DevOps, LLM APIs, RAG, agentic workflows, LangChain, LangSmith, and Mastra.
Why join our team:
- We help you be your best through professional development opportunities, interesting work, and supportive leaders.
- We foster a vibrant and inclusive culture where you can work with talented colleagues to create new solutions and have an impact for colleagues, clients, and communities.
- Our scale enables us to provide a range of career opportunities, as well as benefits and rewards to enhance your well-being.
- A yearly budget and the opportunity to build your flexible benefits package (up to 20 percent of your annual salary).
- 30+ days off, including legal days, birthday, public holiday replacements, and benefits options.
- Performance bonus scheme.
- Matching charity contributions, charity days off, and the Pay it Forward charity challenge.
- Core benefits: Pension, Life and Medical Insurance, Meal Vouchers, Travel Insurance.
Marsh (NYSE: MRSH) is a global leader in risk, reinsurance and capital, people and investments, and management consulting, advising clients in 130 countries. With annual revenue of over $27 billion and more than 95,000 colleagues, Marsh helps build the confidence to thrive through the power of perspective. For more information, visit marsh.com, or follow us on LinkedIn and X.
Marsh is committed to creating a diverse, inclusive and flexible work environment. We aim to attract and retain the best people and embrace diversity of age, background, disability, ethnic origin, family duties, gender orientation or expression, marital status, nationality, parental status, personal or social status, political affiliation, race, religion and beliefs, sex/gender, sexual orientation or expression, skin color, or any other characteristic protected by applicable law.
Marsh is committed to hybrid work, which includes the flexibility of working remotely and the collaboration, connections and professional development benefits of working together in the office. All Marsh colleagues are expected to be in their local office or working onsite with clients at least three days per week. Office-based teams will identify at least one “anchor day” per week on which their full team will be together in person.