AI Validation Engineer

Role Overview Our client is hiring for two QA / AI Quality Engineer roles that will support critical AI initiatives across its digital ecosystem. These roles will align to one of two teams depending on fit and business need: • Analytics AI Team – focused on validating AI-driven insights, reporting, and data-driven decisioning systems • AI Rapid Delivery Team – focused on ensuring quality in fast-moving, production-grade AI solutions and LLM-driven applications As a QA / AI Quality Engineer, you will play a key role in establishing trust in AI systems—ensuring outputs are accurate, reliable, explainable, and production-ready. Depending on team alignment, your work will either center on validating analytics-driven AI outputs and data pipelines or embedding quality into rapid AI development cycles and deployed solutions. What You’ll Do Design and implement validation and QA frameworks for AI-powered systems, including analytics platforms, LLM-driven applications, and agentic workflows Evaluate AI outputs for accuracy, grounding, consistency, and alignment with underlying data and expected behavior Test and validate key components such as: ○ RAG pipelines and retrieval quality ○ Data pipelines and transformations ○ Prompt behavior and response consistency ○ API integrations and downstream system interactions Build automated testing frameworks for structured and unstructured AI systems Define and track metrics around accuracy, data integrity, performance, and system reliability Identify discrepancies between AI outputs and source data or expected outcomes Partner closely with engineering, data, and product teams to improve system quality and reduce defects Support rapid iteration cycles by developing scalable evaluation and regression testing approaches Monitor AI systems post-deployment to detect drift, degradation, and performance issues Help implement guardrails to improve safety, reduce hallucinations, and ensure trustworthy outputs Support auditability and traceability of AI-driven systems and insights Must-Have Qualifications • 3–7+ years of experience in QA, Software Testing, Data Validation, or AI/ML-focused quality engineering • Strong Python skills for automation, testing, and data validation • Experience building automated test frameworks and validation pipelines • Experience testing AI/ML or data-driven systems (LLMs, analytics platforms, or similar) • Strong SQL or data querying skills and ability to validate large datasets • Understanding of data pipelines, transformations, and data quality validation • Experience validating APIs and service integrations • Ability to analyze logs, outputs, and system behavior to diagnose issues • Comfortable working in fast-paced, agile environments with evolving requirements Nice-to-Have • Experience with LLM evaluation, prompt testing, or RAG-based systems • Familiarity with agentic workflows or orchestration patterns • Exposure to LLMOps, AI evaluation tooling, or observability frameworks • Experience with model monitoring, drift detection, or performance tracking • Familiarity with analytics platforms or BI tools • Understanding of explainability techniques, governance, or risk in AI systems • Awareness of cost, latency, and performance considerations in AI solutions What Sets This Role Apart Opportunity to align to one of two high-impact teams: A data and analytics-focused team ensuring trust in AI-driven insights and decision-making A high-velocity engineering team ensuring quality in rapidly deployed AI solutions Direct ownership of quality and trust across enterprise-scale AI systems Exposure to a wide range of AI use cases, including analytics, LLMs, and real-time decisioning Strong collaboration with engineering, data, and product teams Ability to influence how the client builds, validates, and scales AI responsibly Show more Show less