AI Quality & Automation Engineer (LLM Focused)

Location : Remote Communication could be stronger when explaining technical experience and solutions. Extensive of hands-on automation expertise Responses should be less focused on coordination more on technical implementation. Healthcare/insurance domain knowledge preferred We build AI-powered solutions including conversational AI, RAG systems, AI agents, and enterprise automation platforms. Role Join the AI Engineering team to test and optimize Generative AI, conversational AI, and RAG applications. Focus on AI quality, benchmarking, safety, observability, and automation to ensure production-ready systems. Responsibilities Test Generative AI and conversational systems Perform LLM evaluation and benchmark testing Validate RAG (retrieval quality, relevance, grounding) Build automation frameworks (Playwright/Selenium) Conduct API, regression, and E2E testing Perform AI safety, red teaming, and bias validation Support observability, monitoring, and HITL workflows Collaborate with engineering, QA, and DevOps teams Support CI/CD quality pipelines Required Skills QA & automation testing Playwright/Selenium, API testing (Postman, REST) GenAI & RAG validation, LLM evaluation Prompt engineering, conversational AI testing HITL testing, AI safety & observability SQL, Git, Python, CI/CD concepts Preferred OpenAI, Azure OpenAI, ChatGPT, Claude LangChain, CrewAI, LangGraph, MCP Vector DBs (Pinecone, ChromaDB, Weaviate) RAGAS, DeepEval Python/JS, Docker, Kubernetes, GitHub Actions Show more Show less