Agentic Artificial Intelligence Engineer

Lead the Future of AI Quality: AI Engineer - Evaluation & Test Automation! Step into an influential engineering role establishing automated quality control harnesses for enterprise support agents and Retrieval-Augmented Generation platforms. You will benchmark system performance against strict standards and shape the reliability, safety, and precision of next-generation conversational AI tools serving users worldwide. About Us : AECO Trimble is empowering stakeholders through planning, design, construction and the entire asset lifecycle. The company’s innovative and connected ecosystem of solutions improves coordination and collaboration between teams, phases and processes. Automating work and transforming workflows, Trimble is enabling teams to deliver with confidence at every turn. For more information, visit: construction.trimble.com. What Makes This Role Great: In this role, you will be at the absolute forefront of generative AI engineering, defining industry-standard evaluation pipelines and automated safety barriers that prevent hallucinations and regressions before code reaches production. Key Exciting Responsibilities Architect and maintain automated evaluation pipelines to programmatically score Faithfulness, Answer Relevance, and Context Recall across complex AI agentic workflows. Curate and expand high-quality test datasets using real-world customer interactions, edge cases, negative permission paths, and synthetic generation techniques. Execute daily automated regression testing over code commits to ensure all prompt updates and tool contracts meet defined accuracy and security thresholds overnight. Conduct root-cause failure analysis on production escalations and retrieval misses, converting customer friction points into robust automated test scenarios. Benchmark agent resilience against prompt injection attacks, unauthorized tool invocations, privilege escalations, and uncertified response generation. Essential Skills & Experience 3+ years of software engineering or QA automation experience, with at least 1 year dedicated to evaluating LLMs, RAG pipelines, or stateful agentic workflows. Strong proficiency in Python, Pytest, Docker, and REST API automation alongside expertise manipulating evaluation data with JSON/JSONL, SQL, and pandas. Hands-on experience with RAG evaluation frameworks such as Ragas, DeepEval, or TruLens, or custom LLM-as-a-Judge grading harnesses. Proven background integrating automated quality gates into CI/CD pipelines using tools like GitHub Actions, Azure DevOps, or GitLab CI/CD. Bonus Points For Familiarity with stateful agent frameworks like LangGraph or AutoGen and standardized tool protocols such as Model Context Protocol (MCP). Understanding of statistical sampling methods for evaluating unstructured text and complex conversational data. Logistics Location: Westminster, CO (Hybrid: 4 days in office, 1 day from home) Travel Requirement: 0-10% Why You'll Love Working With Us: At Trimble, we're not just a company that "does good"—we are a team dedicated to making a tangible, positive Real-World Impact. We build innovative solutions designed to solve the world's most critical challenges. From construction sites to transportation hubs, our work tangibly improves how people live, build, move, and grow. You'll work on projects that truly matter: Our purpose-driven culture means you'll be helping to build and deliver solutions that make work faster, safer, and more sustainable for millions of people worldwide. Our impact is tangible, from connected machines that save fuel to data-driven insights that reduce waste. Collaborate with like-minded people: Our strong internal culture is a "hidden gem." You will work with a collaborative, supportive team that shares your purpose and fosters a genuine sense of belonging. We're a company of "visionary pragmatists" who think boldly and build things that work. Be an owner: Trimble thrives on individuals who take initiative and embrace ownership. You'll find an entrepreneurial spirit where success is often "self-authored," empowering proactive "doers." AI Engineer, QA Automation, LLM Evaluation, RAG, Eval-as-Code, Python, DeepEval, Ragas, TruLens, Continuous Integration, Model Context Protocol, Test Automation Compensation: Trimble provides the following compensation range and general description of other compensation and benefits that it in good faith believes it might pay and/or offer for this position. This compensation range is based on a full time schedule. Trimble reserves the right to ultimately pay more or less than the posted range and offer additional benefits and other compensation, depending on circumstances not related to an applicant’s sex or other status protected by local, state, or federal law. Hiring Range $170,600.00–$234,600.00 Pay Rate Type Salary Bonus Eligible? Yes Commission Eligible? No Benefits: Trimble offers comprehensive core benefits that include Medical, Dental, Vision, Life, Disability, Time off plans and retirement plans. Most of our businesses also offer tax savings plans for health, dependent care and commuter expenses as well as Paid Parental Leave and Employee Stock Purchase Plan. If this position is identified above as commission- or bonus-eligible, the terms of the commission plan or discretionary bonus plan for which you are eligible will be provided following the employee start date. How to Apply: Please submit an online application for this position by clicking on the ‘Apply Now’ button located in this posting. Application Deadline: Trimble accepts applications on an ongoing basis until the position is filled. At Trimble, we are committed to fostering a diverse, inclusive, and equitable workplace where everyone can thrive. Guided by our core values—Belong, Innovate, and Grow—we embrace and celebrate differences, knowing they make us stronger and more innovative. We are proud to be an equal opportunity employer, welcoming individuals of all backgrounds and advancing opportunities while embracing race, color, gender identity, sexual orientation, religion, disability, veteran status, or any other protected and diverse characteristic. We are committed to offering our candidates and employees with disabilities or sincerely held religious beliefs the ability to seek reasonable accommodations in accordance with applicable law and/or where it would not constitute undue hardship for Trimble. For more, please see Trimble's Code of Business Conduct and Ethics at https://investor.trimble.com, under “Corporate Governance.” Our mission to transform the way the world works starts with transforming how we work together. By actively listening, asking questions, and taking intentional actions, we cultivate a culture that provides equitable opportunities for everyone to contribute and grow. Candidate Privacy Notices by Country If you need assistance or would like to request an accommodation in connection with the application process, please contact careers@trimble.com. Show more Show less