AI Architect

Gen AI Architect – Agentic AI for Data Engineering Platform Automation Role Summary We are seeking a Gen AI Architect to design and lead the implementation of agentic AI solutions that automate and modernize our data engineering platform. This role sits at the intersection of generative AI architecture and data engineering, requiring someone who can design intelligent, autonomous systems (agents) that orchestrate, monitor, and optimize data pipelines, quality checks, transformations, and operational workflows — with minimal human intervention. Experience: 15+ years overall in technology/architecture roles, with demonstrated hands-on experience architecting agentic AI / LLM-based systems, and working knowledge of data engineering platforms and practices. Key Responsibilities Agentic AI Architecture & Strategy Design end-to-end architecture for multi-agent systems that automate data engineering tasks (pipeline orchestration, data quality remediation, schema drift detection, anomaly resolution, metadata management, etc.) Define agent design patterns: planning/reasoning loops, tool-use frameworks, memory/state management, multi-agent orchestration, human-in-the-loop checkpoints Select and evaluate frameworks (e.g., LangGraph, AutoGen, CrewAI, Semantic Kernel, or custom orchestration) suited to enterprise-scale deployment Establish architecture standards for LLM integration, prompt/context management, retrieval-augmented generation (RAG), and tool/function calling within data workflows Data Engineering Platform Automation Partner with data engineering teams to identify high-value automation opportunities across ingestion, transformation, orchestration (Airflow/dbt/Spark, etc.), and observability layers Architect agents that can read pipeline metadata, logs, and lineage to autonomously detect, diagnose, and (where appropriate) remediate failures Ensure automation solutions integrate cleanly with existing data platforms (data warehouses/lakehouses, ETL/ELT tools, orchestration engines, catalogs) Technical Leadership & Governance Define guardrails, evaluation frameworks, and monitoring for agent reliability, safety, and cost (token usage, latency, hallucination risk) Establish patterns for human oversight, escalation, and rollback in autonomous workflows Lead technical design reviews, POCs, and pilot-to-production transitions for agentic solutions Mentor engineering teams on agentic design principles and best practices Experience Required Qualifications 15+ years in software/data/AI architecture roles, including enterprise-scale system design 2+ years of hands-on experience designing or implementing agentic AI / LLM-based systems (not just prototypes — production or near-production exposure preferred) Solid foundational experience in data engineering: pipeline design, ETL/ELT, orchestration tools, data quality frameworks (does not need to be a hands-on data engineer, but must be conversant enough to architect solutions that integrate with these systems) Technical Skills Strong understanding of LLM architectures, prompt engineering, RAG, embeddings/vector stores, and function/tool calling Experience with agent orchestration frameworks (LangGraph, AutoGen, CrewAI, Semantic Kernel, or equivalents) or building custom orchestration layers Familiarity with data engineering tooling: Airflow, dbt, Spark, Kafka, cloud data warehouses/lakehouses (Snowflake, Databricks, BigQuery, Redshift, etc.) Cloud architecture experience (AWS, Azure, or GCP) including AI/ML services Understanding of API design, microservices, and event-driven architectures Familiarity with MLOps/LLMOps practices — model evaluation, observability, cost/latency monitoring Preferred / Nice-to-Have Experience architecting agentic systems specifically for data ops / DataOps automation Exposure to data governance, lineage, and catalog tools (Collibra, Alation, Unity Catalog, etc.) Certifications in cloud architecture or AI/ML (AWS/Azure/GCP Solutions Architect, etc.) Experience with evaluation/guardrail frameworks for autonomous agents (e.g., agent testing harnesses, red-teaming for AI systems) Prior experience in a "platform automation" or "AI-driven operations" initiative Show more Show less