Senior Generative AI Engineer

Senior AI Engineer - LLM Evaluation

26WD98447

Position Overview

If you love building real AI systems that real customers use—and you are passionate about measuring, improving, and scaling AI quality—this role is for you. The Applied AI team in Autodesk's AEC organization builds cloud-native AI solutions that make Architecture, Engineering, and Construction workflows smarter, faster, and more connected.

This role is primarily focused on AI evaluation and quality engineering for Generative AI systems. You will establish evaluation standards, build automated validation frameworks, and drive observability practices that ensure AI systems are reliable, trustworthy, and production-ready. While evaluation is the primary focus, you will also contribute to the design and development of AI-powered applications including RAG systems, agentic workflows, and LLM-based solutions deployed at scale.

Responsibilities

• Lead AI evaluation and quality strategy for Generative AI and ML-powered systems

• Design and implement scalable evaluation frameworks for LLMs, RAG systems, agentic workflows, and traditional ML models

• Fine-tune, evaluate, benchmark, and deploy large language models in production environments

• Implement evaluation and observability standards including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems

• Define and track AI quality metrics including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates

• Build automated evaluation pipelines and integrate them into CI/CD workflows for continuous model validation

• Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tooling

• Build scalable RAG pipelines using structured and unstructured enterprise data sources

• Integrate AI agents with Autodesk platforms and enterprise services using MCP-based integrations or equivalent patterns

• Partner with product managers, architects, developers, designers, and data scientists to deliver AI-powered experiences

• Ensure AI systems align with Responsible AI principles and enterprise governance requirements

• Contribute to reusable AI platform capabilities, SDKs, orchestration patterns, and engineering best practices

• Use AI coding agents to accelerate feature delivery while ensuring rigorous validation through tests, CI/CD, code reviews, and security practices

Minimum Qualifications

• Bachelor’s or Master’s degree in Computer Science, Machine Learning, Data Science, Engineering, or equivalent practical experience
• 5+ years of hands-on experience delivering AI/ML-powered systems into production
• Experience designing evaluation frameworks, automated testing solutions, benchmarking systems, or model validation pipelines
• Strong understanding of LLM evaluation, model quality measurement, and AI observability concepts
• Strong proficiency in Python with experience building automation frameworks and testing infrastructure
• Experience deploying and operating AI applications in production, including monitoring, observability, and continuous improvement
• Experience building cloud-native applications on AWS, Azure, or Google Cloud
• Excellent technical communication and collaboration skills
• Demonstrated experience using AI coding tools to accelerate delivery of production systems

Preferred Qualifications

• Hands-on experience with AI evaluation frameworks such as LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, or similar tools
• Experience evaluating RAG systems, agentic workflows, AI assistants, and prompt-based applications
• Familiarity with Responsible AI, AI governance, model risk assessment, and AI observability platforms
• Experience with cloud-native AI deployment using AWS, Azure, or Google Cloud
• Familiarity with deep learning architectures such as Transformers and modern ML frameworks including PyTorch, Lightning, Ray, or equivalent technologies
• Experience with enterprise knowledge systems, semantic search, vector search, or contextual retrieval systems
• Experience with emerging Agentic AI frameworks such as LangGraph, CrewAI, OpenAI SDKs, Claude SDKs, MCP, A2A, or equivalent technologies
• Contributions to open-source AI projects, technical blogs, conference talks, or publications are a plus

#LI-SJ1

Learn More

About Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Salary transparency

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging
We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

Are you an existing contractor or consultant with Autodesk?

Please search for open jobs and apply internally (not on this external site).

Experience Level

Senior Level