Principal AI Engineer - Agentic Systems and Evaluation Platform

Principal AI Engineer - Agentic Systems and Evaluation Platform

Career Category

Information Systems

Job Description

Job Description

About Amgen 

Amgen harnesses the best of biology and technology to fight the world’s toughest diseases, and make people’s lives easier, fuller and longer. We discover, develop, manufacture and deliver innovative medicines to help millions of patients. Amgen helped establish the biotechnology industry more than 40 years ago and remains on the cutting-edge of innovation, using technology and human genetic data to push beyond what’s known today. 

About the Role 

As a Principal AI Engineer, you will own the architecture, engineering, and ongoing operation of evaluations as a shared component of our enterprise AI platform. Your primary mandate is to build and manage the services, integrations, and infrastructure that enable teams to evaluate agentic AI systems consistently, securely, and at scale. 

You will establish a common evaluation capability across models, agent frameworks, and application types, with self-service interfaces, reusable framework integrations, and consistent management of evaluation configurations, datasets, runs, and results. Application teams and domain experts will use this capability to define and maintain the evaluation content and acceptance criteria for their use cases. 

As the hands-on technical lead for a small engineering team (~2-3 engineers), you will own the component roadmap and production lifecycle, partnering with platform, application, infrastructure, security, and governance teams. The role combines strong platform engineering with a deep understanding of how agentic systems are evaluated. 

What You Will Do 

• Own the evaluation platform component from architecture and delivery through adoption, operations, and continuous improvement; define its roadmap, service boundaries, integration contracts, and engineering standards. 

• Build self-service APIs, SDKs, developer integrations, and user interfaces for configuring evaluations, submitting and tracking runs, and accessing results within existing engineering workflows. 

• Create an extensible integration layer for evaluation frameworks, scorers, model endpoints, and agent runtimes, enabling teams to use supported tools and add evaluators through consistent interfaces. 

• Engineer reliable evaluation execution services with asynchronous orchestration, scheduling, concurrency management, retries, cancellation, and workload isolation across projects and teams. 

• Provide versioning and lifecycle management for datasets, evaluation configurations, evaluator definitions, and result artifacts, with metadata and lineage that make runs reproducible and comparable. 

• Support agentic evaluation through common contracts for inputs, outputs, traces, tool calls, and execution trajectories; provide controlled execution and replay patterns where appropriate. 

• Build reusable evaluator components and configuration patterns for output quality, retrieval, tool use, task completion, safety, latency, and cost, while enabling application teams to supply domain-specific evaluation content. 

• Integrate evaluation execution and results with CI/CD, experiment tracking, observability, and promotion workflows; enable configurable quality thresholds and evidence collection for release and governance review. 

• Deliver consistent results schemas, comparison views, scorecards, and reporting APIs so teams can examine trends and compare model, prompt, and agent versions within project context. 

• Embed authentication, authorization, project isolation, auditability, and data retention controls, working with security and governance partners to protect evaluation datasets, traces, and results. 

• Operate the component as a production service, defining service objectives, monitoring reliability and execution performance, managing capacity and cost visibility, and leading incident resolution and upgrades. 

• Drive adoption through documentation, onboarding patterns, reference integrations, and technical support; use platform telemetry and feedback to prioritize improvements and reduce integration effort. 

What We Expect of You 

• Own evaluations as an enduring platform capability, with clear interfaces, dependable operations, and a roadmap informed by multiple teams. 

• Turn recurring application requirements into reusable capabilities and extension points, balancing standardization with flexibility across agent frameworks and workloads. 

• Measure success through platform adoption, service reliability, reproducibility, evaluation turnaround time, and developer experience. 

• Provide hands-on technical leadership through architecture and code reviews, mentoring, and collaboration across engineering and governance teams. 

Basic Qualifications 

• Bachelor's or master's degree in computer science, AI, or a related field 

• 10+ years of experience in software engineering, AI engineering, ML systems, or platform engineering 

• Strong Python and software engineering skills, including the design of production APIs, reusable libraries, and distributed backend services 

• Experience building and operating shared platform services or infrastructure consumed by multiple applications or engineering teams 

• Hands-on experience integrating AI evaluation frameworks, experiment tracking, or automated AI quality workflows into production engineering environments 

• Understanding of LLM and agentic architectures, including orchestration, RAG, tool use, state management, and the evaluation data and traces these systems produce 

• Experience with asynchronous processing, cloud services, CI/CD, observability, versioning, and secure access to enterprise systems 

• Ability to define service contracts, make architecture tradeoffs, and lead engineering delivery and operations across teams 

Preferred Qualifications 

• Experience owning an evaluation platform component, shared testing infrastructure, or an ML platform service through design, adoption, and production operations 

• Experience integrating tools such as MLflow, RAGAS, DeepEval, or Promptfoo through reusable adapters and common configuration and results interfaces 

• Experience designing plugin architectures, workload isolation, job orchestration, artifact storage, and versioned APIs or schemas 

• Experience connecting agent traces, production feedback, and human review systems to shared evaluation infrastructure 

• Familiarity with the strengths and limitations of deterministic checks, LLM-as-judge scoring, and human assessment, and their implications for platform design 

• Experience with enterprise security and governance controls, usage metering, and capacity and cost management for AI services 

Soft Skills 

• Strong technical leadership and mentoring, with ownership from architecture through production support 

• Ability to translate needs across application teams into a coherent platform roadmap 

• Clear communication of service capabilities, integration patterns, operating standards, and technical tradeoffs 

• Practical engineering judgment and a strong focus on developer experience and service reliability 

Equal Opportunity Statement 

Amgen is an Equal Opportunity employer and will consider you without regard to your race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or disability status.

.

Experience Level

Senior Level