AI Ops - AI/ML Engineer

AI Ops - AI/ML Engineer

FICO (NYSE: FICO) is a leading global analytics software company, helping businesses in 100+ countries make better decisions. Join our world-class team today and fulfill your career potential!

The Opportunity

We're building the AI engine behind root cause analysis for a platform that supports over 100 enterprise customers — this role owns everything from the vector database to the confidence scoring that decides whether an engineer gets notified automatically or a human steps in.

What You'll Contribute

• Design and operate the pgvector schema across all collections (past incidents per tenant, runbooks, operational context, CI/CD change events), owning indexing strategy, query performance tuning, and strict data isolation across 100+ tenants with zero cross-tenant leakage.
• Build and maintain embedding pipelines that convert operational documents into searchable vector representations, selecting embedding models and designing chunking strategies for structured JSON, unstructured prose, and time-series data types.
• Build the RAG retrieval layer that runs at incident time, tuning relevance thresholds, re-ranking strategies, and hybrid (dense and sparse) search to surface the most contextually relevant documents per collection.
• Design the full ingestion lifecycle, including runbook imports, historical incident backfill, CI/CD post-deploy event hooks, and the feedback loop that embeds resolved incidents back into the vector database.
• Design and iterate the prompt that assembles Grafana's stateless RCA, live signal data, and RAG-retrieved context into a single input, instructing Claude to reason across evidence types and produce a structured, actionable RCA readable in under 30 seconds.
• Define and maintain the JSON schema for RCA output (root cause, confidence score, contributing factors, immediate actions, SLA impact, precedent reference, escalation path) and build the Pydantic validation layer that catches schema violations before delivery.
• Own the calibrated 0–1 confidence scoring framework weighting RAG precedent strength, signal consistency, deployment correlation, runbook recognition, root cause specificity, and customer context fit, and build the FastAPI gate logic that routes RCAs to the correct action tier.
• Maintain a golden test set of 20–30 historical incidents and run the full pipeline against it after every prompt or RAG change, using a judge-prompt approach to score accuracy and catch regressions.
• Run monthly calibration checks comparing confidence scores against SRE-confirmed outcomes, calculating Expected Calibration Error (ECE) and adjusting thresholds accordingly.
• Build a prompt versioning system with changelog tracking, and monitor token usage, cost per RCA, and API latency, alerting on unexpected cost trends.
• Track retrieval quality metrics including relevance scores, coverage gaps, cross-tenant isolation verification, and ablation testing of RCA quality with and without each data collection.

What We're Seeking

• 4+ years of experience in AI/ML engineering, data engineering, NLP, or a closely related field, including hands-on LLM work.
• Production experience building and operating a vector database (pgvector, Pinecone, Weaviate, or equivalent).
• Strong understanding of embedding models and how embedding quality affects retrieval accuracy.
• Experience designing and operating data ingestion pipelines at scale (Celery, Airflow, or similar).
• Demonstrated experience prompt engineering for production LLM systems requiring reliable structured outputs, not just chatbots or demos.
• Experience building LLM evaluation frameworks, including golden test sets, judge prompts, and accuracy metrics.
• Strong Python skills, including FastAPI, Pydantic, SQLAlchemy, and async patterns.
• Clear understanding of how RAG context affects LLM reasoning and how to diagnose retrieval-driven errors.
• Familiarity with semantic search concepts: cosine similarity, approximate nearest neighbor, and hybrid search.
• Production experience with the Claude API (messages format, system prompts, structured output patterns) is a plus.
• Experience with LLM output validation frameworks such as Pydantic or Guardrails is a plus.
• Understanding of confidence calibration techniques (ECE, Platt scaling, temperature effects) is a plus.
• Exposure to SRE or observability domains, and familiarity with multi-tenant architectures where tenant isolation is a hard requirement, is a plus.
• Knowledge of re-ranking models (cross-encoders) and experience with Kubernetes/AWS EKS is a plus.

Our Offer to You

• An inclusive culture strongly reflecting our core values: Act Like an Owner, Delight Our Customers and Earn the Respect of Others.
• The opportunity to make an impact and develop professionally by leveraging your unique strengths and participating in valuable learning experiences.
• Highly competitive compensation, benefits and rewards programs that encourage you to bring your best every day and be recognized for doing so.
• An engaging, people-first work environment offering work/life balance, employee resource groups, and social events to promote interaction and camaraderie.

Why Make a Move to FICO?

At FICO, you can develop your career with a leading organization in one of the fastest-growing fields in technology today – Big Data analytics.  You’ll play a part in our commitment to help businesses use data to improve every choice they make, using advances in artificial intelligence, machine learning, optimization, and much more.

FICO makes a real difference in the way businesses operate worldwide:

•    Credit Scoring — FICO® Scores are used by 90 of the top 100 US lenders.

•    Fraud Detection and Security — 4 billion payment cards globally are protected by FICO fraud systems.

•    Lending — 3/4 of US mortgages are approved using the FICO Score.

Global trends toward digital transformation have created tremendous demand for FICO’s solutions, placing us among the world’s top 100 software companies by revenue. We help many of the world’s largest banks, insurers, retailers, telecommunications providers and other firms reach a new level of success. Our success is dependent on really talented people – just like you – who thrive on the collaboration and innovation that’s nurtured by a diverse and inclusive environment. We’ll provide the support you need, while ensuring you have the freedom to develop your skills and grow your career.  Join FICO and help change the way business thinks!

Learn more about how you can fulfil your potential at www.fico.com/Careers

FICO promotes a culture of inclusion and seeks to attract a diverse set of candidates for each job opportunity. We are an equal employment opportunity employer and we’re proud to offer employment and advancement opportunities to all candidates without regard to race, color, ancestry, religion, sex, national origin, pregnancy, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. Research has shown that women and candidates from underrepresented communities may not apply for an opportunity if they don’t meet all stated qualifications. While our qualifications are clearly related to role success, each candidate’s profile is unique and strengths in certain skill and/or experience areas can be equally effective. If you believe you have many, but not necessarily all, of the stated qualifications we encourage you to apply.

Information submitted with your application is subject to the FICO Privacy policy at https://www.fico.com/en/privacy-policy

Experience Level

Mid Level