Senior Specialist - Architecture

Senior Specialist - Architecture

TECHNICAL EXPECTATIONS

REQUIRED

LLM APIs and Agentic SDKs

Production experience with at least one major LLM API Anthropic Claude

OpenAI Azure OpenAI or Amazon Bedrock including the full surface chat

completions streaming token counting context management retrybackoff

and ratelimit handling

Deep familiarity with toolcalling functioncalling defining tool

schemas parsing toolcall results multiturn tool loops and handling

partial or malformed tool outputs gracefully

Experience with structured output constraining LLM output to a JSON

schema validating it and building retry loops that reprompt on validation

failure without thrashing

Understands context window management token budgeting across a multiturn

conversation what to summarise vs what to keep verbatim how context window

size affects latency and cost

Comfortable with multistep agentic loops planning toolcall observe

replan cycles and knows how to detect and recover from loops

hallucinated tool names and stuck states

Agentic Architecture Patterns

Designs humanintheloop workflows as firstclass system components not

as an afterthought interruption but as a structured review step with a clear

proposal format diff and approvalrejection path that feeds back into the

agents context

Understands the difference between orchestrator agents and subagents knows

when to delegate to a specialised subagent vs handle in one prompt and

designs the handoff contract between them

Designs stateful multisession agentic pipelines checkpoint and resume

idempotent reexecution of steps audit trail of every agent action and human

decision

Familiar with ReAct PlanandExecute and reflection patterns knows when

each applies and what their failure modes are

Experience building providerabstraction layers wrapping LLM calls behind

an interface so the pipeline is not coupled to one vendor knows how to handle

providerspecific quirks token limits toolcall formats refusals at the

abstraction layer

RetrievalAugmented Generation RAG and Ingestion

Builds production RAG pipelines chunking strategy by section by semantic

boundary by token count embedding model selection vector store indexing

retrieval and reranking

Experience ingesting heterogeneous source types PDFs Word documents

structured JSONXMLXLSX and unstructured legacy system exports each with

its own extraction and preprocessing strategy

Understands retrieval quality evaluation recallk mean reciprocal rank

how to build a golden dataset and measure retrieval against it

Experience with at least one vector store Pinecone Weaviate pgvector

Chroma Qdrant including index management metadata filtering and hybrid

search semantic keyword

Evaluation and AI Governance

Builds LLM evaluation frameworks defining what correct means for a given

agent task building test sets running evals as a CI gate before promoting a

prompt or pipeline change

Designs AI assurance audit trails every AIassisted action is captured

what was proposed what the human approved with what version of the prompt

and model queryable and immutable

Understands AI failure modes hallucination sycophancy prompt injection

instruction following degradation at long context and how to defend against

each in a production system

Familiar with model evaluation dimensions factual accuracy instruction

following format compliance output safety and knows which matter most for

configurationauthoring tasks

Languages and Frameworks

Proficient in Python for AI pipeline work async patterns Pydantic for

structured output validation httpxaiohttp for LLM API calls pytest for

eval harnesses

Proficient in TypeScriptNodejs for AInative API clients and agent

integrations in the platforms designtime surface

Familiar with at least one agentic framework LangChain LangGraph

LlamaIndex CrewAI AutoGen Haystack or Anthropics agent SDK and can

reason about what the framework buys vs what it obscures

Comfortable writing productiongrade tested observable Python services

not just Jupyter notebooks knows how to package deploy and monitor an

agent pipeline

Observability for AI Systems

Instruments agent pipelines with LLMspecific observability traces that

capture prompt completion token counts latency tool calls and model

version per step

Familiar with tools like LangSmith Langfuse Arize or equivalent for LLM

trace capture and prompt debugging

Builds cost dashboards token spend per pipeline per model per run type

AI pipelines have variable and sometimes surp