Senior Specialist - Architecture

Senior Specialist - Architecture

TECHNICAL EXPECTATIONS

REQUIRED

• LLM APIs and Agentic SDKs
• Production experience with at least one major LLM API Anthropic Claude OpenAI Azure OpenAI or Amazon Bedrock including the full surface chat completions streaming token counting context management retrybackoff and ratelimit handling
• Deep familiarity with toolcalling functioncalling defining tool schemas parsing toolcall results multiturn tool loops and handling partial or malformed tool outputs gracefully
• Experience with structured output constraining LLM output to a JSON
• schema validating it and building retry loops that reprompt on validation failure without thrashing
• Understands context window management token budgeting across a multiturn conversation what to summarise vs what to keep verbatim how context window size affects latency and cost
• Comfortable with multistep agentic loops planning toolcall observe replan cycles and knows how to detect and recover from loops hallucinated tool names and stuck states
• Agentic Architecture Patterns
• Designs humanintheloop workflows as firstclass system components not as an afterthought interruption but as a structured review step with a clear proposal format diff and approvalrejection path that feeds back into the agents context
• Understands the difference between orchestrator agents and subagents knows when to delegate to a specialised subagent vs handle in one prompt and designs the handoff contract between them
• Designs stateful multisession agentic pipelines checkpoint and resume idempotent reexecution of steps audit trail of every agent action and human decision
• Familiar with ReAct PlanandExecute and reflection patterns knows when each applies and what their failure modes are
• Experience building providerabstraction layers wrapping LLM calls behind an interface so the pipeline is not coupled to one vendor knows how to handle
• providerspecific quirks token limits toolcall formats refusals at the abstraction layer
• RetrievalAugmented Generation RAG and Ingestion
• Builds production RAG pipelines chunking strategy by section by semantic
• boundary by token count embedding model selection vector store indexing retrieval and reranking
• Experience ingesting heterogeneous source types PDFs Word documents
• structured JSONXMLXLSX and unstructured legacy system exports each with its own extraction and preprocessing strategy
• Understands retrieval quality evaluation recallk mean reciprocal rank how to build a golden dataset and measure retrieval against it
• Experience with at least one vector store Pinecone Weaviate pgvector
• Chroma Qdrant including index management metadata filtering and hybrid search semantic keyword
• Evaluation and AI Governance
• Builds LLM evaluation frameworks defining what correct means for a given agent task building test sets running evals as a CI gate before promoting a prompt or pipeline change
• Designs AI assurance audit trails every AIassisted action is captured
• what was proposed what the human approved with what version of the prompt and model queryable and immutable
• Understands AI failure modes hallucination sycophancy prompt injection
• instruction following degradation at long context and how to defend against each in a production system
• Familiar with model evaluation dimensions factual accuracy instruction
• following format compliance output safety and knows which matter most for configurationauthoring tasks
• Languages and Frameworks
• Proficient in Python for AI pipeline work async patterns Pydantic for
• structured output validation httpxaiohttp for LLM API calls pytest for eval harnesses
• Proficient in TypeScriptNodejs for AInative API clients and agent integrations in the platforms designtime surface
• Familiar with at least one agentic framework LangChain LangGraph
• LlamaIndex CrewAI AutoGen Haystack or Anthropics agent SDK and can reason about what the framework buys vs what it obscures
• Comfortable writing productiongrade tested observable Python services
• not just Jupyter notebooks knows how to package deploy and monitor an agent pipeline
• Observability for AI Systems
• Instruments agent pipelines with LLMspecific observability traces that
• capture prompt completion token counts latency tool calls and model version per step
• Familiar with tools like LangSmith Langfuse Arize or equivalent for LLM trace capture and prompt debugging
• Builds cost dashboards token spend per pipeline per model per run type
• AI pipelines have variable and sometimes surp