Databricks and PySpark Solution Architect

Databricks + Pyspark Architect

Experience · 8–14 years of overall IT experience · 4–6+ years as a Data / Analytics / Platform Architect · Strong hands-on experience architecting Databricks Lakehouse platforms Databricks Expertise · Deep experience with: o Databricks Lakehouse architecture o Delta Lake o Unity Catalog o MLflow · Strong understanding of batch and streaming data architectures · Experience designing multi-workspace or multi-tenant setups Agentbricks & AI Knowledge · Hands-on or strong working knowledge of Agentbricks · Understanding of: o AI agent concepts (tools, memory, reasoning loops) o RAG architectures and vector search o LLM-driven workflows over enterprise data · Experience integrating agents with data, APIs, and downstream systems Data Engineering & Programming · Strong SQL expertise · Proficiency in Python and PySpark · Experience with ETL/ELT and data orchestration patterns · Familiarity with APIs and event-driven integration • Knowledge of architectural design patterns, performance tuning, database and functional designs • Hands-on experience in Service Oriented Architecture • Ability to lead solution development and delivery for the design solutions • Experience in designing high level and low level documents is a plus • Good understanding of SDLC is a pre-requisite • Awareness of latest technologies and trends • Logical thinking and problem solving skills along with an ability to collaborate Databricks & Lakehouse Architecture · Define and own end-to-end Databricks Lakehouse architecture · Design scalable platforms for: o Structured, semi-structured, and unstructured data o Analytical and AI workloads · Establish architecture patterns, best practices, and reference designs · Lead modernization from legacy data platforms to Databricks-based lakehouse Agentbricks & AI Agent Enablement · Architect and enable AI agent-based solutions using Agentbricks · Design data and system patterns to support: o AI agents reasoning over enterprise data o Tool-calling and action-oriented agents o Retrieval-Augmented Generation (RAG) workflows · Define how agents interact with: o Delta tables o Vector embeddings o Feature stores and semantic layers · Guide teams on agent orchestration, memory, observability, and safety Advanced Analytics & AI Platform Design · Enable data architectures that support: o Machine learning pipelines o Feature engineering and reuse o Real-time and batch inference · Integrate Databricks with: o LLM providers and model endpoints o Vector search and embedding storage · Collaborate with data scientists and AI engineers to productionize AI solutions Databricks Platform Components · Architect and govern solutions using: o Delta Lake o Unity Catalog (governance, lineage, access control) o Databricks Workflows and Jobs o MLflow (experiments, models, lifecycle) · Define cluster, compute, and workspace strategies for multi-team environments Security, Governance & Compliance · Implement secure data and AI access using: o Unity Catalog permissions o Cloud IAM integrations · Define governance for: o AI agent data access o Model and agent versioning o Auditability and traceability · Ensure compliance with enterprise security and regulatory requirements Performance, Cost & Operations · Design for performance optimization and scalability · Optimize compute usage, storage, and agent execution costs · Enable monitoring and observability for: o Data pipelines o ML workloads o AI agents · Partner with platform and FinOps teams on cost control strategies Technical Leadership & Collaboration · Act as the Databricks and AI architecture authority · Review solution designs and guide engineering teams · Mentor data engineers, ML engineers, and architects · Translate business and AI requirements into scalable technical designs