Data Engineer

Chief Lakehouse Architect – Databricks CoE

Chief Lakehouse Architect – Databricks CoE1

Principal architect responsible for defining and executing enterprise-grade Databricks Lakehouse architecture standards, engagements and guiding complex solution designs. Drive cost optimization and financial governance for Databricks platforms across clients and internal workloads.

Key Responsibilities

• Define Databricks reference architectures (batch, streaming, ML, GenAI)

• Own standards for Delta Lake, medallion architecture, Unity Catalog

• Design multi-workspace Databricks platform architectures

• Lead Databricks migrations from legacy data warehouses programs

• Design, Review and approve architecture for all Databricks deals

• Mentor architects and senior Databricks engineers

• Define Databricks FinOps framework

• Optimize cluster policies, DBU usage, and job scheduling

• Work with engineering teams on performance-to-cost optimization

• Advise leadership and clients on Databricks cost strategies

Experience

• 12–15 years in data architecture

• 5–7 years hands-on Databricks & Apache Spark

• 3+ years designing enterprise Databricks platforms, managing Databricks costs at scale

Skills

• Databricks platform internals & optimization

• Spark performance tuning and scalability

• Delta Lake internals (ACID, compaction, Z-order)

• Databricks Jobs, Workflows, CI/CD, IaC

• Cloud-native Databricks (AWS / Azure / GCP)

• Cluster tuning & auto-scaling strategies, Databricks DBU model & pricing

• Cloud billing & monitoring tools

• Cost optimization patterns for Spark workloads

Certifications (Preferred)

• Databricks Data Engineer Professional

• FinOps Practitioner

Experience Level

Senior Level