Databricks Data Engineering Lead

Data Lead

Data Lead2

We are seeking a highly skilled and experienced Databricks Lead to join our Data Engineering team. In this role, you will be responsible for designing, developing, and optimizing large-scale data pipelines and solutions using Databricks on [Azure/AWS/GCP]. You will play a key role in transforming raw data into actionable insights and help drive data-first decisions across the organization.

Key Responsibilities:

Design and implement scalable ETL/ELT pipelines using Apache Spark on Databricks.
Build and optimize data lakes and data warehouses using Delta Lake and related tools.
Develop, schedule, and monitor workflows using Databricks Workflows, Airflow, or other orchestration tools.
Collaborate with data scientists, analysts, and stakeholders to understand data needs and deliver reliable solutions.
Tune Spark jobs for performance, scalability, and cost-efficiency.
Implement CI/CD pipelines and DevOps best practices for data engineering projects.
Ensure data security, compliance, and governance using tools like Unity Catalog.
Troubleshoot and resolve issues related to data ingestion, transformation, and integration.
Participate in data platform migration projects (e.g., from on-premise to Databricks or from legacy platforms to modern cloud-native architectures).
Mentor junior team members and contribute to a culture of technical excellence and innovation.

Required Qualifications:

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
12+ years of hands-on experience in data engineering or big data development.
6+ years of experience working with Databricks, including Spark, Delta Lake, and MLflow.
Strong programming skills in Python and SQL.
Solid understanding of distributed systems and performance tuning in Spark.
Experience with cloud platforms such as Azure, AWS, or GCP (Azure Databricks preferred).
Experience working with structured, semi-structured, and unstructured data.
Familiarity with DevOps practices, including version control (Git), CI/CD, and infrastructure as code (Terraform, ARM templates).
Excellent problem-solving skills and ability to communicate complex technical ideas to non-technical stakeholders.
Experience in Databricks migration projects (e.g., from Hadoop, on-premise, or legacy ETL tools to Databricks)

Ability to thrive in a fast-paced, dynamic, client-facing role where delivering solid work products to exceed high expectations is a measure of success.
Excellent leadership and interpersonal skills.
Eager to contribute to a team-oriented environment.
Strong prioritization and multi-tasking skills with a track record of meeting deadlines,
Ability to be creative and analytical in a problem-solving environment.
Effective verbal and written communication skills.
Adaptable to new environments, people, technologies, and processes
Ability to manage ambiguity and solve undefined problems.

Experience Level

Senior Level