Senior Data Engineering Lead

Program Manager

Description

Role Overview:

We are seeking a highly experienced Senior Data Engineering Lead with a strong focus on Databricks and cloud-based data engineering to lead our data engineering team. This leadership role requires a visionary who can design, develop, and manage scalable data infrastructure and pipelines, while mentoring and inspiring a team of data engineers. You will work closely with cross-functional teams including data scientists, analysts, and software engineers to enable robust data-driven decision-making and support business goals.

Key Responsibilities:

• Lead and manage a team of data engineers, providing mentorship, technical guidance, and fostering a culture of collaboration and innovation.
• Architect, design, and oversee implementation of large-scale data pipelines, data lakes, and cloud-based data warehouses using Databricks, Apache Spark, and Snowflake.
• Develop and optimize ETL/ELT workflows ensuring performance, reliability, and scalability of data infrastructure.
• Collaborate with business stakeholders, data scientists, and software teams to understand requirements and translate them into scalable, efficient data solutions.
• Implement best practices for data quality, governance, security, and compliance.
• Drive continuous improvement of data engineering processes, standards, and tools across the organization.
• Support presales activities by contributing to RFPs, technical proposals, and client engagements.
• Stay abreast of emerging data technologies and trends, recommending innovative solutions to enhance analytics capabilities.
• Manage resource planning, project prioritization, and delivery timelines ensuring alignment with business objectives.
• Lead performance reviews, identify skill gaps, and champion professional development within the data engineering team.
• Facilitate cross-team communication to streamline data workflows and improve overall delivery.

Qualifications & Skills:

• Bachelor’s or Master’s degree in Computer Science, Data Science, Engineering, or a related discipline.
• Minimum 15 years of professional experience in data engineering with at least 9 years in leadership or senior technical roles.
• Deep hands-on expertise with Databricks and Apache Spark for large-scale data processing.
• Strong programming skills in Python, Scala, or Java.
• Extensive experience with cloud data platforms such as AWS, Azure, or GCP, including services like S3, Redshift, BigQuery, Snowflake.
• Solid understanding of data modeling, data warehousing, ETL/ELT design, and data lakes.
• Experience with big data technologies like Hadoop, Kafka, and Databricks ecosystem.
• Knowledge of CI/CD pipelines, data orchestration tools (e.g., Apache Airflow), and data governance best practices.
• Proven experience managing high-performing teams and delivering complex data engineering projects on time.
• Familiarity with analytics solutions and the ability to translate business needs into technical requirements.
• Strong communication skills, capable of engaging with both technical teams and senior leadership.
• Experience supporting presales efforts and client technical discussions is a plus.
• Bonus: Exposure to machine learning lifecycle and model deployment on Databricks.