IN_Senior Associate_Data Engineer Databricks_Data and Analytics_Advisory_Hyderabad

Job Description & Summary:  

We are seeking an experienced Data Engineer – Databricks to design, build, and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform. The role involves hands-on development using Apache Spark, Databricks notebooks, Delta Lake, and cloud-native services, along with close collaboration with analytics, AI/ML, and business teams 

Responsibilities: 

• Design, build, and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL) 

• Implement batch and incremental data processing using Delta Lake and multi-hop architecture 

• Develop and optimize Databricks notebooks, jobs, and workflows 

• Ingest, transform, and curate large-scale structured and semi-structured datasets 

• Support analytics, reporting, and downstream data consumption use cases 

• Ensure data quality, reliability, lineage, and governance 

• Collaborate with data scientists, analysts, and architects on AI/ML workloads 

• Optimize Spark jobs for performance and cost efficiency 

• Adhere to enterprise security, access control, and compliance standards 

• Provide production support and troubleshoot data pipeline issues 

• Document technical designs, data flows, and operational runbooks 

• Mentor junior engineers and contribute to best practices 

Mandatory skill sets: 

4+ years of experience as a Data Engineer with strong Databricks expertise 

• Hands-on experience with Apache Spark, PySpark, and Spark SQL 

• Strong knowledge of Delta Lake and Lakehouse architecture 

• Advanced SQL skills 

• Experience with ETL/ELT patterns and data warehousing concepts 

• Exposure to at least one cloud platform (Azure / AWS / GCP) 

• Understanding of distributed computing concepts 

• Experience working in Agile teams 

Preferred skill sets: 

Experience with Unity Catalog and data governance 

• Exposure to Auto Loader or streaming frameworks 

• CI/CD for data pipelines 

• Python for data engineering and automation 

• Databricks certification (Associate / Professional) 

Years of experience required: 

5 to 8 years 

Education qualification: 

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field (60% above)