Senior Data Engineer with Databricks and AWS Expertise

Senior Software Engineer

Job Description: Data Engineer with Databricks Expertise

Job Summary:

EXP- 4 to 6 Years

Work Location - Pune / Chennai / Bangalore

We are seeking a highly skilled Data Engineer with Databricks certifications and extensive practical experience working with the Databricks Lakehouse platform and its supporting capabilities. The ideal candidate will possess a robust understanding of AWS services relevant to data engineering and demonstrate proficiency in data pipeline development, programming, data modeling, and governance.

Key Responsibilities

• Design and implement data pipelines using AWS and Databricks.
• Manage and orchestrate data workflows, ensuring efficient data ingestion processes.
• Utilize Delta Lake for data storage and management, ensuring secure data sharing across platforms with Delta Sharing.
• Implement data governance and access control using Unity Catalog.
• Apply the Medallion architecture (Bronze, Silver, Gold layers) to projects.
• Develop and manage ETL pipelines using AWS Glue.
• Leverage Amazon S3 for data storage and management.
• Utilize AWS Lambda and Step Functions for serverless computing in data processing.
• Apply data warehousing solutions and best practices in data transformation.
• Monitor and implement data quality checks and governance practices.
• Collaborate with data scientists, analysts, and other stakeholders in an Agile environment.
• Deliver large-scale data processing and analytics projects within cloud environments.

Required Skills and Qualifications

• Databricks certifications with practical experience in the Databricks Lakehouse platform.
• Proficient in AWS services: Glue, S3, Lambda, Step Functions, and data warehousing solutions.
• Strong programming skills in Python, Scala, or Java.
• Proficient in SQL for querying and manipulating data.
• Experience in data modeling for Lakehouse architecture.
• Knowledge of ETL best practices and data transformation techniques.
• Understanding of data governance frameworks and practices.
• Proven track record of delivering data engineering projects, particularly in cloud environments.
• Experience with large-scale data processing and analytics projects.
• Ability to work in Agile teams and collaborate effectively with various stakeholders.

Preferred Qualifications

• Advanced AWS certifications.
• Experience with additional cloud platforms (e.g., Azure).
• Familiarity with other data engineering tools and frameworks.