Azure Databricks

Azure Databricks

Job Description: Data Engineer (Azure Databricks & PySpark)

Position: Data Engineer
Experience: 3 to 5 years
Primary Skills: Azure Databricks, PySpark, SQL
Secondary Skills: ADF (Azure Data Factory)
Location: Bengaluru/Hyderabad
Mode of Work: Hybrid
Notice Period: 0 to 30 Days

Key Responsibilities:

ETL Development & Data Processing:

• Design, develop, and maintain ETL pipelines using Azure Databricks, PySpark, and ADF.
• Extract, transform, and load data from multiple sources into Azure Data Lake Storage.
• Optimize and fine-tune ETL workflows for performance and scalability.

SQL & Query Optimization:

• Write complex SQL queries, joins, subqueries, and functions for data processing.
• Perform SQL tuning and query optimization for improved efficiency.
• Work with large datasets and ensure optimal data handling.

Azure Cloud Integration:

• Implement data solutions using Azure technologies (Databricks, ADF, ADLS).
• Work with batch and streaming data processing for real-time analytics (preferred).
• Ensure data security, compliance, and reliability in cloud-based systems.

Collaboration & Problem-Solving:

• Work closely with cross-functional teams to support data analytics and business intelligence initiatives.
• Identify and resolve data-related issues proactively.
• Provide insights and recommendations for improving ETL performance.

Required Skills & Qualifications:

3 to 5 years of experience in Azure Databricks, PySpark, and SQL.
Strong SQL skills complex joins, functions, procedures, query tuning.
Hands-on experience in ADF (Azure Data Factory) and Azure Data Lake Storage.
Ability to handle large datasets efficiently.
Experience in performance optimization of ETL workflows.
Good understanding of batch and streaming data processing (preferred).
Strong analytical and problem-solving skills.
Ability to work independently and take ownership of tasks.

Experience Level

Mid Level