Senior Azure Data Engineer

Role Description 
PWC India is seeking a highly skilled Azure Data Engineer with 6 + years of experience to join our team and play a key role in designing, developing, and implementing efficient data solutions on the Azure cloud platform. The ideal candidate should have a strong foundation in data engineering, hands-on expertise with Azure services, solid proficiency in data processing technologies, especially PySpark, and a good understanding of AI-enabled data engineering components that support modern analytics and automation use cases. 

Responsibilities 

Data Pipeline Development 

• Design, build, and optimize end-to-end data pipelines on Azure with a focus on scalability, reliability, and performance. 

• Develop and maintain ETL/ELT workflows to support seamless data ingestion and processing. 

Azure Cloud Expertise 

• Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks to deliver effective data engineering solutions. 

• Implement and manage scalable data storage solutions on Azure. 

Data Transformation with PySpark 

• Leverage PySpark for advanced data transformation and processing to ensure high-quality, well-structured outputs. 

• Implement data cleansing, enrichment, and validation processes using PySpark. 

AI-Enabled Data Engineering 

• Support AI and analytics use cases by preparing, transforming, and curating high-quality datasets for downstream consumption. 

• Work with modern Azure and Databricks capabilities to enable AI/ML data pipelines, metadata-driven processing, and intelligent automation where applicable. 

• Collaborate with analytics and data science teams to support model-ready data preparation and scalable feature pipelines. 

Performance Optimization 

• Optimize data pipelines, SQL queries, and PySpark jobs to improve overall performance and scalability. 

• Identify and resolve performance bottlenecks across data processing workflows. 

Mandatory skill sets:

• Proven experience as a Data Engineer, with strong expertise in Azure-based data solutions. 

• Hands-on experience with Azure Data Factory and other relevant Azure services. 

• Strong proficiency in PySpark for data processing and analytics is mandatory. 

• Experience in data modeling, ETL/ELT processes, and data warehousing. 

• Exposure to AI/ML data preparation, feature engineering pipelines, or AI-enabled data platforms is preferred. 

Primary Skill sets:

• Azure Data Factory 

• PySpark 

• Azure Databricks 

• Azure SQL Database 

• Data Modeling and Data Warehousing 

• AI/ML Data Preparation 

• Metadata-Driven Data Engineering