Associate Data Engineer Consultant

Associate Consultant - Data Engineer

Roles & Responsibilities: 

• Data Pipeline ETL Development: Design and optimize ETL/ELT processes using Python/PySpark and SQL to efficiently ingest, transform, and integrate data from various sources into Azure data platforms.​
• Data Architecture Integration: Leverage Azure Data Factory, Databricks, and Synapse Analytics to establish a medallion architecture in ADLS, transforming raw data into refined and actionable formats. ​
• Power BI Integration: Collaborate with power bi developers and data analysts to ensure seamless integration of user-facing elements.​
• Full Software Development Lifecycle: Participate in all phases of the SDLC, including requirement analysis, planning, development, testing, and deployment.​
• Cross-Functional Collaboration: Work closely with cross-functional teams, including data scientists, data analysts and product managers, to solve technical and business challenges collaboratively.
• Good to have: Experience in creating and managing data pipelines and notebooks with Microsoft Fabric. Familiarity with Gen AI techniques for enhancing data engineering tasks.​

Required  Must have and Good to have Skill details:

Must have:

• Python
• SQL (Stored Procedures / Views)
• Apache Spark (PySpark)
• Azure Cloud is Mandatory and the tech stacks as follows,
  • ADF / Synapse Analytics
  • Azure Databricks
  • ADLS
  • Azure KeyVault
• Delta Lake
• Version control: Azure Repos (Preferably) / GitHub

Good to Have:

• Microsoft Fabric
• GenAI (Preferably Azure OpenAI)
• Azure Functions
• PowerBI Basics

• Data Pipeline ETL Development: Design and optimize ETL/ELT processes using Python/PySpark and SQL to efficiently ingest, transform, and integrate data from various sources into Azure data platforms.​
• Data Architecture Integration: Leverage Azure Data Factory, Databricks, and Synapse Analytics to establish a medallion architecture in ADLS, transforming raw data into refined and actionable formats. ​
• Power BI Integration: Collaborate with power bi developers and data analysts to ensure seamless integration of user-facing elements.​
• Full Software Development Lifecycle: Participate in all phases of the SDLC, including requirement analysis, planning, development, testing, and deployment.​
• Cross-Functional Collaboration: Work closely with cross-functional teams, including data scientists, data analysts and product managers, to solve technical and business challenges collaboratively.​
• Good to have: Experience in creating and managing data pipelines and notebooks with Microsoft Fabric. Familiarity with Gen AI techniques for enhancing data engineering tasks.​

 

Educational qualifications

•Bachelor’s or Master's degree in a relevant field. •Familiarity with Azure Cloud platform. •Experience in an Agile development environment.

 

Work experience

  3-4 years of total and relevant  work experience

• Effective communication and interpersonal skills, with the ability to collaborate with cross-functional teams
• Self-motivated, proactive, and able to manage multiple projects simultaneously

Experience Level

Mid Level