Senior Data Engineer

Associate Process Manager

Description

We are seeking a Senior Data Engineer with strong developer DNA — someone who codes daily in PySpark, SQL, and Python, builds end-to-end data solutions, and leads design of robust, production-grade data platforms on Microsoft Fabric, Azure Databricks, and ADF. This is a hands-on leadership role for engineers who can think architecturally and code efficiently.

Key Responsibilities:

• Design, develop, and optimize complex data pipelines and lakehouse architectures.
• Lead development of batch and streaming ingestion frameworks using ADF, Databricks, and Event Hubs.
• Write optimized PySpark/SQL transformations for performance, scalability, and cost efficiency.
• Build reusable data frameworks and automation scripts for continuous integration and deployment.
• Establish unit testing, code review, and versioning standards across engineering teams.
• Integrate with DevOps CI/CD pipelines for automated releases and environment management.
• Mentor junior engineers on coding practices and architecture design.
• Collaborate with data architects on Fabric-based Lakehouse implementations.

Skills & Qualifications:

• 6–9 years of strong hands-on experience in data engineering and development.
• Expertise in:
  • PySpark, SQL, Python, and performance tuning of large-scale transformations.
  • Databricks, ADF, ADLS, Fabric Data Pipelines, and Delta Lake.
• Deep understanding of data architecture, schema design, job orchestration, and data governance.
• Strong knowledge of CI/CD (Azure DevOps) and infrastructure-as-code for data pipelines.
• Optional: Exposure to GCP BigQuery, Dataproc, or Dataflow as secondary platforms.

Preferred Certifications:

• Databricks Certified Data Engineer Professional
• Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
• Microsoft Certified: Azure Solutions Architect Expert (AZ-305)
• Optional:Google Cloud Certified – Professional Data Engineer