Lead Data Engineer

Lead Data Engineer

Job Title - Lead Data Engineer - Level 9 - ACS Song

Management Level: Level 9 – Lead/Specialist

Location: Kochi

Must have skills: Azure and Databricks

Good to have skills:

Experience: 5-8 years of experience is required

Educational Qualification: Graduation (Accurate educational details should capture)

Job Summary

We are seeking a Senior Data Engineer specializing in Azure and Databricks to design, build, and optimize scalable data engineering solutions across cloud-based and distributed data platforms. The role will focus on developing robust ETL/ELT pipelines, workflow orchestration, data processing frameworks, and backend automation using Python. The ideal candidate will bring strong hands-on experience with Databricks, Azure Functions, Apache Airflow, Kedro, Kafka, PostgreSQL, Docker, and modern software engineering practices (all must haves). This position requires close collaboration with architects, UI developers, and data engineering teams to deliver reliable, production-ready, and high-performing data solutions. 

Roles and Responsibilities

• Design and deliver scalable, maintainable, and production-grade data pipelines and data processing workflows using Azure, Databricks, Python, and related technologies. 
• Enable efficient data ingestion, transformation, orchestration, validation, and automation across batch and streaming data environments. 
• Support the development of modular, reusable, and well-documented data engineering components using frameworks such as Kedro and Apache Airflow. 
• Ensure high standards of code quality, performance, reliability, troubleshooting, and collaboration within cross-functional engineering teams. 
• Develop, maintain, and enhance scalable data pipelines, ETL/ELT workflows, and data processing solutions on Azure and Databricks. 
• Work extensively with Kedro for pipeline development, modularization, feature implementation, code changes, and bug fixes. 
• Build, maintain, and optimize workflow orchestration using Apache Airflow, including DAG creation, monitoring, troubleshooting, and issue resolution. 
• Develop and maintain Azure Functions using Python for backend processing, integrations, automation, and event-driven workflows. 
• Write clean, scalable, maintainable, and production-ready Python code, including use of Pydantic models, data validation, modular design, and API/backend integrations. 
• Design, optimize, and troubleshoot complex SQL queries and transformations, with hands-on work in PostgreSQL for database development and data management. 
• Work with Kafka and event-driven or streaming architectures for data ingestion, processing, and integration workflows. 
• Use Docker for containerization and support local development, testing, and deployment workflows. 
• Use GitHub for version control, pull requests, code reviews, branching strategies, and collaborative software development. 
• Collaborate with architects, UI developers, data engineers, and other stakeholders while participating in sprint planning, technical discussions, code reviews, documentation, testing, CI/CD, monitoring, and production support activities 

Professional and Technical Skills:

• Minimum 6 years of total professional experience in data engineering, cloud data platforms, backend data processing, or related software engineering roles. 
• Strong experience designing, developing, and supporting production-grade data pipelines and ETL/ELT workflows. 
• Hands-on experience working in enterprise or production environments with cloud-based data engineering solutions. 
• Proven ability to debug, troubleshoot, and resolve application issues, pipeline failures, DAG issues, and production bugs. 
• Strong programming experience in Python for data engineering, backend processing, pipeline development, and automation. 
• Good understanding of Python programming concepts, including modular code development, Pydantic models, data validation, backend integrations, and API-based workflows. 
• Hands-on production experience with Kedro for pipeline development, modularization, and maintainable data engineering design. 
• Strong hands-on experience with Apache Airflow, including DAG creation, workflow orchestration, scheduling, monitoring, and troubleshooting. 
• Experience with Databricks and distributed data processing environments. 
• Hands-on experience developing and maintaining Azure Functions using Python. 
• Strong working knowledge of the Microsoft Azure ecosystem and cloud-based data engineering services. 
• Experience with Kafka and event-driven or data streaming architectures. 
• Strong SQL skills with hands-on experience in PostgreSQL, including query optimization, data validation, and database development. 
• Hands-on experience with Docker and containerized development or deployment workflows. 
• Experience using GitHub for collaborative development, pull requests, code reviews, and version control.

Additional Information      

Behavioral and Collaboration Skills 

• Strong analytical, troubleshooting, and problem-solving skills. 

• Ability to work effectively in cross-functional engineering teams. 

• Strong communication skills (English) with the ability to participate in technical discussions and explain implementation approaches clearly. 

• Comfortable collaborating with architects, UI developers, data engineers, and other stakeholders. 

• Proactive mindset with ownership of assigned tasks, production issues, and continuous improvement initiatives. 

About Our Company | Accenture (do not remove the hyperlink)