Technical Lead - Azure Databricks & Python Development

Tech Lead

Job Description

Your Responsibilities:

• Python Programming: Lead the development of robust, scalable, and efficient Python code for data processing, transformation, and analysis within the Databricks environment. Ensure adherence to coding standards, quality, and performance best practices.
• Azure Databricks Administration: Oversee Databricks platform configuration, resource management, cluster optimization, and monitoring to ensure high availability and performance. Implement best practices for managing Databricks workspaces, libraries, and notebooks.
• CI/CD for Databricks Artifacts: Design and implement CI/CD pipelines for automating the deployment of Databricks artifacts such as notebooks, libraries, jobs, and Delta tables. Use tools such as Azure DevOps, GitHub Actions, or Jenkins to streamline the deployment process.
• Delta Live Tables: Design and manage Delta Live Tables pipelines for real-time data processing, ensuring data quality, reliability, and optimal performance in the processing layers.
• Auto Loader: Implement and optimize Auto Loader for scalable, fault-tolerant ingestion of data from cloud storage sources into Databricks.
• Unity Catalog: Manage and configure Unity Catalog for centralized governance of data across workspaces, ensuring proper access controls, data lineage, and compliance.
• Databricks Asset Bundles: Leverage Databricks Asset Bundles to manage and share reusable components, including notebooks, libraries, and ML models across teams or projects.
• Team Leadership: Mentor and guide junior team members, fostering a collaborative environment that encourages knowledge sharing, innovation, and personal growth. Provide technical leadership and support to ensure the success of the team’s initiatives.
• Collaboration and Communication: Work closely with stakeholders, including data engineers, data scientists, and business analysts, to understand requirements and deliver end-to-end solutions. Communicate technical challenges and solutions clearly to both technical and non-technical stakeholders.

Your Profile:

• Experience in Python Programming:
  • Minimum of 7 years of experience in Python programming, with a focus on Generative AI, data engineering and data science workflows.
  • Expertise in designing and building scalable Python-based ETL pipelines, using libraries such as Pandas, NumPy, PySpark, and requests.
  • Proficiency in debugging, optimizing, and maintaining Python code to ensure high performance and reliability in a distributed environment.
• Extensive Experience with Azure Databricks:
  • Minimum of 7 years of hands-on experience working with Azure Databricks in a production environment.
  • Strong knowledge of Databricks clusters, workspaces, and the Databricks runtime.
  • Ability to optimize and troubleshoot Spark-based jobs and notebooks, ensuring performance and cost-efficiency in cloud environments.
• CI/CD for Databricks Artifacts:
  • Proven experience in setting up and managing Continuous Integration/Continuous Deployment (CI/CD) pipelines for Databricks artifacts (notebooks, libraries, and jobs).
  • Familiarity with version control systems (Git), and experience with tools such as Azure DevOps, GitHub Actions, or Jenkins for automated deployment.
  • Experience with integrating Databricks with external version control repositories (e.g., GitHub, Azure Repos) for collaboration and artifact management.

City

Bengaluru

State

Karnataka

Location Country

India

• Delta Live Tables:
  • Strong experience in designing, managing, and optimizing Delta Live Tables (DLT) pipelines to support real-time, reliable, and scalable data processing.
  • Ability to handle complex data transformations, ensure data quality, and monitor pipeline performance within the Delta Lake architecture.
  • Expertise in implementing automated data quality checks and validation within Delta Live Tables pipelines.
• Auto Loader:
  • Practical experience with the Auto Loader feature in Databricks for real-time, fault-tolerant, and scalable data ingestion from cloud storage (e.g., Azure Blob Storage, ADLS Gen2).
  • Knowledge of best practices for optimizing Auto Loader performance and ensuring minimal latency in large-scale data ingestion tasks.
• Unity Catalog for Data Governance:
  • In-depth experience in implementing and managing Unity Catalog for central data governance in Databricks.
  • Proven track record in setting up and enforcing fine-grained access control and managing permissions for users, groups, and roles.
  • Understanding of how to implement data lineage and ensure compliance with data privacy and security standards in a multi-workspace environment.
• Databricks Asset Bundles:
  • Familiarity with Databricks Asset Bundles for managing reusable components, such as notebooks, libraries, and models.
  • Experience in defining, packaging, and deploying asset bundles across different environments and teams.
• Databricks Workflows:
  • Expertise in creating, managing, and optimizing Databricks Workflows to automate data pipelines, batch processes, and machine learning model training.
  • Experience with designing multi-step workflows, task dependencies, and orchestration of tasks across Databricks jobs.
  • Knowledge of workflow monitoring, alerting, and troubleshooting to ensure reliable execution of data pipelines.
• Compute Policies:
  • Experience in setting up and managing Compute Policies within Databricks to control cost, performance, and resource allocation across different clusters and workspaces.
  • Ability to define and enforce best practices for cluster configurations, including configuring policies for autoscaling, cluster termination, and resource usage limits.
  • Familiarity with Databricks’ cost management features and using compute policies to optimize resource allocation and prevent over-provisioning in cloud environments.
• Team Leadership and Collaboration:
  • Strong leadership experience, including mentoring and coaching junior engineers, fostering a collaborative and knowledge-sharing environment.
  • Ability to lead teams in agile development processes and deliver projects on time while maintaining high technical standards.
  • Excellent interpersonal and communication skills, with the ability to manage stakeholder relationships and bridge the gap between technical teams and business users.
  • Proven ability to drive architectural decisions, facilitate technical discussions, and provide guidance on best practices for the development team.
• Advanced Problem Solving and Debugging:
  • Strong troubleshooting skills in distributed data environments, with the ability to identify and resolve performance bottlenecks, errors, and resource contention in Databricks notebooks, jobs, and Spark applications.
  • Ability to effectively analyze and resolve issues related to data ingestion, transformation, and storage in a cloud-based data pipeline.
• Cloud and Data Architecture Experience:
  • Solid understanding of cloud-based data architectures, particularly on Azure, including services like Azure Data Lake Storage (ADLS), Azure Blob Storage, Azure SQL, and Azure Synapse.
  • Experience with data lake design, schema evolution, data partitioning strategies, and performance tuning within the Azure ecosystem.
• Data Security and Compliance:
  • Knowledge of data security best practices, including encryption, access control, and auditing within the Azure cloud environment.

About ADM

At ADM, we unlock the power of nature to provide access to nutrition worldwide. With industry-advancing innovations, a complete portfolio of ingredients and solutions to meet any taste, and a commitment to sustainability, we give customers an edge in solving the nutritional challenges of today and tomorrow. We’re a global leader in human and animal nutrition and the world’s premier agricultural origination and processing company. Our breadth, depth, insights, facilities and logistical expertise give us unparalleled capabilities to meet needs for food, beverages, health and wellness, and more. From the seed of the idea to the outcome of the solution, we enrich the quality of life the world over. Learn more at www.adm.com.

Experience Level

Senior Level