Databricks and Snowflake Developer

Databricks & Snowflake Developer – Job Description

Job Summary:
We are seeking an experienced Databricks & Snowflake Developer with strong expertise in PySpark, Delta Lake, Snowflake SQL, and Azure Data Engineering services. The ideal candidate will design, develop, and optimize large-scale data pipelines across both Databricks and Snowflake platforms to support analytics, reporting, and enterprise data initiatives.
This role requires close collaboration with data engineers, analysts, architects, and business stakeholders to deliver scalable, high quality, and cost efficient cloud data solutions.

Key Responsibilities
1. Databricks Development
• Develop, optimize, and maintain Databricks Notebooks, Workflows, and Jobs.
• Build scalable PySpark applications and data transformations.
• Implement and manage Delta Lake tables, including schema evolution and performance tuning.
• Use Git-based version control for notebook and code management.

2. Snowflake Development
• Develop and optimize Snowflake SQL, including complex joins, window functions, and CTEs.
• Build and manage Snowflake pipelines using Stages, File Formats, COPY, and Snow pipe.
• Implement Streams, Tasks, Materialized Views for automated ELT workflows.
• Manage Snowflake performance tuning, virtual warehouse optimization, and cost governance.

3. Data Pipeline & ETL/ELT Engineering
• Design and build data pipelines integrating Databricks, Snowflake, and Azure Data Factory.
• Implement ETL/ELT flows using Spark, SQL, Snowflake pipelines, and ADF/Synapse.
• Optimize pipelines for scalability, performance, and cost efficiency.

4. Data Integration
• Integrate data from relational databases, APIs, file systems, and cloud platforms into Databricks and Snowflake.
• Ensure alignment with enterprise data architecture and modeling standards.

5. Documentation & Compliance
• Document all pipelines, workflows, data models, and architecture across Databricks and Snowflake.
• Ensure compliance with governance, security, and regulatory standards.

6. Collaboration
• Work with business stakeholders to gather data requirements and convert them into technical solutions.
• Support BI, analytics, and downstream consumers across both platforms.

7. Troubleshooting & Support
• Debug Databricks issues including notebook failures, Spark job errors, and ADF pipeline failures.
• Troubleshoot Snowflake execution errors, warehouse performance problems, and ingestion failures.
• Perform root cause analysis and implement preventive measures.

Qualifications
Education
• Bachelor's degree in Computer Science, Information Technology, or related field (or equivalent experience).

Experience
• 4+ years of experience in data engineering or big data development.
• 3+ years hands on with Databricks, PySpark, and Azure Data Engineering.
• 3+ years hands on with Snowflake, Snowflake SQL, and ELT development.
(Databricks components)

Technical Skills
Databricks
• Strong hands-on experience with Databricks, PySpark, Delta Lake, Spark SQL.
• Experience with ADF, Synapse, and ADLS integrations.

Snowflake
• Expert in Snowflake SQL, performance tuning, and warehousing.
• Experience with Snowpipe, Streams, Tasks, and orchestration via ADF/Synapse.
• Knowledge of RBAC, masking policies, and security best practices.
General
• Strong understanding of data warehousing concepts and cloud data architecture.
• Familiarity with CI/CD (Azure DevOps/GitHub Actions) and Git workflows.

Soft Skills
• Strong analytical and problem solving abilities.
• Excellent communication and stakeholder management.
• Ability to work independently and collaboratively in agile teams.

Company Culture
Our company values innovation, teamwork, and continuous learning. We encourage creativity and provide opportunities for professional growth.