Senior Software Engineer

Senior Software Engineer

JOB DESCRIPTION - SENIOR DATABRICKS ENGINEER

Azure Databricks | Spark & PySpark | Delta Lake | Lakeflow | Unity Catalog

Role Summary

Seeking a hands-on Senior Databricks Engineer with 8-10 years of data engineering experience and strong expertise in Azure Databricks. The role will design, build, test, optimize and operationalize scalable batch and streaming pipelines, Delta Lake data products and governed Lakehouse solutions for analytics, BI and AI workloads, supported by Insurance and Benefits Management domain experience.

Key Responsibilities

• Develop and optimize ETL/ELT pipelines using Azure Databricks, Spark, PySpark, Spark SQL and Python.
• Build Bronze, Silver and Gold data layers using Delta Lake, schema enforcement, incremental processing and reusable engineering patterns.
• Implement ingestion from APIs, databases, files, cloud storage and streaming sources using Auto Loader, Lakeflow and Azure orchestration services.
• Develop batch and near-real-time pipelines using Spark Structured Streaming, checkpoints, watermarking and resilient recovery patterns.
• Create curated tables, views and data products for Databricks SQL, Power BI, Genie, analytics and downstream AI/ML consumption.
• Apply Unity Catalog controls, lineage, data classification, access policies, masking and employer-level data isolation.
• Implement data-quality rules, validation, reconciliation, error handling, audit controls and operational observability.
• Optimize Spark jobs, Delta tables, queries, partitioning, file sizes, caching, Photon workloads, compute utilization and cloud cost.
• Build automated tests and CI/CD pipelines for notebooks, jobs, Lakeflow pipelines and data assets using Git and Azure DevOps.
• Participate in design and code reviews, troubleshoot production issues, maintain technical documentation and mentor junior engineers.

Mandatory Technical Skills

Microsoft Azure; Azure Databricks; Apache Spark; PySpark; Spark SQL; Python; advanced SQL; Delta Lake; Lakehouse and Bronze/Silver/Gold architecture; Auto Loader; Lakeflow Connect, pipelines and jobs; Spark Structured Streaming; Databricks Workflows; Databricks SQL and Photon; Unity Catalog; ADLS Gen2; Azure Data Factory or equivalent orchestration; REST APIs; JSON, CSV, Parquet and Avro; data modelling; ETL/ELT; data quality and reconciliation; Git; Azure DevOps or GitHub Actions; CI/CD; Terraform awareness; monitoring, logging, ing, security, performance tuning and cost optimization.

Experience and Qualifications

• 8-10 years of experience in data engineering, big-data development, ETL/ELT, data warehousing or cloud analytics.
• At least 4 years of strong hands-on experience developing production solutions on Azure Databricks or an equivalent Databricks environment.
• Proven experience building and supporting scalable Spark and Delta Lake pipelines for structured and semi-structured data.
• Experience with Unity Catalog, data lineage, access controls, data-quality frameworks and production deployment practices.
• Experience working with data architects, modellers, business analysts, QA, DevOps, security and BI teams in Agile delivery.
• Bachelor’s degree in Computer Science, Information Technology, Data Engineering or a related discipline; Databricks or Azure certification is preferred.

Domain Experience

• Insurance or employee-benefits experience involving employer, client, plan, member, enrollment, carrier, claims or renewal data is preferred.
• Experience handling privacy-sensitive and regulated information with traceability, auditability and role-based access controls.
• Familiarity with EDI, HRIS, benefits-administration, CRM or document-derived data is advantageous.

Core Competencies

Hands-on engineering depth, structured problem-solving, production ownership, data-quality discipline, performance tuning, collaboration, mentoring and the ability to balance scalability, reliability, governance, maintainability and cost.