Technical Specialist - Data Engineering

Technical Specialist-Data Engg

Area(s) of responsibility

Experience:

• 4+ years of experience in data engineering or distributed systems development.
• Strong programming skills in Python and PySpark.
• Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks SQL.
• Knowledge of big data frameworks like Hadoop, Spark, and Kafka.

 

Data Engineering and Processing:

• Develop and manage data pipelines using PySpark on Databricks.
• Implement ETL/ELT processes to process structured and unstructured data at scale.
• Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.

Databricks Platform Expertise:

• Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL).
• Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.
• Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.s.

Experience Level

Mid Level