Senior Data Engineer

Primary Databricks AWS Pyspark Python

Secondary Snowflake

5 to 8 years of IT experience and at least 4 years of experience in Databricks and have experience on latest features Delta Lake Unity Catalog

Should be able to work with end users to understand the problem they are facing while using Databricks and provide solutions

Should be able to research latest Databricks features do POCs for use cases and create best practice documents for end users

Should have better understanding and experience on AWS cloud services S3 Lambda SNS Ste Function Glue

Should have experience on job cost optimization techniques and minimize the run time

Need to provide instance support on Spark related issues

Should involve in code review process and provide recommendations

Databricks COE experience is added advantage

Should have experience on AWS cloud services S3 Lambda SNS Step functions Glue

Should have good knowledge and experience in working with Python core libraires

Good analytical and logical skills in writing SQLs Stored Procedures and creating MartsViews in any RDBMS

Good knowledge on ETL concepts and end to end ETL landscape starting from source system warehouse mart and reports

Should have experience on CICD for code deployment Terraform or others

Should have good experience with GitHub or any equivalent VCS

Should have experience on Autosys or any other scheduling tool

Should have knowledge on Agile methodology and principles

Should be able to understand requirements and create design and solution docs

Should have experience in handling mid size teams and managing work deliverables

Should be able to mentor junior members and guide them on the development

Demonstrates ability to break down a problem to manageable pieces and implement effective timely solutions

Should take ownership of assigned work and ready to work on agile manner

Strong communication and presentation skills