Azure Databricks Data Engineer

Azure Databricks Data Engineer/Developer

Requirements: • Experience in Databricks, SQL, PySpark, Spark, Python, and Azure Data Factory (ADF). • Experience in designing, developing, and maintaining data pipelines and data streams. • Experience in moving/transforming data across layers (Bronze, Silver, Gold) using ADF, Python, and PySpark. • Experience in working with stakeholders to understand their data needs and provide solutions. • Experience in collaborating with other teams to ensure data quality and consistency. • Experience in developing and maintaining data models and data dictionaries. • Experience in developing and maintaining ETL processes. • Experience in developing and maintaining data quality checks. • Experience in developing and maintaining data governance policies and procedures. Experience in developing and maintaining data security policies and procedure Knowledge of more than one technology • Basics of Architecture and Design fundamentals • Knowledge of Testing tools • Knowledge of agile methodologies • Understanding of Project life cycle activities on development and maintenance projects • Understanding of one or more Estimation methodologies, Knowledge of Quality processes • Basics of business domain to understand the business requirements • Analytical abilities, Strong Technical Skills, Good communication skills • Good understanding of the technology and domain • Ability to demonstrate a sound understanding of software quality assurance principles, SOLID design principles and modelling methods • Awareness of latest technologies and trends • Excellent problem solving, analytical and debugging skills Responsibilities: • Design and build modern data pipelines and data streams. • Move/Transform data across layers (Bronze, Silver, Gold) using ADF, Python, and PySpark. • Develop and maintain data pipelines and data streams. • Work with stakeholders to understand their data needs and provide solutions. • Collaborate with other teams to ensure data quality and consistency. • Develop and maintain data models and data dictionaries. • Develop and maintain ETL processes. • Develop and maintain data quality checks. • Develop and maintain data governance policies and procedures. • Develop and maintain data security policies and procedures. • Provide technical guidance and mentorship to junior data engineers