Data Engineer

Azure Data Engineer - Assistant Manager

• Bachelor’s or higher degree in Computer Science or a related discipline; or equivalent (minimum 7 years work experience). 
• At least 4+ years of consulting or client service delivery experience on Azure
• At least 4+ years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases such as SQL server and data warehouse solutions such as Azure Synapse
• Extensive experience providing practical direction with using Azure Native services.
• Extensive hands-on experience implementing data ingestion, ETL and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Azure Logic App Synapse/DW, Azure SQL DB, Event Hub, IOT Hub, Azure Stream Analytics, Azure Analysis Service, Databricks, Cosmo DB etc.
• Minimum of 4+ years of hands-on experience in Azure and Big Data technologies such as Java, Python, SQL, ADLS/Blob, PySpark and SparkSQL, Databricks, HD Insight and live streaming technologies such as EventHub, Azure Stream Analytics etc.
• Well versed in DevSecOps and CI/CD deployments
• Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, etc.
• Minimum of 4+ years of RDBMS experience
• Experience in using Big Data File Formats and compression techniques.
• Experience working with Developer tools such as Azure DevOps, Visual Studio Team Server, Git, Jenkins, etc.
• Experience with private and public cloud architectures, pros/cons, and migration considerations.

A Databricks Data Engineer is responsible for designing, building, and maintaining data solutions that leverage Databricks' powerful capabilities on the cloud platform. Key responsibilities include:

 

• Designing and deploying data pipelines that efficiently process vast amounts of data using Databricks, optimizing for performance and scalability.
• Collaborating with data scientists and business stakeholders to understand requirements and provide data solutions tailored to those needs.
• Implementing robust data models and architectures utilizing Databricks, ensuring data quality and integrity.
• Monitoring and troubleshooting data pipelines, proactively identifying and resolving issues to ensure smooth data flows.
• Ensuring data security, governance, and privacy measures are in place, adhering to industry best practices.
• Optimizing data storage and processing costs while maintaining high performance.
• Engaging in continuous improvement activities, researching new technologies and methodologies to enhance data solutions.

Experience Level

Mid Level