Databricks Data Engineer

IT Delivery

Technology->Cloud Platform->Databricks AI Services,Technology->Data Engineering->Databricks • Knowledge of more than one technology • Basics of Architecture and Design fundamentals • Knowledge of Testing tools • Knowledge of agile methodologies • Understanding of Project life cycle activities on development and maintenance projects • Understanding of one or more Estimation methodologies, Knowledge of Quality processes • Basics of business domain to understand the business requirements • Analytical abilities, Strong Technical Skills, Good communication skills • Good understanding of the technology and domain • Ability to demonstrate a sound understanding of software quality assurance principles, SOLID design principles and modelling methods • Awareness of latest technologies and trends • Excellent problem solving, analytical and debugging skills Key Responsibilities: • Lead the design and development of scalable data pipelines and ETL/ELT workflows using Databricks on Azure. • Architect and implement data lakehouse solutions, optimizing storage, compute, and query performance for large datasets. • Develop, optimize, and maintain Databricks notebooks, jobs, and workflows for batch and near real-time data processing. • Define and enforce coding standards, best practices, and reusable frameworks for Databricks-based data engineering. • Collaborate with data architects, analysts, and business teams to understand requirements and translate them into technical designs. • Implement robust data quality checks, validation rules, and monitoring to ensure reliability and trust in data products. • Optimize Spark jobs for performance and cost efficiency, including cluster sizing, caching strategies, and partitioning. • Ensure security, governance, and compliance of data solutions, including access control and data lineage within Databricks and Azure. • Troubleshoot complex data issues, perform root cause analysis, and drive long-term fixes and improvements. • Mentor junior engineers, provide technical guidance, conduct code reviews, and support knowledge sharing within the team. • Contribute to roadmap planning, effort estimation, and delivery management for data engineering initiatives.