Data Engineer and Technology Lead - Scala & PySpark

IT Delivery

• Primary skills:Domain->Finacle-Core-Functional->Finacle-Core-WMS->Grand Master,Technology->Java->Apache • Knowledge of more than one technology • Basics of Architecture and Design fundamentals • Knowledge of Testing tools • Knowledge of agile methodologies • Understanding of Project life cycle activities on development and maintenance projects • Understanding of one or more Estimation methodologies, Knowledge of Quality processes • Basics of business domain to understand the business requirements • Analytical abilities, Strong Technical Skills, Good communication skills • Good understanding of the technology and domain • Ability to demonstrate a sound understanding of software quality assurance principles, SOLID design principles and modelling methods • Awareness of latest technologies and trends • Excellent problem solving, analytical and debugging skills Key Responsibilities: • Design, develop, and maintain scalable batch and near real‑time data pipelines using Scala and PySpark on distributed data platforms. • Lead end‑to‑end implementation of data engineering solutions, from requirement analysis and design to deployment and support. • Optimize Spark jobs for performance, reliability, and cost efficiency, including partitioning, caching, and resource tuning. • Implement robust data quality checks, validation frameworks, and monitoring to ensure accuracy and completeness of data. • Collaborate with architects and stakeholders to define data models, data flows, and integration patterns aligned with business needs. • Establish and enforce coding standards, best practices, and review processes for Scala and PySpark development. • Mentor junior engineers, provide technical guidance, and foster a culture of knowledge sharing and continuous improvement. • Troubleshoot complex production issues, perform root cause analysis, and implement long‑term preventive solutions. • Contribute to roadmap planning, effort estimation, and prioritization of data engineering initiatives as a technology lead.