Spark

Spark

• Primary skills:Technology->Big Data - Data Processing->Spark Minimum Qualifications: • Bachelor’s degree (or equivalent) in Engineering/Technology/Computer Science or related field (BTech/BE/MSc or equivalent). • 3–5 years of experience working on data engineering or big data processing initiatives. • Hands-on experience building and maintaining Spark-based data processing solutions. • Strong understanding of distributed computing concepts and data processing fundamentals. • Ability to work independently on assigned modules and collaborate effectively within a team. Preferred Qualifications: • Master’s degree (MTech/MCA or equivalent) in a relevant discipline. • Proven experience delivering end-to-end Spark pipelines, including development, testing, and production support. • Experience improving job performance and stability through Spark tuning and structured troubleshooting practices. • Familiarity with building reusable frameworks/components to standardize Spark development across projects. • Strong communication skills to explain technical trade-offs and align solutions with stakeholder expectations. Good to have skills: Hadoop, Hive, Kafka, Airflow, Delta Lake Key Responsibilities: • Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads. • Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks. • Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans. • Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables. • Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting. • Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards. • Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.