Pyspark

Pyspark

Good to have skills: SQL, Hadoop, Hive, Kafka, Airflow Preferred Qualifications: • Experience optimizing Spark workloads (tuning partitions, managing skew, memory/executor settings) for performance and cost efficiency. • Exposure to building end-to-end data pipelines with strong data quality checks and automated validations. • Familiarity with distributed processing patterns and designing reusable PySpark modules for scalable development. • Experience collaborating in agile teams, participating in code reviews, and improving engineering standards for data pipelines. Key Responsibilities: • Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and/or incremental processing. • Build and optimize Apache Spark jobs with focus on performance, partitioning strategy, caching, and efficient transformations/actions. • Perform data cleansing, validation, and reconciliation to ensure accuracy, completeness, and consistency of datasets. • Collaborate with cross-functional teams to understand requirements and translate them into robust data processing solutions. • Troubleshoot pipeline failures, analyze logs, identify bottlenecks, and implement fixes to improve reliability and throughput. • Write clean, maintainable code with reusable components and clear documentation for pipelines and data flows. • Support deployment and operationalization of Spark workloads, including monitoring and basic production support activities. • Contribute to code reviews and follow engineering best practices to improve quality and maintainability. Minimum Qualifications: • Education: BTECH, MTECH, MCA, MSC. • 2–3 years of experience in data engineering or large-scale data processing roles. • Strong hands-on experience with PySpark for building data pipelines and transformations. • Working knowledge of Apache Spark concepts such as RDD/DataFrame, joins, shuffles, and performance considerations. • Ability to debug Spark applications and resolve data/job issues effectively.