Senior Software Engineer - Big Data and Streaming Platforms

Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)Job Posting Title: Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)
________________________________________
Who is Mastercard?
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive.
________________________________________
Job Summary:

We are looking for a highly skilled Senior QA Engineer / SDET with deep expertise in testing large scale batch and real time data platforms built on Apache Spark, Kafka, and Apache NiFi.
This role focuses on validating streaming heavy architectures, ensuring data correctness, latency, fault tolerance, and reliability across complex distributed systems leveraging object storage platforms such as Apache Ozone and Ceph.

The ideal candidate is passionate about data quality, test automation, observability, and production readiness for mission critical big data systems.


# Key Responsibilities

- Analyze business, regulatory, and technical requirements to design risk based and audit compliant test scenarios.

- Develop, maintain, and execute test plans, test cases, and test data with appropriate documentation and traceability to requirements.

- Perform functional, regression, and system integration testing in accordance with enterprise quality standards and release controls.

- Log, track, and manage defects in JIRA, ensuring clear reproduction steps, impact assessment, and lifecycle traceability.

- Validate JSON payloads, cloud storage data, and Apache NiFi data flows to ensure data accuracy, completeness, and integrity.

- Collaborate with development teams to support timely defect resolution and adherence to quality gates.

- Participate in design reviews, test plan reviews, and technical documentation

- Participate in Agile ceremonies, providing clear and accurate test execution status, risk assessments, and quality metrics.

- Test Kafka based streaming pipelines, including topic partitioning, offsets, throughput, and ordering guarantees
- Verify event replay and reprocessing flows using Kafka offsets and Spark checkpoints
- Test Apache NiFi flows for ingestion, routing, back pressure handling, error paths, and retry semantics
- Execute performance, load, and soak testing for Spark batch and streaming workloads
- Collaborate with engineering teams to validate partitioning strategies, resource tuning, and scalability assumptions

- Build and maintain test automation frameworks for data validation, streaming verification, and regression testing
- Automate validation using SQL, PySpark, and Python based test harnesses
- Integrate automated tests into CI/CD pipelines for data applications

- Participate in production validation, go live readiness, and post deployment verification
- Support incident analysis through data validation, replay testing, and root cause analysis
- Validate observability metrics such as lag, throughput, latency, error rates, and data drift


# Required Qualifications

- Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience
- Strong hands on experience of 6+ years in testing big data platforms in production environments
- Solid understanding of Apache Spark (batch and streaming) from a testing and validation perspective
- Working knowledge of Apache NiFi, JSON, and cloud based data validation.
- Strong analytical, problem solving, and communication skills, with the ability to work effectively across cross functional teams.
- Strong SQL skills for data validation and reconciliation
- Proficiency with Linux, shell scripting, Git, and basic CI/CD concepts
- Proficiency in JIRA for defect tracking, reporting, and quality metrics

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;

• Ensure the confidentiality and integrity of the information being accessed;

• Report any suspected information security violation or breach, and

• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

Experience Level

Senior Level