Data Engineer

Data Engineer

Role Summary
NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.
The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads.
This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines.
Experience
10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark.
________________________________________
Key Responsibilities
Migration Framework Engineering
•    Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code.
•    Build reusable Hive metadata/schema discovery and migration-configuration components.
•    Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history.
•    Implement PNC-defined Data Movement Controls including:
o    checkpointing,
o    restart/recovery,
o    idempotency,
o    retries,
o    control totals,
o    audit logging,
o    exception handling.
PB-Scale Migration Execution
•    Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale.
•    Design migration patterns for small, medium, large and very-large tables.
•    Tune parallel Spark execution across tables, partitions and chunks.
•    Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables.
•    Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks.
•    Support migration-wave execution and production issue resolution.
Cyber / Tokenization Integration
•    Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines.
•    Implement secure processing of sensitive fields.
•    Handle service failures, retries, throttling and restart scenarios.
•    Measure and optimize tokenization impact on end-to-end migration throughput.
•    Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules.
Apache Iceberg Engineering
•    Create/configure Iceberg tables using approved target standards.
•    Implement appropriate partitioning and file-sizing strategies.
•    Optimize compaction and small-file handling.
•    Work with Iceberg snapshots, schema evolution and metadata.
•    Troubleshoot Iceberg write/commit/catalog issues.
•    Support table maintenance and performance optimization.
Data Validation & Reconciliation
•    Develop automated source-target validation covering:
o    row counts,
o    schema conformity,
o    partition completeness,
o    control totals,
o    aggregates/checksums,
o    null/duplicate checks,
o    tokenization validation where applicable.
•    Isolate reconciliation failures at table/partition level.
•    Implement selective retry/reprocessing of failed data.
•    Produce auditable migration evidence.
Production Engineering
•    Monitor and troubleshoot distributed Spark jobs.
•    Analyze Spark UI, executor logs and resource utilization.
•    Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues.
•    Build operational telemetry covering migration status, throughput, retries and failures.
•    Participate in code reviews, testing, CI/CD and engineering standards.
•    Mentor mid-level/junior Data Engineers.
________________________________________
Mandatory Skills
Strong hands-on experience with:
•    Python
•    PySpark
•    Apache Spark / Spark SQL
•    Hadoop / HDFS
•    Hive / HiveQL / Hive Metastore
•    SQL
•    Apache Iceberg
•    Parquet
•    Linux
•    Git
•    distributed data processing
•    large-volume ETL/data migration
•    source-target reconciliation
•    Spark performance tuning.
Practical Iceberg experience should include:
•    table creation/read/write,
•    partitioning,
•    snapshots,
•    schema evolution,
•    file sizing,
•    compaction,
•    Spark integration.
Strong understanding of:
•    parallelism,
•    checkpoint/restart,
•    idempotency,
•    retry/recovery,
•    large-table processing,
•    small-file problems,
•    distributed-system troubleshooting.
________________________________________
Preferred Skills
•    Apache Polaris/Nessie/Glue or another Iceberg catalog.
•    Kubernetes/OpenShift/Spark Operator.
•    S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred.
•    TaaS/Protegrity or equivalent tokenization/masking technology.
•    PostgreSQL for migration configuration/control/audit metadata.
•    Trino.
•    Control-M/Airflow or equivalent enterprise scheduler.
•    CI/CD for Python/Spark applications.
•    Banking/financial-services experience.
•    PB-scale migration experience.
•    HiveQL→Spark SQL modernization experience.
•    Bronze/Silver/Gold lakehouse experience.

Experience Level

Senior Level