Senior Data Engineer

Senior Data Engineer

Role Summary

NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves.

This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers.

 

Experience

10+ years of Data Engineering experience, including:

•             5+ years hands-on Apache Spark/PySpark.

•             Strong Hadoop/Hive experience.

•             Experience leading Data Engineering teams.

•             Large-scale data migration/data modernization experience.

•             Strong data modelling and source-to-target mapping experience.

Multi-hundred-TB/PB-scale migration experience is strongly preferred.

 

Key Responsibilities

 

Technical Leadership

•             Lead a team of Data Engineers through assessment, development, migration, validation and production execution.

•             Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.

•             Own technical estimates, engineering backlog, work allocation and technical delivery.

•             Conduct design and code reviews and enforce engineering standards.

•             Mentor Data Engineers and provide hands-on technical guidance.

•             Act as the primary engineering escalation point for complex migration and production issues.

•             Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.

•             Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions.

 

Hadoop-to-Iceberg Migration

•             Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.

•             Harden and extend the reusable, configuration-driven migration framework developed during the POC.

•             Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.

•             Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.

•             Design execution approaches for small, medium, large and very-large tables.

•             Optimize parallel migration across tables and partitions.

•             Implement checkpointing, restart/recovery, idempotency and safe retries.

•             Lead technical execution and troubleshooting of migration waves.

 

Data Modelling & Schema Engineering

•             Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.

•             Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.

•             Perform source-to-target data mapping and schema/data-type mapping.

•             Define and implement table structures consistent with approved Iceberg standards.

•             Analyze:

o             entities and relationships,

o             primary/foreign-key relationships,

o             data types,

o             partition keys,

o             business keys,

o             reference/cross-reference structures.

•             Identify schema incompatibilities and recommend remediation.

•             Work with the Data Architect on partitioning, file-layout and model-design decisions.

•             Ensure migrated models preserve required business relationships and downstream compatibility.

•             Support future Bronze/Silver/Gold modelling requirements where applicable.

Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards.

 

Spark/PySpark Engineering & Performance

•             Develop and review production-grade Python/PySpark components.

•             Tune Spark jobs for PB-scale migration.

•             Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.

•             Optimize table/partition/chunk parallelism.

•             Tune workloads to meet migration-throughput objectives.

•             Identify bottlenecks across:

Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation

•             Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework.

 

Apache Iceberg Engineering

•             Implement approved Iceberg table standards.

•             Develop/configure Iceberg tables and Spark→Iceberg processing.

•             Implement:

o             partitioning,

o             schema evolution,

o             snapshots,

o             file sizing,

o             compaction,

o             metadata maintenance,

o             small-file remediation.

•             Troubleshoot Iceberg catalog/write/commit issues.

•             Optimize Iceberg table performance and storage characteristics.

 

Cyber-3 / Tokenization

•             Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.

•             Integrate tokenization into Spark migration pipelines.

•             Implement batching, retry, timeout and recovery handling.

•             Troubleshoot tokenization failures/performance bottlenecks.

•             Measure impact of tokenization on end-to-end migration throughput.

•             Support Cyber/tokenization validation based on PNC-defined rules.

 

Migration Controls & Reconciliation

•             Lead implementation of PNC Data Movement Controls, including:

o             checkpointing,

o             idempotency,

o             control totals,

o             restart/recovery,

o             retry,

o             audit logging,

o             exception handling.

•             Build reusable source-to-target reconciliation capabilities covering:

o             row counts,

o             schema validation,

o             partition completeness,

o             control totals,

o             aggregates,

o             hash/checksum where appropriate,

o             null/duplicate validation,

o             tokenization validation.

•             Isolate discrepancies at table/partition level.

•             Enable selective reprocessing rather than full-table reruns.

•             Produce technical evidence required for migration acceptance.

 

Mandatory Skills

Strong hands-on expertise in:

•             Python

•             PySpark

•             Apache Spark / Spark SQL

•             Hadoop / HDFS

•             Hive / HiveQL / Hive Metastore

•             SQL

•             Apache Iceberg

•             Data modelling

•             Source-to-target mapping

•             Parquet

•             Linux

•             Git

•             Large-volume ETL/data migration

•             Data validation/reconciliation

•             Spark performance tuning

•             Distributed-system troubleshooting.

 

Data Modelling

Strong experience with:

•             logical and physical data modelling,

•             source-to-target mapping,

•             schema/data-type mapping,

•             entity relationships,

•             keys and referential relationships,

•             dimensional/relational modelling concepts,

•             large-scale analytical data models.

 

Technical Leadership

Demonstrated experience:

•             leading Data Engineering teams,

•             technical estimation/planning,

•             detailed solution design,

•             code/design reviews,

•             mentoring engineers,

•             production troubleshooting,

•             root-cause analysis,

•             communicating technical issues with architects/customer teams.

 

Preferred Skills

•             Apache Polaris/Nessie/Glue.

•             OpenShift/Kubernetes/Spark Operator.

•             S3-compatible object storage.

•             NetApp StorageGRID/ONTAP.

•             TaaS/Protegrity or equivalent tokenization technology.

•             Trino.

•             PostgreSQL.

•             Control-M/Airflow.

•             CI/CD for Spark/Python applications.

•             Banking/financial-services experience.

•             PB-scale data migration.

•             Bronze/Silver/Gold modelling and implementation.

•             HiveQL→Spark SQL modernization.

 

Experience Level

Senior Level