Senior Lead Infrastructure Cloud Engineer - Kafka Platform

Sr. Lead, Infra Cloud - Kafka

About Northern Trust:

Northern Trust, a Fortune 500 company, is a globally recognized, award-winning financial institution that has been in continuous operation since 1889.

Northern Trust is proud to provide innovative financial services and guidance to the world’s most successful individuals, families, and institutions by remaining true to our enduring principles of service, expertise, and integrity. With more than 130 years of financial experience and over 22,000 partners, we serve the world’s most sophisticated clients using leading technology and exceptional service.

Main Duties:  

Northern Trust Enterprise Middleware Services is seeking a highly experienced Sr. Lead Confluent Kafka Platform Engineer to join the team on a full-time basis. This role will lead the strategy and execution of a modern Kafka platform, ensuring scalability, security, and high availability across hybrid environments. The individual will drive automation, enable self-service capabilities, and accelerate cloud adoption to simplify operations and improve onboarding efficiency. Working closely with engineering teams, this position will also play a key role in advancing real-time data streaming capabilities while strengthening platform reliability, governance, and overall engineering maturity.

Responsibilities

• Lead the design, implementation, and optimization of enterprise Kafka platforms across on-prem and cloud environments.
• Architect and manage hybrid Kafka solutions integrating Confluent Platform and Confluent Cloud.
• Ensure high availability, scalability, and resiliency through multi-region and multi-zone deployments.
• Drive platform modernization including migration from on-prem Kafka to cloud-native streaming platforms.
• Implement cluster linking and data replication strategies for hybrid and DR use cases.
• Develop and maintain automation frameworks for Kafka provisioning, configuration, and lifecycle management.
• Implement no-touch / low-touch Kafka operations using Infrastructure as Code and CI/CD pipelines.
• Design and deliver self-service and web-based platforms for Kafka onboarding and management.
• Collaborate with application teams to design event-driven architectures and real-time streaming solutions.
• Plan and execute Kafka platform upgrades ensuring minimal downtime and operational risk.
• Implement monitoring, alerting, and observability frameworks for platform health and reliability.
• Perform quarterly capacity planning, performance tuning, and scaling of Kafka clusters.
• Enforce enterprise security standards and governance for Kafka platforms.
• Provide L3 support and perform root cause analysis for production incidents.
• Lead and mentor Kafka engineering teams and act as a technical SME.

SKILLS & EXPERIENCE  

Qualifications:

• Bachelor’s degree in Computer Science, Information Systems, or related field
• 8–12+ years of experience in Middleware / Messaging systems.
• Strong focus on Apache Kafka / Confluent Kafka administration and engineering.
• Experience working in enterprise-scale, production-critical environments.
• Proven leadership experience in managing technical teams or platforms.

Strong Experience In:

• Building self-service platforms or developer portals for infrastructure provisioning
• CI/CD pipelines and GitOps-based infrastructure management
• Observability tools like Prometheus, Grafana, or enterprise monitoring platforms
• Kafka upgrades, migration, and platform lifecycle management
• Working with multi-region DR architectures and failover strategies
• Supporting mission-critical production systems with strict SLAs

Solid experience In:

• Kafka ecosystem components such as Kafka Connect, Kafka Streams, Schema Registry
• Monitoring and observability tools (Prometheus, Grafana, Dynatrace).
• Kafka security implementation (SSL/mTLS, SASL, ACLs, RBAC).
• Managing production incidents and performing root cause analysis.
• Working with application teams for Kafka onboarding and integration.
• Implementing self-service and automation platforms.

Strong Knowledge Of:

• Distributed systems design and event-driven architecture patterns
• Streaming architectures, data pipelines, and integration patterns
• High availability, fault tolerance, and replication strategies
• Kafka internals (partitioning, replication, leader election, ISR, retention)
• Security frameworks including mTLS, RBAC, ACLs, authentication/authorization
• Cloud networking concepts (private endpoints, VNet, DNS, secure connectivity)

Experience in:

• Migrating applications from on-prem Kafka to cloud-based Kafka platforms
• Building automation-first infrastructure platforms with minimal manual operations
• Designing self-service onboarding workflows for developers
• Implementing Kafka governance including topic lifecycle, retention, and access controls
• Working in audit-compliant and regulated environments
• Collaborating across cloud, networking, security, and application teams

Ability To:

• Design scalable, resilient, and future-ready Kafka platforms.
• Drive platform standardization and engineering maturity across teams.
• Automate repetitive operational tasks and reduce manual effort.
• Quickly diagnose and resolve complex system issues.
• Influence architecture decisions and guide engineering teams.
• Communicate effectively with both technical and non-technical stakeholders.

Demonstrated ability to:

• Lead Kafka platform initiatives end-to-end with strong ownership and accountability.
• Mentor engineers and build high-performing technical teams.
• Drive large-scale transformation initiatives such as cloud migration and automation adoption.
• Deliver measurable improvements in performance, reliability, and operational efficiency.
• Establish Kafka as a core enterprise data streaming platform.

Strong mindset for:

• Automation-first and platform engineering thinking
• Continuous improvement and modernization of legacy systems
• Building scalable, reusable, and developer-friendly platforms
• Proactive problem-solving and operational excellence
• Driving innovation in hybrid cloud and streaming technologies

Nice to Have

• Experience with Confluent Cloud or managed Kafka services
• Exposure to hybrid cloud Kafka architectures (on-prem + cloud integration)
• Experience with Kafka Streams, KSQLDB, or real-time stream processing
• Experience building self-service or web-based Kafka platforms
• Knowledge of DevOps practices, GitHub Actions, or cloud-native automation
• Experience with other messaging systems (IBM MQ, Solace, etc.)
• Exposure to multi-region DR architecture and cluster linking
• Experience working with cloud networking (Private Link, DNS, VNet design)
• Familiarity with data governance, data lineage, and schema validation tools

Northern Trust Corporation (Nasdaq: NTRS) is a leading provider of wealth management, asset servicing, asset management and banking to corporations, institutions, affluent families and individuals. Founded in Chicago in 1889, Northern Trust has offices in the United States in 19 states and Washington, D.C., and 20 international locations in Canada, Europe, the Middle East and the Asia-Pacific region. 

Working with Us:

As a Northern Trust partner, greater achievements await. You will be part of a flexible and collaborative work culture in an organization where financial strength and stability is an asset that emboldens us to explore new ideas.

Movement within the organization is encouraged, senior leaders are accessible, and you can take pride in working for a company committed to assisting the communities we serve! Join a workplace with a greater purpose.

We’d love to learn more about how your interests and experience could be a fit with one of the world’s most admired and sustainable companies! Build your career with us and apply today. #MadeForGreater

Reasonable accommodation

Northern Trust is committed to working with and providing reasonable accommodations to individuals with disabilities. If you need a reasonable accommodation for any part of the employment process, please email our HR Service Center at MyHRHelp@ntrs.com.

We hope you’re excited about the role and the opportunity to work with us. We value an inclusive workplace and understand flexibility means different things to different people.

Apply today and talk to us about your flexible working requirements and together we can achieve greater.

About Our Pune Office

The Northern Trust Pune office, established in 2016, is now home to over 3,000 employees. The office handles various functions, including Operations for Asset Servicing and Wealth Management, as well as delivering critical technology solutions that support business operations across the globe.

Our Pune team takes our commitment to service to heart. In 2024, they volunteered more than 10,000+ hours into the communities where they live and work. Learn more.

Experience Level

Senior Level