Lead Software Engineer - Python, ETL, AWS Glue, Spark

Lead Software Engineer – Python, ETL, GLUE, Spark, AWS.

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorganChase within the consume and community banking- JPMWM you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.

Job responsibilities

 

• Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
• Develops secure high-quality production code, and reviews and debugs code written by others
• Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems
• Leads evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture
• Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies
• Adds to team culture of diversity, opportunity, inclusion, and respect

 

 

Required qualifications, capabilities, and skills

 

• Formal training or certification on software engineering concepts and 3+ years applied experience
• Design and implement end-to-end ETL/ELT pipelines on AWS using AWS Glue (PySpark) and/or EMR/EMR Serverless.
• Build and maintain data lakes/lakehouses on Amazon S3 using columnar formats (Parquet/ORC) with effective partitioning, compaction, and schema evolution via Glue Data Catalog.
• Orchestrate workflows with AWS Step Functions and/or Airflow (MWAA), including error handling, retries, and idempotency.
• Optimize Spark jobs using best practices (predicate pushdown, broadcast joins, AQE, file sizing, caching) and monitor performance with CloudWatch and Spark UI.
• Implement data quality checks and observability , SLAs/SLOs, and lineage/metadata practices & y, scalability, and cost; produce clear documentation and runbooks.
• Enforce security and compliance: IAM least-privilege, KMS encryption, VPC endpoints/private networking, Secrets Manager/Parameter Store, Lake Formation access controls.
• Collaborate on data modeling (dimensional/star, data vault, wide tables) and interface contracts for analytics, ML features, and downstream services.
• Drive CI/CD for data code , infrastructure as code (Terraform or AWS CDK), and automated deployments.
• Participate in architectural reviews, propose improvements for reliability, scalability, and cost; produce clear documentation and runbooks.
• Mentor engineers, conduct code/design reviews, and promote engineering standards.
• Mentor engineers, conduct code/design reviews, and promote engineering standards.

Experience Level

Senior Level