Lead Software Engineer - Apache Spark, AWS

Job Description

We have an opportunity to impact your career and provide an adventure where you can push the limits of what’s possible.

As a Lead Software Engineer at JPMorgan Chase within the Engineer Services Platform (ESP) in the Chief Technology Office (CTO) , you are an integral part of an agile team that builds and operates a core capability in the firm’s artifact engineering ecosystem that enables secure, stable, and scalable consumption of open source software. As a core technical contributor, you will lead critical technology solutions that optimize open source software usage across the firm by strengthening governance controls and delivering actionable transparency through usage and compliance health metrics (per internal open source governance documentation).

Job responsibilities

• Design, develop, and troubleshoot platform services that govern open source acquisition, usage, and compliance at scale
• Design and implement scalable data platforms and pipelines (batch and streaming) with strong reliability and data quality controls.
• Build event-driven/CDC-based architectures to enable near real-time data movement and downstream consumption patterns.
• Develop and operate pipelines using technologies such as Spark, Kafka, and Airflow , focusing on maintainability and observability.
• Design distributed data systems with strong understanding of partitioning/sharding, scalability, and performance optimization .
• Build solutions on AWS and cloud-native data services; ensure security, governance, and cost-aware design.
• Optimize end-to-end performance (ingestion, processing, storage, query), including SLAs and operational readiness.
• Partner with stakeholders to define data contracts, integration patterns, and operational runbooks for production support.
• Contribute to engineering excellence via design reviews, code reviews, and best practices for pipeline testing and operational stability.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• Hands-on practical experience delivering system design, application development, testing, and operational stability for platform services used by many engineering teams.
• Advanced in one or more programming language(s), with demonstrated ability to build production services and integrations used in critical developer workflows.
• Proficiency in automation and continuous delivery methods, including designing and operating integrations that can enforce policy decisions in CI/CD pathways (per internal AMS integration expectations).
• Proficient in all aspects of the Software Development Life Cycle, with strong engineering hygiene around secure coding, peer review, observability, and incident response.
• Strong data engineering skills, including hands-on experience designing and building data pipelines and well-versed with the Medallion Architecture (Bronze, Silver, and Gold layers) to support reliable, scalable, and audit-ready data flows for compliance telemetry and open source health metrics.
• Advanced understanding of agile methodologies and modern engineering practices such as CI/CD , application resiliency, and security, particularly as they apply to supply chain controls and governance automation.
• Demonstrated proficiency in software applications and technical processes within a technical discipline such as cloud-native services, distributed systems, developer platforms, or security tooling.
• In-depth knowledge of the financial services industry and its IT systems, including expectations for traceability, control alignment, and audit-ready evidence.
• Practical cloud-native experience, including service design, deployment patterns, and runtime operations.

Preferred qualifications, capabilities, and skills

• Experience with NoSQL technologies such as Cassandra, MongoDB or DynamoDB .
• Familiarity with big data technologies such as Spark , Scala , and HDFS .
• Familiarity with modern infrastructure technology and architecture, including cloud, virtualization, and object stores.
• Experience implementing or operating open source governance controls, including license risk classification approaches (e.g., Red/Yellow/Green models) and approval workflows for restricted licenses (per internal license governance guidance, including examples such as AGPL-3.0 and enforcement expectations).
• Experience building compliance and posture dashboards that translate technical telemetry into operational health metrics and actionable insights for engineering and governance stakeholders.

About Us

About the Team