Lead Data Engineer - AWS/Databricks/PySpark

Job Description :We are looking for a Lead Data Engineer to own the data foundation for a high-volume client engagement where every number has to reconcile. The role involves designing data pipelines, maintaining accuracy through scale and schema changes, and balancing cost, reliability, and data integrity. It requires architectural judgement and hands-on engineering in equal measure.Roles and Responsibilities :- Own ingestion, transformation, and serving pipelines for high-volume event data on AWS, Databricks, and PySpark.- Make build-versus-buy decisions that balance cost, latency, and reliability.- Maintain a coherent lakehouse architecture as data sources, volumes, and schemas evolve.- Manage pipeline infrastructure as code, with CI/CD for data jobs and orchestration through Airflow or Databricks Workflows.- Control cloud costs through partitioning, file formats, and compute sizing.- Track cost per pipeline and cost per query as first-class metrics.- Diagnose and fix bottlenecks in Spark jobs and downstream serving layers.- Build reconciliation and data quality checks to catch discrepancies before reporting or billing.- Monitor and alert on pipeline failures, schema drift, and silent data loss.- Ensure data accuracy where downstream business logic depends on it.- Define schema contracts, naming conventions, and documentation standards across pipelines.- Establish access and data-handling practices that meet client and regulatory requirements.- Review changes to shared datasets and warn downstream consumers before breaking changes occur.- Direct engineers on the engagement without formal reporting lines.- Represent data engineering with clients by explaining trade-offs, defending decisions, and flagging risks early.- Mentor engineers through design reviews, debugging sessions, and code reviews.Key Skills:- 8 - 12 years of experience in data engineering, with ownership of large-scale production pipelines.- Deep hands-on experience with AWS, Databricks, and PySpark, including debugging jobs when they fail.- Experience building systems that handle schema evolution and upstream changes while keeping downstream consumers running.- Experience in high-volume, high-stakes environments where accuracy and timeliness have business consequences.- Cost optimisation experience, including reducing cloud costs or resolving runaway pipelines.- Hands-on experience with Infrastructure as Code (Terraform or equivalent), workflow orchestration, and CI/CD for data pipelines.- Strong consulting skills, including handling client scrutiny, shifting priorities, and high communication standards.- Strong technical leadership and problem-solving skills, with the ability to lead without formal authority. (ref:hirist.tech)