Sign up to access all features of our service
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Lead Data Engineer - AWS/Databricks/PySpark

Upcruit

Job Description :We are looking for a Lead Data Engineer to own the data foundation for a high-volume client engagement where every number has to reconcile. The role involves designing data pipelines, maintaining accuracy through scale and schema changes, and balancing cost, reliability, and data integrity. It requires architectural judgement and hands-on engineering in equal measure.Roles and Responsibilities :- Own ingestion, transformation, and serving pipelines for high-volume event data on AWS, Databricks, and PySpark.- Make build-versus-buy decisions that balance cost, latency, and reliability.- Maintain a coherent lakehouse architecture as data sources, volumes, and schemas evolve.- Manage pipeline infrastructure as code, with CI/CD for data jobs and orchestration through Airflow or Databricks Workflows.- Control cloud costs through partitioning, file formats, and compute sizing.- Track cost per pipeline and cost per query as first-class metrics.- Diagnose and fix bottlenecks in Spark jobs and downstream serving layers.- Build reconciliation and data quality checks to catch discrepancies before reporting or billing.- Monitor and alert on pipeline failures, schema drift, and silent data loss.- Ensure data accuracy where downstream business logic depends on it.- Define schema contracts, naming conventions, and documentation standards across pipelines.- Establish access and data-handling practices that meet client and regulatory requirements.- Review changes to shared datasets and warn downstream consumers before breaking changes occur.- Direct engineers on the engagement without formal reporting lines.- Represent data engineering with clients by explaining trade-offs, defending decisions, and flagging risks early.- Mentor engineers through design reviews, debugging sessions, and code reviews.Key Skills:- 8 - 12 years of experience in data engineering, with ownership of large-scale production pipelines.- Deep hands-on experience with AWS, Databricks, and PySpark, including debugging jobs when they fail.- Experience building systems that handle schema evolution and upstream changes while keeping downstream consumers running.- Experience in high-volume, high-stakes environments where accuracy and timeliness have business consequences.- Cost optimisation experience, including reducing cloud costs or resolving runaway pipelines.- Hands-on experience with Infrastructure as Code (Terraform or equivalent), workflow orchestration, and CI/CD for data pipelines.- Strong consulting skills, including handling client scrutiny, shifting priorities, and high communication standards.- Strong technical leadership and problem-solving skills, with the ability to lead without formal authority. (ref:hirist.tech)

Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer - AWS/Databricks/PySpark in Bangalore vacancy
  •  ...Req ID:  376347   NTT DATA strives to hire exceptional, innovative...  ...with DevOps, UI/UX, engineering, and product teams. Key Candidate...  ...Kubernetes Cloud platforms – AWS, Azure, or GCP Agile /...  ...innovation. We are one of the world's leading AI and digital infrastructure... 
    Suggested
    Hybrid work
    Work at office
    Remote job
    Flexible hours

    NTT DATA Services

    Bangalore
    2 hours ago
  •  ...Position overview   AWS Cloud DBA Admin Advisor with a strong background in database management, cloud infrastructure, automation,...  ...including optimization and scaling, to enhance performance and ensure data integrity within our client’s cloud infrastructure.... 
    Suggested
    Shift work

    NTT DATA Services

    Bangalore
    2 hours ago
  •  ...Req ID:  396733   NTT DATA strives to hire exceptional, innovative and passionate individuals...  ...(IN). Java, Kafka, SQL, Snowflake, AWS, Docker, Kubernetes, Jenkins - Developer...  ...responsible innovation. We are one of the world's leading AI and digital infrastructure providers,... 
    Suggested
    Hybrid work
    Work at office
    Remote job
    Flexible hours

    NTT DATA Services

    Bangalore
    2 hours ago
  •  ...Req ID:  384673   NTT DATA strives to hire exceptional, innovative and passionate individuals...  ...(MFT) platforms. The role involves leading transition activities from incumbent vendors...  ...model • Knowledge of cloud platforms (Azure/AWS) is an added advantage • Monitoring tool... 
    Suggested
    Hybrid work
    Work at office
    Remote job
    Flexible hours

    NTT DATA Services

    Bangalore
    2 hours ago
  •  ...Neo4j, and Memgraph. Experience with Git, Maven or Gradle, Docker, Kubernetes, Jenkins or GitLab CI, and Postman. Familiarity with AWS, Azure, or GCP and messaging systems such as Kafka, NATS, or MQ. Experience with AI-driven or AI-assisted code-generation tools.... 
    Suggested

    Capco

    Bangalore
    3 hours ago
  •  ...Responsibilities Lead the architecture, design, and implementation of Generative...  ...Guide multidisciplinary teams across Data Science, Data Engineering, ML Engineering, Cloud, Analytics, DevOps...  ...RAG, and LLM fine-tuning across Azure, AWS, and GCP. ~ Experience leveraging open... 
    Full time

    Bosch Global

    Bangalore
    3 hours ago
  •  ...12 Yrs Category: Software Development/ Engineering Shift: 10: to 6 PM (3 Days Mandatory Work...  ...using Django REST Framework (DRF). • Lead backend development initiatives and define...  ...Experience working with cloud platforms such as AWS, Azure, or GCP. • Strong understanding... 
    Full time
    Work at office
    Local area
    Shift work
    Bangalore
    4 hours ago
  • Senior Data Engineer Position Description About CGI Founded in 1976, CGI is among the world...  ...Title: Senior Data Engineer Position: Lead Analyst Experience: 7 - 9 years...  ...Factory, SQL DB, Synapse, Stream Analytics; AWS Glue, Kinesis, Redshift; Airflow; GitHub,... 
    Permanent employment
    Full time
    Hybrid work
    Local area
    Shift work
    Bangalore
    4 hours ago
  • Full Stack Engineer (Python) Position Description About CGI Founded...  ...Engineer (Python) Position: Lead Software Engineer Experience...  ..., Energy Price Calculator, EDP Data Platform). The role builds and...  ...workloads and functions to AWS or Azure using cloud native CI/... 
    Permanent employment
    Full time
    Hybrid work
    Local area
    Shift work
    Bangalore
    4 hours ago
  • Senior Project Manager – Data Engineering & AI Position Description Role: Senior Project Manager...  ...related discipline Job Description Lead end to end delivery of Data Engineering...  ...vendor management Secondary Azure / AWS / GCP; SQL, Python, Spark, Databricks, Snowflake... 
    Full time
    Hybrid work
    Bangalore
    4 hours ago
  •  ...management, and key rotation Monitor and tune performance, availability, and security of Vault services What you need Python AWS Terraform Azure PowerShell GCP Good to have IAM Kubernetes Microservices CI/CD Bash REST APIs Job... 
    Full time

    Simeio

    Bangalore
    7 days ago
  •  ...as GitHub Advanced Security, Snyk and Checkmarx, keeping the signal high enough that engineers act on the findings What you need OWASP LLMs Git CI/CD Good to have AWS GCP Azure Job details Location: Remote, anywhere in India Education: Any... 
    Full time
    Remote job

    CoverGo

    Bangalore
    4 hours ago
  •  ...are too, let's talk. About the Team & Impact At Okta, Demo Engineering bridges the gap between product innovation and go-to-market execution...  ...Expertise: Strong proficiency in React.js , Node.js , and AWS Serverless architectures (Lambda, DynamoDB, SQS, SNS). Multi... 
    Long term contract
    Full time
    Local area
    Worldwide

    Okta

    Bangalore
    4 hours ago
  •  ...looking for an experienced Software Engineer with strong skills in Python, algorithm...  ..., scientific computing, and data-driven solution development. The...  ...data platform evolves from the current AWS-based environment toward Databricks, the candidate will also contribute... 
    Remote job
    Full time
    Work at office
    Working Monday to Friday
    Bangalore
    4 days ago
  •  ...and local enablement teams. Markets, BMC, AWS, Finance and Accounting, Risk Management, Procurement...  ...looking to build a dedicated to support a leading alternative investment / private markets...  ..., and management reports. Perform data validation and quality checks before report... 
    Local area
    Immediate start
    Worldwide
    Flexible hours

    Ernst & Young

    Bangalore
    5 hours ago
  •  ...and local enablement teams. Markets, BMC, AWS, Finance and Accounting, Risk Management, Procurement...  ...delivery. By plugging into our market-leading global network, you'll gain the experience...  ...and by providing your consent to data privacy. Successful candidates advance to... 
    Local area
    Immediate start
    Worldwide
    Flexible hours

    Ernst & Young

    Bangalore
    5 hours ago
  •  ...capabilities, as well as the ability to work with data and communicate it in a user-friendly...  ...Automate - is of advantage First team lead experience / playing the role of senior point...  ..., and cloud-based services (Azure, AWS, etc.), and their integration, Knowledge... 
    Work at office
    Flexible hours
    Shift work
    3 days week
    Bangalore
    1 day ago
  •  ...Overview As a Staff Backend Engineer, you will work on distributed systems...  ...day! What You Will Do Lead architecture design for...  ...distributed systems, storage, data processing, and server technologies...  ...oriented design ~ Experience with AWS-based service development and... 
    Hybrid work

    Coupang

    Bangalore
    6 hours ago
  •  ...building a scalable, AI-first healthcare data platform that turns raw clinical and financial...  ...You will work closely with architects and engineering teams to design and deliver high-scale,...  ...Neptune) Cloud platforms (Azure preferred, AWS acceptable) Distributed systems and... 
    Full time

    RhythmX AI

    Bangalore
    7 hours ago
  •  ...infrastructure, including: ~ Longitudinal data on 300M patients ~4.4B annual claims ~...  ...deep collaboration with architects and engineering teams to design and deliver high-scale, reliable...  ...) Cloud platforms (Azure preferred, AWS acceptable) Distributed systems and... 
    Full time

    RhythmX AI

    Bangalore
    7 hours ago
  •  ...Engineering Manager – Platform Reliability About GW RhythmX Our Vision...  ..., including: ~ Longitudinal data on 300M patients ~4.4B annual...  ...Platform Reliability, you will lead and drive our DevOps, Site...  ...operations across Microsoft Azure and AWS. Drive infrastructure... 
    Full time

    RhythmX AI

    Bangalore
    7 hours ago
  •  ...DevOps Engineer About GW RhythmX Our Vision: Forever change medicine by turning intelligence...  ..., including: ~ Longitudinal data on 300M patients ~4.4B annual claims ~...  ...Microsoft Azure, Azure Kubernetes Service (AKS), AWS. Containerization: Kubernetes, Docker,... 
    Full time

    RhythmX AI

    Bangalore
    7 hours ago
  •  ...re all united in the vision to lead the programmatic industry and make...  ...As a SE II - Fullstack in our Engineering department, you’ll have the...  ...Web application frameworks, Big Data technologies, Angular/React UI...  ...stacks comprising Java, Python, AWS, Docker, Kubernetes, Spark, Qubole... 
    Hybrid work
    Work at office
    Afternoon shift

    MiQ Digital

    Bangalore
    7 hours ago
  •  ...for marketers and agencies. We connect data from multiple sources to do interesting...  ...THE ROLE: MiQ is looking for an Engineering Manager to lead the development teams that build its proprietary...  ...with cloud environments such as AWS, Azure or GCP. ~ Prior experience with... 

    MiQ Digital

    Bangalore
    7 hours ago
  •  ...Role As a Senior Full Stack AI Engineer at JazzX AI, you will play a...  ...features and services end to end Lead the design and development of...  ...for AI agent orchestration, data pipelines, and micro-frontends...  ...(Storage, Monitor, Identity), AWS (S3, boto3), Docker, Kubernetes... 
    Long term contract
    Full time
    Worldwide

    JazzX AI

    Bangalore
    7 hours ago
  •  ...Staff Software Engineer (Data Platform) Join our dynamic team at the forefront of cutting-edge technology as we seek a seasoned Staff Data Engineer...  ...systems such as Kafka. ● Experience with cloud platforms (AWS/GCP) and big data tools such as Spark. ● Strong understanding... 

    Arcana Analytics

    Bangalore
    7 hours ago
  •  ...and we're rebuilding it from the ground up. The Role As a Lead Data Engineer, you will take ownership of designing and delivering high-...  ...Golang, and SQL. ~ Proven experience with cloud-native services (AWS, GCP) and big data tools (e.g., Spark, Flink, Datafusion, Doris... 
    Hybrid work
    Work at office
    Flexible hours

    Arcana Analytics

    Bangalore
    7 hours ago
  •  ...forward. From cloud-native platforms and data-driven solutions to DevOps automation, enterprise...  .... We are looking for a hands-on DevOps Engineer who enjoys self-managed open-source...  ...major plus. Cloud experience with GCP, AWS, or Azure; GCP is strongly preferred. Experience... 

    Arcana Analytics

    Bangalore
    7 hours ago
  •  ...Lead Product Engineer Location: Bengaluru, India Work Model: Hybrid We're looking for a Lead...  ...into production Lead API design and data modeling decisions; write clean, well-tested...  ...similar), and cloud infrastructure on AWS/GCP/Azure ~ Daily, disciplined use of... 
    Hybrid work
    Worldwide

    Picarro, Inc

    Bangalore
    7 hours ago
  •  ...architectures. Consultative Implementation: Lead the end-to-end deployment of UEM, MDM, RMM...  ...loop between customers and our Product/Engineering teams to advocate for feature requests and...  ...-native security concepts, architectures (AWS/Azure/GCP), and associated technologies.... 
    Full time
    Worldwide
    Shift work

    Mitsogo Inc

    Bangalore
    7 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer - AWS/Databricks/PySpark. Be the first to apply!