Senior Data Engineer : Big Data

Lead Data Engineer Skills – Mandatory- S3, Glue, Lambda, Step Function, Snowflake .
Skills - Primary - PySpark, Snowflake.
Skills - Good to have - Cassandra
Total Experience - 7+ years ; Relevant Experience - 5+ years
Work Location : Cochin/TVM

Job Purpose
Responsible for consulting for the client to understand their data needs and delivering data solution to the client.

Job Description / Duties & Responsibilities
•
Data Ingestion Framework:
oDesign & Development: Design, develop, and maintain an end-to-end data ingestion framework that efficiently extracts, transforms, and loads data from diverse sources.
oFramework Optimization: Use AWS services such as AWS Glue, Lambda, EC2 and Step Functions to build highly scalable, resilient, and automated data pipelines.
•
Data Quality & Validation:
oValidation Processes: Develop and implement automated data quality checks, validation routines, and error-handling mechanisms to ensure the accuracy and integrity of incoming data.
oMonitoring & Reporting: Establish comprehensive monitoring, logging, and alerting systems to proactively identify and resolve data quality issues.
•
Collaboration & Agile Practices:
oCross-Functional Communication: Work closely with business stakeholders, and application development teams to understand requirements and translate them into technical solutions.
oAgile Development: Participate in sprint planning, code reviews, and agile ceremonies, while contributing to continuous improvement initiatives and CI/CD pipeline development (using tools like GitLab).

Job Specification / Skills and Competencies:

Experience & Technical Skills:
oProfessional Background: At least 5 years of relevant experience in data engineering with a strong emphasis on analytical platform development.
oProgramming Skills: Proficiency in PySpark, Snowflake for developing ETL processes and handling large-scale data manipulation.

Job Description
oAWS Expertise: Extensive experience using AWS services including AWS Glue, Lambda, Step Functions, and S3 to build and manage data ingestion frameworks.
oData Platforms: Familiarity with big data systems (e.g., AWS EMR, Apache Spark, Apache Iceberg).
oCI/CD & Agile: Hands-on experience with CI/CD pipelines (preferably with GitLab) and Agile development methodologies.