Infosys - Data Engineer - AWS Platform

Key Responsibilities :- Design, develop, and maintain scalable batch and real-time data pipelines using AWS cloud services.- Build robust ETL/ELT workflows for data ingestion, transformation, validation, and loading from multiple structured and unstructured data sources.- Develop and maintain cloud-based data lakes using Amazon S3 and related AWS services.- Design and optimize data warehouse solutions using Amazon Redshift.- Develop data integration and orchestration frameworks using AWS Glue, Lambda, EMR, and other AWS services.- Develop PySpark-based data processing applications for large-scale distributed data workloads.- Write optimized SQL queries for data transformation, analysis, and reporting requirements.- Implement data ingestion frameworks to process data from databases, APIs, files, applications, and other enterprise systems.- Optimize data pipelines for performance, scalability, reliability, and cost efficiency.- Monitor data pipelines and troubleshoot failures, latency, data quality issues, and performance bottlenecks.- Implement data validation and quality checks to ensure accuracy, consistency, completeness, and reliability of enterprise data.- Build and maintain data processing solutions using AWS Athena for querying data stored in data lakes.- Work with Amazon EMR for distributed processing and large-scale data transformation workloads.- Implement appropriate data security, access controls, encryption, and governance practices across AWS data platforms.- Automate infrastructure provisioning and deployment using Infrastructure as Code practices.- Contribute to CI/CD processes for data engineering workflows and production deployments.- Ensure data platforms are highly available, scalable, secure, and resilient.- Collaborate with data analysts, data scientists, architects, application teams, and business stakeholders to understand data requirements.- Participate in technical design discussions, code reviews, solution development, and production support.- Maintain technical documentation related to data pipelines, data models, integrations, and platform architecture.Technical Skills :- Strong hands-on experience in AWS Data Engineering and cloud-native data solutions.- Proficiency in AWS Glue, Amazon S3, Amazon Redshift, Amazon Athena, Amazon EMR, and AWS Lambda.- Strong programming experience in Python, with hands-on expertise in PySpark.- Strong SQL skills, including complex queries, joins, aggregations, window functions, and query optimization.- Experience designing and implementing data lakes and enterprise data warehouse solutions.- Strong understanding of ETL/ELT architecture, data ingestion, transformation, orchestration, and processing.- Experience working with distributed data processing frameworks, particularly Apache Spark and PySpark.- Understanding of data partitioning, file formats, schema management, and performance optimization.- Experience with data pipeline monitoring, logging, troubleshooting, and failure recovery.- Knowledge of cloud data security, access management, encryption, and data governance practices.- Experience with Infrastructure as Code and automated deployment practices.- Familiarity with CI/CD processes and Agile/Scrum development methodologies. (ref:hirist.tech)