Data Engineer - Spark/Kafka

Role Overview:We are seeking a seasoned Data Engineer to join our core engineering team to architect and maintain high-performance data pipelines that power our analytical platforms. In this role, you will work closely with data scientists, product managers, and software engineers to transform raw data into actionable insights, ensuring that our infrastructure remains scalable, reliable, and efficient. By building robust data ecosystems, you will directly influence business decision-making and enhance the customer experience through data-driven product features. Your work will serve as the backbone for our strategic initiatives, enabling the organization to leverage information as a competitive advantage.Key Responsibilities:- Design and implement scalable ETL/ELT pipelines using Spark to process large-scale datasets, ensuring high data quality and availability for downstream analytics teams.- Architect real-time streaming solutions using Kafka to facilitate low-latency data ingestion, enabling the business to respond to customer behavior in real-time.- Manage and optimize cloud-native data infrastructure on AWS, ensuring cost-effectiveness and high availability for mission-critical applications.- Collaborate with cross-functional stakeholders to translate complex business requirements into technical data models that drive product innovation.- Monitor and troubleshoot data pipeline performance, proactively identifying bottlenecks to maintain seamless data flow across the enterprise.Required Skillset:- Demonstrated expertise in building distributed data systems using Spark and Kafka, with a proven ability to handle high-volume, complex data environments.- Advanced proficiency in AWS cloud services, including S3, EMR, Redshift, and Glue, to architect secure and performant data storage and processing solutions.- Strong analytical mindset with the ability to communicate technical data concepts to non-technical stakeholders, fostering a culture of data-driven collaboration.- Proven track record of working effectively within agile, cross-functional teams, showing adaptability to evolving project priorities and technical requirements.- A Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field, complemented by a minimum of 4 to 6 years of hands-on experience in data engineering roles. (ref:hirist.tech)