Senior Data Engineer Job Description Template
Our company is looking for a Senior Data Engineer to join our team.
Responsibilities:
- Build and maintain ETL pipelines and data-driven systems;
- Requirements gathering and assessment;
- Implement the pipelines using execution frameworks including but not limited to MapReduce, Spark, Hive, using Java/Scala/Python for application design;
- Mentoring juniors in a dynamic team setting;
- Engineer scalable data pipelines using big data technologies including but not limited to Hadoop, HDFS, Kafka, HBase, Elastic;
- Breakdown complexity and translate requirements to specification artifacts and story boards to build towards, using a test-driven approach;
- Manage stakeholders with proactive communication upholding TheDataTeam’s brand and values;
- Maintain the maturity of existing and new data pipelines. This involves taking decisions around prioritizing technical debts while building features;
- Own and deliver high performing systems and contribute to decision making in system architectures;
- Contribute to information and data design across the organisation and become an integral part of data-driven design architecture;
- Contribute to existing EL (extract & load) data pipelines while building new systems in parallel.
Requirements:
- Detail oriented with willingness to proactively own software tasks as well as management tasks, and see them to completion with minimal guidance;
- Azure Databricks, Spark and experience with big data technologies will be highly valuable;
- Strong programming skills in at least one high-level language – Python, Java, C++, C#
- 8+ years of experience working in data engineering related projects including analytics, data warehousing applications;
- Bachelor ‘s Degree in computer science , mathematics, or other technical/quantitative discipline;
- Should be well versed with atleast one database(Mysql/Oracle/MongoDB/Postgres);
- Experience in Containerization (docker/Heroku);
- Exposure to microservices;
- Experience in Performance tuning of big data applications;
- Strong understanding of various storage formats(ORC/Parquet/Avro);
- Experience in Cloud enviornment(GCP/AWS/Azure);
- Minimum 8 years of experience;
- Exposure to DevOps practices;
- Should have hands on experience in one of the Hadoop distributions(Hortoworks/Cloudera/MapR);
- Should have experience in full life-cycle of one big data application.