Senior Data Engineer

Senior Data Engineer Job Description Template

Our company is looking for a Senior Data Engineer to join our team.

Responsibilities:

  • Build and maintain ETL pipelines and data-driven systems;
  • Requirements gathering and assessment;
  • Implement the pipelines using execution frameworks including but not limited to MapReduce, Spark, Hive, using Java/Scala/Python for application design;
  • Mentoring juniors in a dynamic team setting;
  • Engineer scalable data pipelines using big data technologies including but not limited to Hadoop, HDFS, Kafka, HBase, Elastic;
  • Breakdown complexity and translate requirements to specification artifacts and story boards to build towards, using a test-driven approach;
  • Manage stakeholders with proactive communication upholding TheDataTeam’s brand and values;
  • Maintain the maturity of existing and new data pipelines. This involves taking decisions around prioritizing technical debts while building features;
  • Own and deliver high performing systems and contribute to decision making in system architectures;
  • Contribute to information and data design across the organisation and become an integral part of data-driven design architecture;
  • Contribute to existing EL (extract & load) data pipelines while building new systems in parallel.

Requirements:

  • Detail oriented with willingness to proactively own software tasks as well as management tasks, and see them to completion with minimal guidance;
  • Azure Databricks, Spark and experience with big data technologies will be highly valuable;
  • Strong programming skills in at least one high-level language – Python, Java, C++, C#
  • 8+ years of experience working in data engineering related projects including analytics, data warehousing applications;
  • Bachelor ‘s Degree in computer science , mathematics, or other technical/quantitative discipline;
  • Should be well versed with atleast one database(Mysql/Oracle/MongoDB/Postgres);
  • Experience in Containerization (docker/Heroku);
  • Exposure to microservices;
  • Experience in Performance tuning of big data applications;
  • Strong understanding of various storage formats(ORC/Parquet/Avro);
  • Experience in Cloud enviornment(GCP/AWS/Azure);
  • Minimum 8 years of experience;
  • Exposure to DevOps practices;
  • Should have hands on experience in one of the Hadoop distributions(Hortoworks/Cloudera/MapR);
  • Should have experience in full life-cycle of one big data application.