Big Data Architect

Big Data Architect Job Description Template

Our company is looking for a Big Data Architect to join our team.

Responsibilities:

  • Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming;
  • Experience with Cloudera/Hortonworks;
  • Experience with HDF and NiFi;
  • Strong understanding of open source technologies and databases (SQL & NoSQL);
  • Proficient understanding of distributed computing principles;
  • Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala;
  • Experience with Spark;
  • Ability to solve any ongoing issues with operating the cluster;
  • Defining data retention policies;
  • Monitoring performance and advising any necessary infrastructure changes;
  • Mentoring junior engineers and working with other architects to deliver best in class solutions;
  • Experience with various messaging systems, such as Kafka or RabbitMQ;
  • Implementing ETL/ELT process and orchestration of data flows;
  • Skills and Qualifications;
  • Recommend and drive adoption of newer tools and techniques from big data ecosystem.

Requirements:

  • Experience with SaaS platforms like AWS is a must;
  • Advanced degree in Computer Science is a must;
  • Expertise and hands-on experience in building large scale data pipelines using open source components and AWS services;
  • Experience in building, monitoring and optimizing reliable and cost efficient pipelines for SaaS a must;
  • Experience with SaaS based data management products is a plus;
  • Strong experience in dealing with and integrating with data storage systems like sql and nosql databases, file systems and object storage like s3;
  • 10+ years of industry experience in building and managing big data systems;
  • Experience with Big Data technology like Hadoop, Spark, NoSQL based Data-warehousing solutions;
  • Exposure to cloud (AWS), Provisioning, capacity planning and performance analysis atvarious levels;
  • Data warehousing, Reporting including customization, Hadoop, Spark, Kafka, Core java, Spring/IOC, Design patterns;
  • Understanding of web-based SOA architecture implementation with Design Pattern experience is useful;
  • Experience with Reporting solutions like Pentaho, PowerBI, Looker including their customizations;
  • Experience developing high concurrency, high performance applications that are database-intensive and have interactive, browser-based clients;
  • Exposure to messaging systems, JMS, Active MQ, Rabbit MQ, Kafka;
  • RDBMS, NoSQL like Vertica, Redshift, Data Modelling with physical design and SQL performance optimization.