Big Data Architect Job Description Template
Our company is looking for a Big Data Architect to join our team.
Responsibilities:
- Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming;
- Experience with Cloudera/Hortonworks;
- Experience with HDF and NiFi;
- Strong understanding of open source technologies and databases (SQL & NoSQL);
- Proficient understanding of distributed computing principles;
- Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala;
- Experience with Spark;
- Ability to solve any ongoing issues with operating the cluster;
- Defining data retention policies;
- Monitoring performance and advising any necessary infrastructure changes;
- Mentoring junior engineers and working with other architects to deliver best in class solutions;
- Experience with various messaging systems, such as Kafka or RabbitMQ;
- Implementing ETL/ELT process and orchestration of data flows;
- Skills and Qualifications;
- Recommend and drive adoption of newer tools and techniques from big data ecosystem.
Requirements:
- Experience with SaaS platforms like AWS is a must;
- Advanced degree in Computer Science is a must;
- Expertise and hands-on experience in building large scale data pipelines using open source components and AWS services;
- Experience in building, monitoring and optimizing reliable and cost efficient pipelines for SaaS a must;
- Experience with SaaS based data management products is a plus;
- Strong experience in dealing with and integrating with data storage systems like sql and nosql databases, file systems and object storage like s3;
- 10+ years of industry experience in building and managing big data systems;
- Experience with Big Data technology like Hadoop, Spark, NoSQL based Data-warehousing solutions;
- Exposure to cloud (AWS), Provisioning, capacity planning and performance analysis atvarious levels;
- Data warehousing, Reporting including customization, Hadoop, Spark, Kafka, Core java, Spring/IOC, Design patterns;
- Understanding of web-based SOA architecture implementation with Design Pattern experience is useful;
- Experience with Reporting solutions like Pentaho, PowerBI, Looker including their customizations;
- Experience developing high concurrency, high performance applications that are database-intensive and have interactive, browser-based clients;
- Exposure to messaging systems, JMS, Active MQ, Rabbit MQ, Kafka;
- RDBMS, NoSQL like Vertica, Redshift, Data Modelling with physical design and SQL performance optimization.