Big Data Platform Administrator - Apache Spark & Airflow
Samridhi Placements
Job Description :Key Responsibilities :- Administer and support Hadoop ecosystem components (HDFS, YARN) across Production, Test, and Development environments- Manage and maintain Apache Spark and Airflow infrastructure- Monitor cluster health, troubleshoot performance issues, and optimize system performance- Implement and maintain High Availability (HA), Disaster Recovery (DR), backup, and restore strategies- Manage logging, monitoring, and alerting systems using Prometheus, Grafana, and ELK Stack- Perform Linux (Ubuntu) system administration, including patching, upgrades, and security hardening- Develop and maintain automation scripts using Shell and/or Python- Handle Kerberos authentication setup and maintenance- Support incident management, root cause analysis (RCA), and ensure SLA adherence- Manage capacity planning and system scalability- Follow CI/CD and DevOps best practices for platform improvements and deploymentsRequired Skills :- Strong experience with Hadoop ecosystem (HDFS, YARN)- Handson expertise in Apache Spark and Apache Airflow- Proficiency in Linux system administration (Ubuntu), including security hardening- Scripting skills in Shell and Python for automation- Familiarity with monitoring and observability tools: Prometheus, Grafana, ELK Stack- Knowledge of High Availability, Disaster Recovery, backup, and restore concepts- Experience with Kerberos authentication setup and maintenance (ref:hirist.tech)
Vacancy posted 18 days ago
Similar jobs that could be interesting for youBased on the Big Data Platform Administrator - Apache Spark & Airflow in Chennai vacancy
- ...enhance our clients’ journey around Data Virtualization. Your mission:... ...knowledge of the Denodo Platform, be able to deliver a superb technical... ...Platform. * Denodo Certified Administrator (8.0 / 9.0), Denodo Certified... .... * Experience working with Big Data and/or noSQL environments...Big DataFixed term contractRemote job
- ...systems that handle large amounts of data.- Collaborate with product owners... ...SQL experience.- 2+ years using Spark and other frameworks to process large... ...language.- Experience with big data and/or infrastructure (e.g., Hive, Spark, Airflow).- Understanding of data organization...Big Data
- Job Description :We are looking for a junior database administrator (DBA) to join our IT team and help us manage and maintain our databases. You... ...the security, availability, performance, and integrity of our data. You will also assist in database design, development, backup, recovery...SuggestedRemote job
- ...Overview :We are looking for an experienced Data Scientist with strong expertise in... ...or Generative AI.- Exposure to cloud platforms such as GCP, AWS, or Azure.-... ...ML pipeline automation.- Knowledge of big data technologies such as Spark and Databricks.- Experience building...Big Data
- ...Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation.Job... ...requirements.Nice-to-Have Skills : - Experience with cloud and big data platforms (AWS, Azure, SQL Server, Data Warehouse, Azure Data Warehousing...Big Data
- ...first and fastest cloud-native automotive platform that includes the revolutionary... ...platform uses cutting-edge technology, big data, machine learning, and AI to seamlessly... ...practices. ~ Strong experience in database administration, including backup/recovery, replication,...Big DataFull time
- ...the paradigm with the first and fastest cloud-native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for... .... The transformative platform uses cutting-edge technology, big data, machine learning, and AI to seamlessly bring together OEMs, retailers...Big DataLong term contractFull time
- ...Job Type: Full-time We are looking for an experienced Senior Data Scientist with strong expertise in Python, Machine Learning,... ...systems. ~ Experience with relational databases, SQL, and NoSQL or big-data technologies such as MongoDB, Cassandra, or Hadoop. ~ Proficiency...Big DataFull time
- ...Role Purpose The Senior Database Administrator will own database reliability, availability, performance, security, recoverability, and operational... ...alone as the success criterion. 4. Table Partitioning and Data Lifecycle Management Design, implement, and maintain table...Permanent employmentFull timeTemporary workHybrid work
- ...department in charge of developing the Denodo Platform. You will join one of the teams that... ...scheduling and parallel execution, self-service data discovery, client tools, etc. The... ...multidimensional, parallel, document-oriented, etc.), Big Data sources, SaaS sources, Web Services,...Big Data
- ...advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is... ...input management, including shopper data integration, panel expansion,... ...measurable outcomes. ~ Proficiency in Big Data technologies such as Spark, Snowflake, and Databricks. ~ Hands...Big DataFull timeLocal areaFlexible hours
- ...are looking for an experienced Databricks Platform Engineer to manage, maintain, secure, and... ...will have strong expertise in Databricks administration, Unity Catalog, AWS infrastructure,... ...access controls.- Manage Unity Catalog, data governance, permissions, and workspace security...
- ...1. Designing, developing, and implementing OIC integration solutions using OIC pre-built adapters, REST/SOAP web services, File based data import/export (FBDI), HDL.2. Handling Transformations and orchestration of interfaces between systems.3. Implementing fault handling and...Shift work
- ...Responsibilities :- Define and drive enterprise data architecture, data strategy, and... ...management and analytics solutions using modern Big Data platforms.- Lead the transformation of traditional... ...platforms (Hadoop, Presto, Tez, Hive, Spark etc.Hands-on experience in work with...Big Data
- ..., forecasting, and planning cycles end-to-end- Develop and maintain business rules, Groovy scripts, and automation scripts- Coordinate data integration, validation, and issue resolution activities- Support UAT, go-live, and post-deployment activitiesRequired Skills :- 5+ years...Afternoon shift
- ...Summary :We are looking for a Senior Data Engineer who is creative,... ...scalability and performance.- Utilize big data tools and frameworks to... ...working with enterprise data platforms, including building and... ...design concepts.- Proficiency in Spark (Python/Scala) and Spark Streaming...Big Data
- ...Description :We are looking for a Lead Data Engineer to join our Data Platform team and take ownership of designing,... ...management using tools such as Airflow, Azkaban, or Luigi.- Guide implementation... ...in Data Engineering, Data Platforms, Big Data, or related technology roles.- Strong...Big DataPermanent employment
- ...Description :Eucloid is looking for a Data Engineer to join our Data Platform team supporting various business... ...data from various resources, managing big data pipelines that are easily accessible... ...management tools : Azkaban, Luigi, Airflow, etc.- Good experience with at least...Big Data
- ...Systems Engineer with deep expertise in Data DevOps/MLOps to join our organization.... ...or comparable positions Skilled in cloud platforms such as Azure, AWS, or GCP Experienced with... ...with data processing frameworks such as Apache Spark and Databricks Skilled in programming languages...Big Data
- ...dedicated Lead Systems Engineer who brings Data DevOps/MLOps experience to enhance... ...machine learning models through cloud-based platforms Streamline key processes like data validation... ...using data processing frameworks such as Apache Spark and Databricks Strong Python skills along...Big Data
- ...Systems Engineer who brings deep expertise in Data DevOps/MLOps to strengthen our... ...or comparable positions Skilled in cloud platforms such as Azure, AWS, or GCP Experience working... ...experience using data processing frameworks like Apache Spark and Databricks Programming skills in...Big Data
- ...for an experienced Senior Clinical Data Engineer to design, build, and... ...source systems into enterprise data platforms such as data lakes and data warehouses... ...-on experience with distributed/big-data processing technologies such as Apache Spark . ~ Experience with workflow...Big Data
- ...is a global leader in enterprise-grade data analytics and AI solutions, committed to... ...fueled by our partnership with EQT.Role & Administrative Details : Total exp : 11 years.Location... ...: Senior Software Engineer Core Java & Apache Spark.Role Summary : We are hiring an elite Senior...Big DataWorldwide
- ...bridge the gap between complex data science and robust software engineeringdeveloping... ...and unstructured data using big data tools (GCP BigQuery,... ..., Data Fusion, PySpark, Airflow).- Manage data persistence... ...Strong expertise in Google Cloud Platform (GCP) preferred (or AWS/Azure equivalent...Big DataHybrid work
- ...Oracle Cloud ERP security operations across FSCM and HCM in a production support model.- Administer and maintain user access, custom roles, data roles, security profiles, AOR assignments, and HDL-related security activities.- Support secure end-to-end business processes by...
- Qualifications : BE, BTech, MCA, or equivalent degree in Computer Science, IT, or a related field with 6 to 12+ years of experience.Required Skills : - Install, configure, and maintain DB2 database systems (z/OS).- Manage database infrastructure including instances, partitions...
- Role Overview :We are looking for an experienced Big Data Engineer with strong hands-on expertise in Apache Spark and Scala to design, develop, and optimize large-scale data processing solutions. The role will involve building scalable data pipelines, processing high-volume...Big Data
- Professional Summary : Experienced Hadoop Administrator with 8+ years of expertise in designing, implementing, administering, monitoring, troubleshooting, and supporting enterprise Big Data platforms.Strong hands-on experience with Cloudera CDP, Cloudera Manager, HDFS, YARN...Big Data
- ...Global Operations Technology and Client Data Platforms: Development / Support/ testing of the technology... ...-facing operational applications. The Big data technologies used in this are Hadoop... ...real time streaming applications, like Spark Streaming, Kafka. We are actively...Big DataWork at officeFlexible hours
- ...Senior Data Engineer (Engineer 3) Position Overview We are seeking... ...infrastructure, pipelines, and platforms for collecting, storing,... ...transformation mechanisms and big data tools, considering long-term... ...Proficiency with Terraform, Tekton, and Apache Airflow. Data processing and...Big DataLong term contractFull time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Platform Administrator - Apache Spark & Airflow. Be the first to apply!
