Big Data Developer
AGILE VENTURES PRIVATE LIMITED
Job Description : We are looking for an experienced Big Data Developer with 7-11 years of experience to design, develop, and optimize large-scale data processing solutions. The ideal candidate should have strong expertise in Apache Spark, Python, Hadoop, distributed computing, streaming technologies, and cloud-based data platforms, along with the ability to provide technical guidance to development teams.Key Responsibilities : - Design, develop, and maintain scalable Big Data solutions for high-volume and complex data workloads.- Develop distributed data processing applications using Apache Spark and Python.- Design and implement batch and real-time data processing pipelines.- Work with Hadoop ecosystem technologies including HDFS, MapReduce, Sqoop, Hive, and Impala.- Develop stream-processing solutions using Spark Streaming, Apache Storm, or equivalent technologies.- Design and integrate messaging and event-streaming solutions using Apache Kafka, RabbitMQ, or similar platforms.- Build data pipelines integrating data from multiple sources including RDBMS, ERP systems, APIs, and files.- Develop and optimize SQL queries, joins, stored procedures, and relational data models.- Work with NoSQL databases such as HBase, Cassandra, or MongoDB based on application requirements.- Implement ETL processes and data integration frameworks for large-scale data environments.- Perform Spark job optimization and performance tuning to improve processing efficiency and resource utilization.- Design and implement cloud-based Big Data solutions using Azure Databricks and related Azure data services.- Troubleshoot production issues and ensure reliability, scalability, and performance of data platforms.- Collaborate with data engineers, architects, analysts, and business stakeholders to translate requirements into technical solutions.- Provide technical guidance and mentor team members while contributing to design and architecture decisions.Required Skills : - 7+ years of experience in Big Data and data engineering technologies.- Strong understanding of distributed computing principles and architectures.- Expert-level hands-on experience with Apache Spark.- Strong programming experience in Python.- Proficiency in Hadoop v2, HDFS, MapReduce, and Sqoop.- Experience with Spark Streaming, Apache Storm, or equivalent stream-processing technologies.- Hands-on experience with Kafka, RabbitMQ, or similar messaging platforms.- Strong knowledge of Hive, Impala, and Big Data querying technologies.- Strong SQL skills, including complex joins, stored procedures, and relational schemas.- Experience working with NoSQL databases such as HBase, Cassandra, or MongoDB.- Strong understanding of ETL techniques and data integration frameworks.- Experience in Spark performance tuning and optimization.- Hands-on experience with Azure Databricks and cloud-based data services.- Strong problem-solving, analytical, and communication skills.- Experience leading or technically guiding a team is required.- Ability to design and implement scalable, reliable, and high-performance data solutions. (ref:hirist.tech)
- ...Engineer - AI Engineer You’ll make an impact by: Design, develop, and deploy LLM-powered applications (chatbots, copilots,... ...Integrate GenAI models with enterprise systems, APIs, and data sources Fine-tune or adapt foundation models (e.g., GPT,...Suggested1 day week
- ...for all. Job Description We are looking for a team player who brings a strong testing mindset with a focus on ETL, Database, Data Validation , and Functional Testing, along with exposure to test automation. Has 3 to 5 years of experience in ETL testing ,...Suggested
- ...spectrum of the automotive retail ecosystem through one seamless platform. The transformative platform uses cutting-edge technology, big data, machine learning, and AI to seamlessly bring together OEMs, retailers/dealers and consumers. With its highly configurable integration...Big DataFull time
Rs 3.2 - 4 lakhs p.a.
...Actively hiring Agentic AI Developer Bangalore (Hybrid) Start Date Starts Immediately CTC (ANNUAL) ₹ 3,20,000 - 4,00,000...SuggestedHybrid workWork at officeImmediate startRemote job3 days week- ...review, automated testing, and secure coding standards. Designs, develops, codes, and troubleshoots with consideration of upstream and... ...Gathers, analyzes, and draws conclusions from large, diverse data sets to identify problems and contribute to decision-making in service...Suggested
- ...Knowledge Graphs, LLM, and experience in working with massive amounts of data to build systems that reach JP Morgan scale. Job... ...learning, Chain of thought and other prompt engineering techniques. Develop production-grade agentic workflows using modern orchestration frameworks...
- ...approaches to build solutions or breakdown technical problems Develops secure and high-quality production code, and reviews and debugs code... ...of pipelines and DAG's (Directed Acyclic graph) for processing data and/or machine learning Demonstrated proficiency in software...
- ...Job Responsibilities Participates in, design and develop scalable and resilient systems using Java to contribute to continual... ...synthesizes, and develops visualizations and reporting from large, diverse data sets in service of continuous improvement of software applications...
- ...Principal data architects at Thoughtworks play a key part in developing modern data architecture approaches to meet business objectives and provide end-to-end data solutions... ...designing application system architecture based on big data, artificial intelligence and related...Big DataRemote job
Rs 12 - 30 lakhs p.a.
...0+ years Location: Bengaluru, Karnataka, India JobType: full-time Requirements Responsibilities include: Lead a team of data quality engineers supporting validation efforts across multiple data pipelines and domains. Define and drive automation strategies to...Full timeWeekday work- ...Type: Full-time We are looking for an experienced Manager – Data Quality Engineering to lead data quality initiatives across large... ...environments . ~ Proven ability to hire, mentor, manage, and develop engineering talent . ~ Strong communication and stakeholder-...Full time
- ...across the CAP microservice fleet and building Cloud Logging dashboards for cross-service visibility. • You'll partner with platform, data, and AI teams to design robust, production-ready solutions and improve system integration. System scalability, resilience, and...Permanent employmentFull timeHybrid workLocal areaWorldwideFlexible hoursShift work
- ...In this role, you will apply advanced machine learning and data science techniques to develop rapid prototypes, deliver business value, and push the boundaries... ...Ability to collect and process necessary data utilizing big data infrastructures and data processing/transformation...Big DataPermanent employmentFull timeHybrid workLocal areaWorldwideFlexible hours
- ...decisions and improve outcomes. Through our unparalleled science, data, technology and laboratory network, we advance diagnostics,... ...completion of all mandatory training. The incumbent is expected to develop a comprehensive understanding of stakeholder requirements and maintain...Full timeFlexible hoursShift work
- ...research and contextual analytics. We empower enterprises with data-driven insights, leveraging advanced AI, machine learning, and cloud... ....- Collaborate with cross-functional teams including AI Developers, Cloud Architects, and Governance leaders to scale operations.Preferred...
- ...seeking a highly skilled and experienced AI/ML Engineer to design, develop, and deploy scalable machine learning and generative AI solutions... ..., AI Agents, and prompt engineering techniques.- Create scalable data pipelines for model training, evaluation, and inference.-...Full time
- ...DeveloperExperience : 7+ YearsLocation : BangaloreEmployment Type : Full-TimeJob Summary : We are looking for an experienced Senior Python Developer with 7+ years of professional experience to join our development team in Bangalore. The ideal candidate should have strong expertise...Full time
- ...production ML systems. The ideal candidate will work closely with Data Scientists, ML Engineers, Software Engineers, and DevOps teams to... ..., and efficient deployments.Key Responsibilities :- Design, develop, and maintain end-to-end MLOps pipelines for model training, validation...Full time
- ...Preferred)- OpenTelemetry, Prometheus, Grafana, ELK (Good to Have)- Design Patterns, Data Structures & Algorithms- Production Support & Performance OptimizationJob Responsibilities : - Design, develop, and deploy scalable Python backend services and APIs.- Build integrations with...
- AI Model design and build : - Work closely with data scientists and business to design and implement AI algorithms, frameworks and architectures... ...scalability, and cost-efficiency. - Building Agentic Systems : Developing intelligent AI agents that can reason, plan, and execute tasks...
- ...: We are looking for an experienced Test Engineer AI to design, develop, and execute comprehensive testing strategies for AI/ML-based applications... ..., perform root-cause analysis, and work closely with developers, data scientists, and ML engineers to resolve issues.- Develop test...
- ...Build and deploy AI applications using Azure OpenAI, Azure AI Foundry, Azure AI Search, Azure AI Services, and Azure Machine Learning.- Develop practical Generative AI solutions using LLMs, Prompt Engineering, RAG, Vector Databases, AI Agents, and Agentic AI.- Create AI-...
- ...SAST, OSS technologies- Ability to perform Python code reviews with minimal assistance- Expertise in monitoring, alerting, reporting, data analysis is desired.- Experience with application build environments like Jenkins, GitHub Actions, Teamcity etc.- DevOps container/orchestration...
- Job Summary : As a Python Developer with Databricks, you will be responsible for developing and maintaining scalable data pipelines, managing cloud environments on Azure, and ensuring smooth integration with APIs.The ideal candidate will be proficient in Python, Databricks...
- Position : AI/ML Engineer (Azure AI, Gen AI)Overall/Total Experience : 3 - 7 years onlyLocation : BengaluruWorking Days : 5 Days from officeNotice Period Requirement : 30 Days (Maximum)Client's Company Size : Startup / Small EnterpriseRoles & Responsibilities :- Azure AI Deployment...
- ...Tooling and MCP Integration: Design and implement MCP-connected data sources, agentic pipelines, and AI orchestration layers using frameworks... ...prompt engineering, MCP servers beyond just using AI tools as a developer.- Experience with financial data systems, portfolio analytics, or...
- ...intuitive and impactful solutions for marketing teams. The company is currently building its core team in Bangalore.Role Overview:As a Data Platform Engineer, you will contribute to building the technology and data foundation for Different AI's products. The role involves working...
- Job Description : Senior Data Scientist (AI/ML).Experience : 5-7 Years.Work mode : On-site... ...to join the team. The role focuses on developing, validating, and deploying machine learning... ...marketing optimization.- Experience with big-data platforms and distributed processing...Big DataFlexible hoursAfternoon shift
- ....Job Description :We are seeking a talented and passionate GenAI Developer with strong expertise in Google Cloud Platform (GCP) and Python to... ...Cloud Functions, BigQuery).- Collaborate with product managers, data scientists, and other engineers to understand requirements and translate...Full time
- ...Description : We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate... ...Engineering or related field.- Knowledge of other big data technologies such as Hadoop, Hive, or Kafka.-...Big Data
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Developer. Be the first to apply!
