Software Engineering LMTS- Big Data Engineer

Description

About the Role

The mission of Unified Intelligence Platform (UIP) is to enable Salesforce teams to deeply understand and optimize their services, operations through data. UIP is a modern, trusted, turn-key data platform built with cutting-edge technologies with an exceptional user experience. Massive amounts of data are generated each day at Salesforce. It is critical to process and store large volumes of data efficiently and enable the users to discover and analyze the data easily.

UIP is a modern, cloud-based data platform built on advanced data lake engines like Spark and Trino, incorporating a diverse suite of tools and technologies—including Airflow, DBT, Jupyter Notebooks, Sagemaker, Iceberg, and Open Metadata—for efficient data processing, storage, querying, and management. With curated datasets, we empower machine learning and AI use cases, enabling both model development and inference. Our team is fast-paced, agile, and highly collaborative, working across all areas of our tech stack to provide critical business services, support complex computing requirements, drive big data analytics, and pioneer cutting-edge engineering solutions in the cloud, defining the next generation of trusted enterprise computing.

Who are we looking for?

• Passionate about tackling big data challenges in distributed systems.
• Highly collaborative, working across teams to ensure customer success.
• Drive end-to-end projects that deliver high-performance, scalable, and maintainable solutions.
• Adaptable and versatile, taking on multiple roles as needed, whether as a Platform Engineer, Data engineer, Backend engineer, DevOps engineer, or support engineer. for the platform and customer success
• Strong foundation in software engineering, with the flexibility to work in any programming language.
• Committed to engineering excellence, consistently delivering high-quality products.
• Open and respectful communicator, fostering transparency and team alignment.
• Embraces a growth mindset, continuously learning and seeking self-improvement.
• Engages actively in support channels, providing insights and collaborating to support the community.
• Champions a Service Ownership model, minimizing operational overhead through automation, monitoring, and alerting best practices.

Job Responsibilities:

• Lead the architecture, design, development, and support of mission-critical data and platform services, ensuring full ownership and accountability.
• Drive multiple self-service, metadata-driven data pipelines, services, and applications to streamline ingestion from diverse data sources into a multi-cloud, petabyte-scale data platform.
• Collaborate closely with product management and client teams to capture requirements and deliver scalable, adaptable solutions that drive success.
• Architect robust data solutions that enhance ingestion, processing, quality, and discovery, embedding security and governance from the start.
• Promote a service ownership model, designing solutions with extensive telemetry and control mechanisms to streamline governance and operational management.
• Develop data frameworks to simplify recurring data tasks, ensure best practices, foster consistency, and facilitate tool migration.
• Implement advanced data quality services seamlessly within the platform, empowering data analysts, engineers, and stewards to continuously monitor and uphold data standards.
• Build Salesforce-integrated applications to monitor and manage the full data lifecycle from a unified interface.
• Establish and refine CI/CD processes for seamless deployment of platform services across cloud environments.
• Oversee and maintain key components of the UIP technology stack, including Airflow, Spark, Trino, Iceberg, and Kubernetes.
• Collaborate with third-party vendors to troubleshoot and resolve platform-related software issues.
• Architect and orchestrate data pipelines and platform services optimized for multi-cloud environments (e.g., AWS, GCP).