Skan.ai - Senior Data Engineer - Apache Flink

Be at the Forefront of the Agentic AI Revolution :At Skan AI, you'll be part of the team pioneering the context engine for human and agentic execution, bringing context from enterprise operators, systems, and processes to power how the world's largest organizations execute their most complex, mission-critical work.Why Join Skan AI :We're in hyper-growth mode at exactly the right moment in history. As enterprises race to adopt agentic AI, we're uniquely positioned to deliver the clear signal they desperately need: a platform that trains and grounds AI Agents in trillions of real execution signals, enabling reliable, compliant automation of their most complex processes. Backed by Dell Technologies Capital and other leading investors, we're the only company that can bridge the gap between AI's promise and enterprise reality, making us perfectly positioned to define the agentic era for modern enterprises. Our diverse, collaborative team of 250+ innovators is solving category-defining challenges at the intersection of AI, process intelligence, and enterprise work. Diverse perspectives fuel breakthrough thinking, cross-functional collaboration is the norm, and our work directly transforms how Fortune 500 companies operate. We are shaping the future of work itself.Role Overview :We are looking for an experienced Apache Flink ETL Lead to own the design, development, and delivery of our real-time data pipeline infrastructure. You will lead data engineering efforts responsible for synchronising data from PostgreSQL to StarRocks using Apache Flink's CDC and streaming pipeline capabilities. You will be hands-on in development while providing technical leadership and driving best practices across the engineering organisation. This is an individual contributor and lead role. We value hands-on engineers who can also mentor and drive delivery. If you love debugging Flink checkpoints as much as you enjoy growing a team, we want to hear from you.Key Responsibilities :1. Pipeline Development and Architecture :- Design, build, and maintain high-performance Apache Flink ETL/ELT pipelines for real-time data synchronisation from PostgreSQL to StarRocks.- Architect robust CDC (Change Data Capture) solutions using Flink CDC connectors and Debezium for PostgreSQL source ingestion.- Implement Flink SQL and DataStream API pipelines for complex transformation logic, aggregations, and data enrichment.- Develop and maintain custom Flink connectors and sinks for StarRocks integration using the StarRocks Flink Connector.- Design fault-tolerant, exactly-once or at-least-once pipelines with appropriate checkpointing and state management strategies.- Evaluate and implement schema evolution strategies to handle upstream PostgreSQL schema changes gracefully.2. Performance Optimisation :- Profile and tune Flink job performance: parallelism settings, task manager memory, operator chaining, and back-pressure management.- Optimise StarRocks loading strategies (Stream Load vs. Routine Load) for high-throughput ingestion.- Monitor pipeline latency and throughput SLAs; proactively identify and resolve bottlenecks.- Implement efficient watermarking and windowing strategies for time-sensitive data flows.- Manage Flink state backends (RocksDB / heap) and configure appropriate TTLs to control state size.3. Troubleshooting and Reliability :- Own end-to-end pipeline reliability : diagnose and resolve issues including data lag, job failures, checkpoint timeouts, and OOM errors.- Establish alerting and observability for pipeline health using Flink metrics, Prometheus, and Grafana (or equivalent).- Define and implement data quality checks, reconciliation processes, and dead-letter queue (DLQ) strategies.- Perform root-cause analysis on data discrepancies between PostgreSQL source and StarRocks target.- Maintain comprehensive runbooks for common failure scenarios and recovery procedures.4. Team Leadership and Delivery :- Lead Data Engineers : assign tasks, conduct code reviews, and ensure delivery against sprint goals.- Mentor engineers on Flink internals, best practices, and performance considerations.- Collaborate with data consumers (analysts, BI teams) to understand requirements and translate them into pipeline specifications.- Drive technical decisions on tooling, frameworks, and deployment strategies (Flink on Kubernetes on GCP).- Maintain technical documentation including architecture diagrams, data flow documentation, and operational guides.5. Deployment and DevOps : - Manage Flink cluster deployment and configuration on Kubernetes (GKE) using Helm charts and Harness CI/CD pipelines.- Build and maintain CI/CD pipelines using Harness for Flink job packaging, testing, and deployment to GKE.- Manage Flink cluster configuration using Helm charts; maintain Helm values and chart templates for environment-specific configurations.- Coordinate with infrastructure and DBA teams for PostgreSQL slot management and StarRocks table design.Qualifications and Experience :Technical Skills (Must Have) :- 5+ years of experience in data engineering with a focus on ETL/ELT pipeline development.- 2+ years of hands-on production experience with Apache Flink (Flink SQL and/or DataStream API).- Strong proficiency in Java, Scala, or Python for Flink job development.- Experience with Change Data Capture (CDC) patterns : Flink CDC, Debezium, or equivalent.- Practical experience with PostgreSQL as a data source, including replication slots and WAL configuration.- Demonstrated experience with columnar OLAP databases (StarRocks, Doris, ClickHouse, or similar).- Solid understanding of distributed systems concepts: fault tolerance, exactly-once semantics, state management, and watermarking.- Experience with Kafka or similar message brokers as part of streaming architectures.Leadership and Soft Skills :- Proven ability to lead engineering teams, including task planning, code review, and mentoring.- Strong analytical and problem-solving skills with the ability to debug complex distributed pipeline issues.- Excellent written and verbal communication skills with the ability to document technical decisions clearly.- Self-driven with the ability to work autonomously and manage priorities in a fast-paced environment.Preferred Qualifications :- Hands-on experience with StarRocks Flink Connector and StarRocks primary-key table designs.- Hands-on experience with Flink on Kubernetes (GKE) using Flink Kubernetes Operator or native K8s mode.- Experience with Harness CI/CD and Helm chart management for data workloads.- Experience with Flink Table API and Flink SQL for unified batch and stream processing.- Knowledge of Apache Iceberg, Delta Lake, or Hudi for lakehouse architectures.- Experience with orchestration tools such as Apache Airflow or Prefect for hybrid batch/streaming workflows.- Familiarity with observability stacks : Prometheus, Grafana, and Flink metrics reporters.- Prior experience in a technical lead or senior engineer role with delivery accountability.- Understanding of data modelling best practices for analytical workloads in StarRocks.Domain Knowledge (Nice to Have) :- Experience in data-intensive domains such as data mining, large-scale data processing, or analytical platform engineering.- Understanding of end-to-end data process flows: from source system ingestion through transformation, aggregation, and consumption layers.- Familiarity with data governance, lineage tracking, and metadata management.- Exposure to real-time analytics use cases such as dashboards, operational reporting, or data exploration pipelines.Skan AI is an equal opportunity employer committed to building a diverse, inclusive, and respectful workplace around the world. We do not discriminate based on race, color, religion or belief, sex (including pregnancy, sexual orientation, gender identity, or gender expression), national origin, ancestry, age, disability, medical condition, genetic information, marital or family status, military or veteran status, or any other characteristic protected by applicable laws in the locations where we operate. We welcome people from all backgrounds and provide reasonable accommodations throughout the hiring process. (ref:hirist.tech)