Principal Architect - Data Platform

Principal Architect : Experience : 1215 years.Location : Pune (onsite).About the Role : We are looking for a Principal Architect to lead the design and development of a Lakehouse solution end to end.This is an opportunity to architect and build a reusable data platform framework from the ground up.You will own the technical architecture, write code for the core framework, mentor a growing engineering team, and work closely with leadership on the overall technology direction.The ideal candidate is a hands-on architect with a proven track record of building complex frameworks and platforms using open-source big-data technologies such as Spark, Hadoop, and modern Lakehouse stacks (e.g., Iceberg, Trino, Databricks, etc).Key Responsibilities : 1. Architecture (40%) : - Own the end-to-end architecture of the Lakehouse platform and its reusable framework components.- Design a metadata-driven data engineering framework built on Spark SQL and PySpark.- Architect a pipeline engine supporting dynamic SQL generation, incremental loads, CDC, SCD implementations, and schema evolution.- Design Kubernetes-native deployment with multi-tenancy, resource isolation, scaling strategy, high availability, and disaster recovery.- Define API contracts, plugin/extension architecture, and integration patterns.- Drive and document key architecture decisions (ADRs).2. Hands-on Engineering (40%) : - Build core framework components : pipeline engine, dynamic SQL generation, data quality checks, error handling and retry mechanisms.- Implement observability across the platform.- Optimize Spark job performance (partitioning, shuffle tuning, memory management, query plans).- Establish and implement CI/CD pipelines, automated testing (unit and integration), and release practices.- Embed data governance and security into the platform (access control, auditability, cataloging).3. Engineering Leadership (20%) : - Mentor and guide junior engineers; conduct design and code reviews.- Define coding standards, engineering best practices, and documentation norms.- Contribute to broader technical direction and team growth.- Continuously raise the bar on engineering quality and delivery practices.Required Experience : - 12 to 15 years of software engineering experience overall.- 8+ years in data platform engineering.- 5+ years as an Architect on Spark/Hadoop-based platforms.- Proven success building complex, reusable frameworks or platforms (not just applications) using open-source technologies.- Expert-level proficiency in Python, PySpark, Spark SQL, and Spark internals.- Deep knowledge of Apache Iceberg, distributed systems, Kubernetes, Docker, REST APIs, microservices, Linux, Git, and CI/CD.Technical Stack : - Data Platform : Apache Spark, Apache Iceberg, Apache Ozone, Apache Ranger, Trino, Kafka, Airflow.- Platform Engineering : Kubernetes-native deployments, Helm, Terraform, Prometheus, Grafana, containerization, multi-tenant architecture, storage management, HA/DR.- Framework Capabilities You'll Build : JSON-driven pipelines, metadata-driven ETL, dynamic SQL generation, incremental loads & CDC, SCD handling, schema evolution, data quality, lineage, plugin architecture, unit testing, performance optimization.What We're Looking For : - A hands-on architect who designs systems and writes the code that proves the design.- Strong architecture and design instincts with the judgment to keep frameworks simple and extensible.- Complex problem-solving ability and speed in picking up new technologies.- Experience with Agile and spec-driven development.- Clear communication able to explain architecture decisions to engineers and leadership alike.Nice to Have : - Contributions to open-source data projects (Spark, Iceberg, Trino, etc.).- Experience with Databricks or comparable managed Lakehouse platforms.- Prior experience taking a platform or framework from concept to production. (ref:hirist.tech)