Lead Data Engineer/Manager - Databricks

About the Role:We are seeking a highly capable, hands-on Lead Data Engineer (Manager) with deep expertise in Databricks to lead the design and delivery of scalable data solutions.In this role, you will:- Lead engineering teams across distributed environments- Own end-to-end delivery of modern data solutions- Work closely with business and technical stakeholders- Drive innovation across data platformsYou will play a key role in enabling clients to unlock value from their data while ensuring scalability, reliability, and performance.Key Responsibilities:- Lead, manage, and grow a distributed Databricks engineering team- Own end-to-end delivery of data engineering solutions (design - deployment - optimisation)- Provide hands-on technical leadership in building scalable data pipelines and data products- Design and implement Lakehouse architecture aligned with enterprise standards- Translate business requirements into robust, scalable data solutions- Establish and enforce best practices (code quality, testing, documentation, CI/CD)- Ensure governance, security, and compliance across data platforms- Act as escalation point for complex technical challenges- Collaborate with architecture, DevOps, and platform teams- Drive adoption of new Databricks features, tools, and AI capabilitiesSkills & Experience:Essential:- Proven experience leading and managing distributed data engineering teams- Strong hands-on expertise with Databricks (Spark PySpark/SQL)- Experience building data platforms on Azure (ADLS, ADF, Databricks) or similar cloud ecosystems- End-to-end data engineering expertise (ingestion, transformation, modelling, serving)- Deep understanding of Lakehouse architecture (Delta Lake, Medallion - Bronze/Silver/Gold layers)- Strong stakeholder management and business translation skills- Experience establishing engineering standards and code review practices- Expertise in CI/CD & DevOps (Azure DevOps, Git, Terraform)- Excellent leadership, mentoring, and communication skills- Relevant Databricks certificationsDesirable:- Experience in onshore/offshore delivery models and vendor management- Exposure to both Databricks and Snowflake- Knowledge of Unity Catalog and data governance frameworks- Experience with real-time/streaming data (Kafka, Structured Streaming)- Exposure to AI/ML or GenAI workloads (feature engineering, RAG, model pipelines)- Experience working in large, complex enterprise environments- Experience in cost optimisation and performance tuning- Exposure to Azure AI (OpenAI, Cognitive Services, AI Foundry)- Experience contributing to a Data/AI Centre of ExcellenceBehavioural Competencies:- Collaborative: Builds strong cross-functional relationships- Influential: Guides decisions and drives alignment- Delivery-focused: Owns outcomes and drives execution- Structured thinker: Creates clarity in complex environments- Adaptable: Thrives in dynamic and evolving contexts- Detail-oriented: Ensures accuracy and quality- Proactive: Anticipates risks and mitigates early- Continuous improvement mindset: Drives innovation and optimization- Resilient: Performs effectively under pressureEducation & Certifications:- Microsoft Azure Certifications- Databricks Certified (Associate / Professional)- SnowPro Data Engineer (Advanced)- Microsoft Fabric Certifications (DP-600 / DP-700) (ref:hirist.tech)