Azure Databricks Engineer

Role Overview : We are seeking a seasoned Azure Databricks Engineer to join our high-performing data engineering team in Bangalore/Chennai. In this role, you will architect and maintain robust data pipelines, transforming complex raw data into actionable insights that drive strategic decision-making. You will collaborate closely with cross-functional teams, including data scientists, business analysts, and product stakeholders, to ensure data integrity and performance at scale. By leveraging your deep technical expertise, you will play a pivotal role in optimizing our cloud data infrastructure, directly impacting the efficiency of our analytics platforms and the quality of data-driven outcomes for our global clients.Key Responsibilities : - Design and implement scalable end-to-end data pipelines using Azure Data Factory and Azure Databricks to support high-volume data processing requirements.- Develop complex data transformation logic using PySpark and Apache Spark to ensure high-performance data ingestion and cleaning for downstream consumption.- Optimize SQL queries and Spark jobs to improve system performance, reduce latency, and minimize cloud infrastructure costs for the organization.- Partner with data architects and business stakeholders to translate complex business requirements into technical data solutions that meet evolving organizational needs.- Maintain rigorous standards for data quality, security, and governance across all Azure-based data platforms to ensure compliance and reliability.- Mentor junior team members and conduct code reviews to foster a culture of technical excellence and continuous improvement within the engineering team.Required Skillset : - Demonstrated expertise in building and managing large-scale data solutions within the Azure ecosystem, with a deep understanding of Azure Databricks and Azure Data Factory.- Advanced proficiency in PySpark and Apache Spark for distributed computing, coupled with the ability to write highly optimized SQL for complex data manipulation.- Strong analytical and problem-solving abilities, with a proven track record of troubleshooting performance bottlenecks in distributed data environments.- Excellent communication skills, with the ability to articulate technical concepts clearly to non-technical stakeholders and collaborate effectively in a hybrid work environment.- A proactive mindset toward learning new technologies and adapting to changing business requirements in a fast-paced, professional setting.- A Bachelors or Masters degree in Computer Science, Information Technology, or a related quantitative field, supported by 6-12 years of relevant industry experience. (ref:hirist.tech)