Senior Data Engineer - Azure Data Services

Role : Senior Data EngineerJob Overview : We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives. The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.Key Responsibilities : - Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes- Create and maintain data models, schemas, and documentation to ensure data quality and consistency- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed- Ensure data security, privacy, and compliance with industry standards and regulations- Mentor junior team members and provide technical guidance on data engineering best practices- Stay current with emerging technologies and industry trends in data engineering and cloud computing- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables- Design and implement data governance frameworks and data quality monitoring solutionsRequired Qualifications : - 8+ years of total professional experience in software development, data engineering, or related fields- 7+ years of hands-on experience in data engineering roles with proven expertise in designing and implementing data solutions- Extensive experience with Azure Data Factory (ADF) for building and orchestrating data pipelines- Strong proficiency in Azure Databricks and Apache Spark for large-scale data processing- Hands-on experience with Lake flow Declarative Pipelines and modern data pipeline architectures- Advanced SQL skills with experience in query optimization and database design- Proficient in PySpark and Python for data processing and transformation tasks- Experience with Azure cloud services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics- Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT processes- Knowledge of data governance, data quality, and data security best practices- Experience with version control systems (Git) and CI/CD pipelines- Excellent problem-solving skills and attention to detail- Strong communication skills and ability to work effectively in cross-functional teamsPreferred Qualifications : - Bachelor's degree in Computer Science, Engineering, Mathematics, or related technical field- Azure certifications such as Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert- Experience with additional Azure services like Azure Stream Analytics, Azure Event Hubs, or Azure Cosmos DB- Knowledge of machine learning pipelines and MLOps practices- Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform- Familiarity with containerization technologies (Docker, Kubernetes)- Experience with real-time data processing and streaming technologies- Knowledge of other programming languages such as Scala or Java- Experience with data visualization tools and business intelligence platforms- Previous experience in mentoring and leading technical teamsTechnical Skills : - Cloud Platforms : Microsoft Azure (primary focus)- Data Processing : Azure Databricks, Apache Spark, PySpark- Data Orchestration : Azure Data Factory (ADF), Lake flow Declarative Pipelines- Databases : SQL Server, Azure SQL Database, Azure Synapse Analytics, Azure Data Lake- Programming Languages : Python, SQL, PySpark- Development Tools : Git, Azure DevOps, Visual Studio Code- Data Formats : JSON, Parquet, Avro, CSV, XML- Monitoring & Logging : Azure Monitor, Application InsightsWhat We Offer : - Competitive salary and comprehensive benefits package- Opportunity to work with cutting-edge Azure technologies and data engineering tools- Professional development opportunities and certification support- Collaborative and innovative work environment- Flexible work arrangements and work-life balance- Health, dental, and vision insurance- Retirement savings plan with company matching- Paid time off and holidays- Learning and development budget for continuous skill enhancementEmployment Type : Full-time, PermanentApplication Process : Interested candidates are encouraged to submit their resume along with a cover letter highlighting their relevant experience with Azure data engineering technologies. Please include specific examples of projects involving Azure Data Factory, Databricks, and data pipeline development. (ref:hirist.tech)