Senior Data Engineer

Job Summary: We are seeking a skilled and motivated Data Engineer with hands-on experience in Databricks and Airflow to join our Engineering team. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that enable efficient data collection, processing, and storage using modern cloud-based technologies. The Data Engineer will work closely with Engineering, Data Scientists, Analysts, and Business Stakeholders to enable data-driven decision-making and actionable insights. Key Responsibilities: Design, develop, and maintain efficient, reliable, and scalable data pipelines using Databricks , Apache Spark, Apache Airflow and related technologies to process large volumes of structured and unstructured data. Build and manage ETL workflows or jobs to transform raw data into optimized, analytics-ready datasets. Collaborate with Data Scientists, Analysts, and Business Teams to gather data requirements and translate them into robust technical solutions. Develop and maintain data lakes and data warehouses using Databricks, cloud storage (S3, ADLS), and query engines (Delta Lake, Redshift, BigQuery). Implement automated data quality checks and monitoring mechanisms to ensure data accuracy and reliability. Optimize large-scale distributed data processing jobs for performance, scalability, and cost efficiency. Implement secure, governed, and auditable data access and sharing practices in line with company policies and regulations (GDPR, HIPAA). Monitor and troubleshoot Databricks clusters, jobs, and performance issues. Document data models, pipeline architecture, and processes for clarity, maintainability, and handover. Stay updated on new features and best practices for Databricks and related data engineering technologies. Required Qualifications: Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related