Data Engineer with Databricks

Overview

Project overview: Join a cutting-edge data team working on a large-scale data platform that enables advanced analytics and decision-making. This project involves building and optimizing cloud-based data solutions, integrating complex datasets, and ensuring scalable and efficient data pipelines. The team is focused on delivering high-quality data solutions to support scientific and operational advancements in the life sciences industry.

• Responsibilities: Data Architecture & Engineering: Design, develop, and maintain robust, scalable, and efficient data pipelines using AWS and Databricks
• Data Warehousing: Implement and optimize data warehouse solutions to ensure seamless data integration and performance
• ETL & Data Processing: Develop and optimize ETL processes for structured and unstructured data, ensuring data accuracy, consistency, and reliability
• Cloud Technologies: Leverage AWS services (S3, Lambda, Glue, Redshift, etc.) to build scalable and cost-efficient data solutions
• Collaboration & Stakeholder Engagement: Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver actionable data solutions
• Performance Optimization: Monitor, troubleshoot, and improve the performance of data workflows, ensuring minimal downtime and optimal processing speeds
• Security & Compliance: Implement best practices for data governance, security, and compliance in cloud environments

• 5+ years of experience in Data Engineering, with a strong focus on data warehouses, AWS, and Databricks
• Hands-on experience with AWS services such as S3, Glue, Lambda, Redshift, EMR, and IAM
• Strong expertise in Databricks (Delta Lake, Spark, MLflow, Notebooks)
• Proficiency in SQL, Python, and/or Scala for data transformation and pipeline development
• Experience designing and optimizing ETL/ELT processes for large-scale datasets
• Deep understanding of data modeling, partitioning, and performance tuning in cloud environments
• Experience with CI/CD for data pipelines and infrastructure as code (Terraform, CloudFormation)
• Strong problem-solving skills, with a focus on performance and scalability