Lead Data Engineer - SQL, Python, PySpark, Databricks

We are seeking a skilled Data Quality Engineer to ensure the accuracy, completeness, consistency, and reliability of data across our systems. The Data Quality Engineer will play a key role in developing and enforcing data quality standards, identifying issues, and resolving data quality challenges in the organization.

Key Responsibilities:

• Develop and maintain data quality standards, processes, and procedures.
• Identify, profile, and resolve data quality issues within complex datasets.
• Perform data validation and reporting to ensure accuracy and completeness.
• Work closely with cross-functional teams to define and enforce data quality requirements.
• Continuously monitor data quality and implement improvements to enhance data reliability and consistency.

Must-Have Skills:

• Advanced SQL: Strong proficiency in SQL for querying, profiling, validation, and reporting on large datasets.
• PySpark: Expertise in data processing, transformation, validation, and analysis using PySpark.
• Python: Proficiency in Python for automating tasks, writing scripts, and building data solutions.
• Databricks: Experience using Databricks to collaborate on analytics, scale data pipelines, and automate processes using Spark and related technologies.

Nice-to-Have Skills:

• Google Cloud Platform (GCP): Familiarity with GCP tools such as BigQuery, Google Composer, and Spark Serverless is a plus as these are key components of our federated data platform.
• DBT: Knowledge of DBT (Data Build Tool) is advantageous as we are transitioning to this framework.
• Great Expectations: Experience with the Great Expectations data quality tool for performing data quality checks is a plus.

SQL, Pyspark, Python, Databricks