Senior Azure Data Engineer

Overview

Duties and Responsibilities:

• Demonstrate skills in abstract development frameworks.
• Demonstrate accountability to assigned work & timelines.
• Demonstrated leadership skills.
• Demonstrated technical competency for the role.
• Demonstrated timely delivery of assigned tasks and effective communication on project status and escalation when needed.

Required Skills/Certifications:

Minimum Requirements:

5+ years of experience overall in IT

Current - 2+ years of experience in Azure Cloud as a Data Engineer

Current - 1.5+ years of hands-on experience on Databricks/Azure Databricks

Proficient in Python/PySpark

Proficient in SQL/TSQL

Proficient in Data Warehousing Concepts (ETL/ELT, Data Vault Modelling, Dimensional Modelling, SCD, CDC)

Primary Skills:

Azure Cloud, Databricks, Azure Data Factory, Azure Synapse Analytics, SQL/TSQL, PySpark, Python

Technical Expertise:

ETL/ELT Pipelines: Design, implementation, and optimization.

Azure ADF and Synapse: Proficiency in creating pipelines, developing notebooks, and performing data transformations.

Databricks: Hands-on experience with creating notebooks and performing data transformations.

Familiarity with Dataflows, Data Wrangling, and integration with REST APIs.

Experience with Parquet and Delta file processing, along with optimization techniques

Familiarity with Azure DevOps and CI/CD

Concepts and Methodologies:

Data Warehousing: Comprehensive understanding of concepts and implementation

Data Modelling Techniques: Familiarity with Data Vault (if not, then Dimensional Modelling Technique)

SCD and CDC: Strong grasp of Slowly Changing Dimensions (SCD) and Change Data Capture (CDC) methodologies

Agile Methodology

Programming Skills:

Python/PySpark/Spark SQL – Strong proficiency in any of these

SQL/T-SQL

A must-have:

Demonstrates strong association with team dynamics and the ability to maintain harmony in a collaborative environment.

Proven ability to be a team player who contributes to shared goals while supporting peers.

Good to have:

Basic knowledge of Delta Tables, Delta Live Tables (DLT), Partitioning, and Optimization.

Knowledge of Unity Catalog and Job Clusters is an added advantage.

Familiarity with Azure Monitor, Log Analytics, and Spark UI is an added advantage.

It is good to know real-time data ingestion and streaming pipelines.

Experience with framework-driven projects (Metadata framework, Audit control framework)

Relational and No-SQL Databases

Certifications:

AZ 900 – Azure fundamentals

DP 203 – Azure Data Engineer

Databricks Associate Data Engineer

Job Type: Full-time

Pay: ?3,000,000.00 - ?5,000,000.00 per year

Experience:

• total work: 5 years (Preferred)

Work Location: In person