Big Data Engineer

Overview

Overview

We are seeking a skilled Data Engineer to join our team. The ideal candidate will play a key role in designing, developing, and maintaining data pipelines and systems to support data analytics and machine learning initiatives.

Responsibilities

• Design and implement scalable data pipelines for both batch and real-time processing
• Collaborate with data scientists to deploy machine learning models into production
• Develop and maintain data warehouse solutions for efficient data storage and retrieval
• Create and optimize ETL processes to ensure data quality and integrity
• Work on model training, RESTful APIs, Azure Data Lake, and Big Data technologies
• Utilize Shell Scripting, SQL, VBA, and Agile methodologies for effective data engineering
• Analyze complex datasets to derive actionable insights

Requirements

• Proficiency in SQL for querying and manipulating large datasets
• Experience with Azure Data Lake or similar cloud-based data platforms
• Strong knowledge of data warehouse design principles
• Familiarity with Shell Scripting for automation tasks
• Ability to work in an Agile environment and collaborate with cross-functional teams
• Nice-to-have skills include VBA proficiency, RESTful API development, and experience with analyzing Big Data sets.

Mandatory Skills: Azure data bricks , azure data factory , python and pyspark

Job Type: Contract

Pay: ₹1,000,000.00 - ₹2,500,000.00 per year

Work Location: In person