Sr. Data Engineer

Overview

Job Title: Senior Data Engineer

Industry: Manufacturing (Steel/Metal Industry) or similar Industry.

Location: Pune

Job Type: Full-Time

Job Description

We are seeking a skilled and experienced Senior Data Engineer to join our team. The ideal candidate will play a crucial role in designing, building, and maintaining robust data pipelines and architectures to support advanced analytics and decision-making processes in the manufacturing industry, with a strong focus on steel/metal production.

Key Responsibilities

· Design, build, and manage scalable ETL/ELT pipelines to process structured and unstructured data from sources such as IoT devices, MES, SCADA, LIMS, and ERP systems.

· Develop scalable, high-performance data architectures for analytics and data science initiatives.

· Develop and optimize real-time and batch data processing workflows to support critical business needs, ensuring low latency and high throughput.

· Implement real-time data streaming pipelines using tools like Kafka, Azure Event Hub, or similar technologies, ensuring low latency.

· Architect and maintain efficient data storage solutions, including data lakes and data warehouses, and implement modern methodologies like medallion architecture to ensure scalability and efficient data processing.

· Work with cloud platforms (preferably Azure) to design and deploy robust data solutions.

· Monitor and optimize data workflows, implementing alerting mechanisms to ensure quality, reliability, and stability.

· Optimize data pipelines to improve performance and support both batch and real-time processing.

· Automate data workflows using DevOps practices while ensuring security and compliance with industry standards.

· Collaborate with data scientists, analysts, process engineers, and stakeholders to deliver data solutions aligned with manufacturing KPIs.

· Ensure high data availability, reliability, and proactive resolution of quality issues.

· Implement best practices for data governance and regulatory compliance.

Desired Skills and Qualifications

· Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.

· 4+ years of experience in data engineering, preferably in the manufacturing industry.

· Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud (preferably Azure). Certifications are an added advantage.

· Strong experience with data engineering tools and frameworks such as Apache Spark, Hadoop, Kafka, or Flink.

· Proficiency in programming languages like Python, Java, PySpark, or Scala.

· Expertise in relational databases (e.g., SQL Server, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

· Knowledge of data warehouse and lakehouse methodologies, including medallion architecture.

· Proven expertise in designing and implementing ETL pipelines using tools such as Azure Data Factory (ADF) or Databricks to orchestrate workflows

· Familiarity with manufacturing systems and protocols such as SCADA, MES, and OPC-UA.

· Expertise in data pre-processing techniques, including data cleaning, normalization, deduplication, transformation, encoding, scaling, and type conversion.

· Familiarity with predictive analytics, AI/ML applications, and IoT in manufacturing is a plus.

· Strong analytical and problem-solving skills with attention to detail.

· Excellent communication and collaboration skills to work effectively with technical and non-technical stakeholders.

· Excellent communication and collaboration skills, with a proven ability to work effectively as a team player.

· Comfortable working in a dynamic, fast-paced startup environment, adapting quickly to changing priorities and responsibilities.

Job Type: Full-time

Pay: ₹445,288.26 - ₹1,681,268.64 per year

Experience:

• data engineering, manufacturing industry.: 5 years (Preferred)

Work Location: In person