Data Engineer

Overview

Role: Data Engineer (Spark Scala)

Role: Data engineer

Notice period - Only immediate Joiners- 15 days ( serving )

What you will do

• Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently
• Ability to write and optimize code in Scala for Spark applications.
• Designing, creating, and maintaining Scala-based applications.

Expertise You'll Bring

• Experience in Scala (mandatory) + Spark + AWS (Mandatory )
• Develop, and maintain scalable ETL pipelines using Spark (with Scala) to process large datasets in a cloud-based environment.
• Implement data transformation logic using Spark for batch and real-time processing, ensuring efficient handling of structured and unstructured data.
• Integrate data from various sources (e.g., databases, cloud storage, APIs) into a centralized data warehouse or lake, leveraging Spark and other relevant technologies.
• Implement data validation, cleansing, and quality checks within the data pipeline to ensure the accuracy, consistency, and integrity of non-sensitive data.
• Optimize Spark jobs for better performance, reduced processing time, and cost efficiency, applying best practices for cluster and resource management.
• Pyspark Databricks lead for next gen migration
• Converting complex requirements from nextgen into pyspark notebook using dataframes and spark sql3.
• integrating airflow with databricks notebooks and troubleshooting issues in migration related to data mismatch.

• Benefits
• Competitive salary and benefits package
• Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Our company fosters a values-driven and people-centric work environment that enables our employees to:

• Accelerate growth, both professionally and personally
• Impact the world in powerful, positive ways, using the latest technologies
• Enjoy collaborative innovation, with diversity and work-life wellbeing at the core
• Unlock global opportunities to work and learn with the industry’s best

Job Type: Full-time

Pay: ₹1,000,000.00 - ₹2,700,000.00 per year

Benefits:

• Flexible schedule
• Health insurance
• Provident Fund
• Work from home

Schedule:

• Day shift

Work Location: In person