Senior Data Engineer

Overview

Greeting From Ashra Technologies

We are Hiring

Role; Data engineer

Exp: 5+Years

Location - PUNE / BANGALORE ONLY

Required Skills –

CANDIDATE MUST HAVE HANDS-ON EXPERIENCE

• Python (MUST), Pyspark (MUST) Sql (MUST) , Airflow, Trino, Hive, Snowflake, Agile Scrum
• Excellent Business English communication – Many of our candidate are having articulation and getting escalated.

MUST Required Skills –

• Python, Pyspark, SQL, Airflow, Trino, Hive, Snowflake, Agile Scrum, Spark real time application with ETL pipeline

Good to have–

• Linux, OpenShift, Kubernetes, Superset

Candidate need to do code for the given scenarios(Python, Pyspark,SQL).

Job Description :

• Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
• Basics of Spark, hive etc
• Python Language is MUST (Hands-on).
• SQL (should be able to write complex SQL Queries) is MUST - (Example : join and Aggregate window Functions)
• Spark and Hive: Optimization of Spark/PySpark and Hive apps is MUST
• Core Python (example: apply validation rules to csv file, String comparison, collections and Basic constructs)
• Spark Optimization and Spark Submit command.
• Should be able to write code for the given scenario.
• Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
• Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
• Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
• Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
• Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
• Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
• Stay up-to-date with emerging technologies and best practices in data engineering.
• Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Responsibilities:

• Bachelor's degree in Computer Science, Engineering, or a related field
• 5 + years experience in data engineering, ELT development, and data modeling.
• Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
• Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
• Demonstrated experience in developing custom connectors for data ingestion from various sources.
• Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
• Experience implementing DataOps principles and practices, including data CI/CD pipelines.
• Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
• Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.
• Familiarity with data visualization tools Apache SuperSet and dashboard development.
• Understanding of distributed systems and working with large-scale datasets.
• Familiarity with data governance frameworks and practices.
• Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
• Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
• Experience with Agile development methodologies and working in cross-functional Agile teams.
• Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
• Excellent analytical and problem-solving skills, with a keen attention to detail.
• Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.

Interested share your resume to akshitha@ashratech.com/8688322632

Job Type: Full-time

Pay: ₹1,000,000.00 - ₹2,000,000.00 per year

Schedule:

• Day shift

Work Location: In person