Data Engineer (AWS)

Overview

Role: AWS Engineer

Location: Gurugram

Mode: Hybrid

Type: Permanent

Job Description:

We are seeking a talented and motivated Data Engineer with requisite years of hands-on experience to join our growing data team. The ideal candidate will have experience working with large datasets, building data pipelines, and utilizing AWS public cloud services to support the design, development, and maintenance of scalable data architectures. This is an excellent opportunity for individuals who are passionate about data engineering and cloud technologies and want to make an impact in a dynamic and innovative environment.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and optimize end-to-end data pipelines for extracting, transforming, and loading (ETL) large volumes of data from diverse sources into data warehouses or lakes.
• Cloud Infrastructure Management: Implement and manage data processing and storage solutions in AWS (Amazon Web Services) using services like S3, Redshift, Lambda, Glue, Kinesis, and others.
• Data Modeling: Collaborate with data scientists, analysts, and business stakeholders to define data requirements and design optimal data models for reporting and analysis.
• Performance Tuning & Optimization: Identify bottlenecks and optimize query performance, pipeline processes, and cloud resources to ensure cost-effective and scalable data workflows.
• Automation & Scripting: Develop automated data workflows and scripts to improve operational efficiency using Python, SQL, or other scripting languages.
• Collaboration & Documentation: Work closely with data analysts, data scientists, and other engineering teams to ensure data availability, integrity, and quality. Document processes, architectures, and solutions clearly.
• Data Quality & Governance: Ensure the accuracy, consistency, and completeness of data. Implement and maintain data governance policies to ensure compliance and security standards are met.
• Troubleshooting & Support: Provide ongoing support for data pipelines and troubleshoot issues related to data integration, performance, and system reliability.

Qualifications:

Essential Skills:

• Experience: 8+ years of professional experience as a Data Engineer, with a strong background in building and optimizing data pipelines and working with large-scale datasets.
• AWS Experience: Hands-on experience with AWS cloud services, particularly S3, Lambda, Glue, Redshift, RDS, and EC2.
• ETL Processes: Strong understanding of ETL concepts, tools, and frameworks. Experience with data integration, cleansing, and transformation.
• Programming Languages: Proficiency in Python, SQL, and other scripting languages (e.g., Bash, Scala, Java).
• Data Warehousing: Experience with relational and non-relational databases, including data warehousing solutions like AWS Redshift, Snowflake, or similar platforms.
• Data Modeling: Experience in designing data models, schema design, and data architecture for analytical systems.
• Version Control & CI/CD: Familiarity with version control tools (e.g., Git) and CI/CD pipelines.
• Problem-Solving: Strong troubleshooting skills, with an ability to optimize performance and resolve technical issues across the data pipeline.

Desirable Skills:

• Big Data Technologies: Experience with Hadoop, Spark, or other big data technologies.
• Containerization & Orchestration: Knowledge of Docker, Kubernetes, or similar containerization/orchestration technologies.
• Data Security: Experience implementing security best practices in the cloud and managing data privacy requirements.
• Data Streaming: Familiarity with data streaming technologies such as AWS Kinesis or Apache Kafka.
• Business Intelligence Tools: Experience with BI tools (Tableau, Quicksight) for visualization and reporting.
• Agile Methodology: Familiarity with Agile development practices and tools (Jira, Trello, etc.)

Job Type: Permanent

Pay: ₹3,000,000.00 - ₹4,000,000.00 per year

Benefits:

• Work from home

Experience:

• Pyspark: 3 years (Required)
• Data Engineering: 8 years (Required)
• ETL Processes: 3 years (Required)
• ETL and/or data pipeline implementations: 4 years (Required)
• AWS services (Airflow, S3, Athena, Kinesis, Redshift): 4 years (Required)
• Hadoop, Hive, Athena, Presto: 3 years (Required)

Work Location: In person