AWS Data Engineer

Role: AWS Data Engineer
Experience: 4 to 6 Years
Location: Pune, Gurgaon & Bangalore – Hybrid
Shift Time: 12:00 PM - 10:00 PM

Clairvoyant is a global technology consulting and services company founded in 2012, headquartered in Chandler, US, and has delivery centers across the globe. We help organizations maximize the value of data by providing data engineering, analytics, machine learning, and user experience consulting and development projects to multiple Fortune 500 clients. Clairvoyant clients rely on its deep vertical knowledge and best-in-class services to drive revenue growth, boost operational efficiencies, and manage risk and compliance. Our team of experts with direct industry experience in data engineering, analytics, machine learning, and user experience has your back every step of the way.

“Our Values: Passion, Continuous Learning, Adaptability, Teamwork, Customer Centricity, Reliability”

Job Summary:

We are seeking a highly motivated and experienced AWS Engineer, to join our MarTech team at the NFL. This position requires an individual with AWS cloud experience and ambition to continually keep up with best practices when it comes to cloud development. The successful candidate must be able to seek out requirements and create best-in-class cloud-native solutions. The engineer must always create solutions that are repeatable, scalable and well-governed. They will deploy and rigorously test solutions to ensure they are robust and secure. The engineer will be responsible for creating and maintaining diagrams associated with any solutions that are deployed into production.

Must have:

• 4+ years of experience as a Data Engineer, ETL Developer, or in a similar role


• Proficiency in SQL and Python for data manipulation, automation, and processing


• 3+ years of extensive experience with AWS services, including: S3, Lambda, Step Functions, Glue, EC2, EMR, SNS, SQS, Redshift, and Athena.


• Strong understanding of data lake, data warehouse, and lakehouse concepts and the ability to design solutions for these architectures.


• Experience with ETL frameworks and tools to build data pipelines.


• Hands-on experience with workflow orchestration tools like Airflow.


• Solid understanding of data governance, security, and compliance best practices.


• Experience in building data models, optimizing databases, and query performance tuning.


• Familiarity with data lake architectures and modern data integration patterns


• Experience building serverless applications in AWS.


• Experience in building real-time/streaming data pipelines.


• Knowledge of enterprise integration patterns.


• Expertise in building high-performance, highly scalable, cloud-based applications.


• Experience with SQL and No-SQL databases.


• Good collaboration and communication skills, highly self-driven, and take ownership.


• Experience in Writing well-documented, Clean, and Effective codes is a must.

Good to have:

• AWS Cloud Certifications.


• Good experience in Airflow, MWAA


• 1-2 years of experience in DBT with Data Modeling, Airflow, SQL, Jinja templating, and packages/macros to build robust, performant, and reliable data transformation and feature extraction pipelines.


• 1-2 years of experience in Airbyte building ingestion modules for streaming, batch.


• Familiarity with Big Data Design Patterns, modeling, and architecture.


• Knowledge of Jinja templating in Python.


• Proficient in SQL, Python, and PySpark.


• Good experience building Real-Time streaming data pipelines with Kafka, Kinesis etc.

Responsibilities:

• Design and develop scalable and efficient ETL pipelines to process structured and unstructured data.


• Work extensively with AWS services including S3, Lambda, Step Functions, Glue, EC2, EMR, SNS, SQS, Redshift, and Athena to manage and optimize data pipelines and workflows.


• Develop, monitor, and troubleshoot workflows using Airflow and other scheduling/orchestration tools.


• Build and maintain data lakes, data warehouses, and lakehouse architectures, ensuring seamless integration and optimal data flow.


• Implement and optimize SQL queries and Python scripts for data extraction, transformation, and loading (ETL).


• Collaborate with data scientists, analysts, and stakeholders to understand business requirements and translate them into robust data solutions.


• Optimize data pipelines for performance, reliability, and scalability to handle growing datasets and evolving business needs.


• Leverage best practices for data security, compliance, and governance in cloud-based environments


• Perform data quality checks and ensure the accuracy and reliability of the data.


• Stay up to date with the latest technologies and trends in data engineering, cloud platforms, and ETL development.

Education:

• BE/B.Tech/MS/M.Tech/ME from a reputed institute.

Every individual comes with a different set of skills and qualities so even if you don’t tick all the boxes for the role today, we urge you to apply as there might be a suitable/unique role for you tomorrow!