Bangalore, Karnataka, India
Information Technology
Full-Time
Wolters Kluwer
Overview
Job Description
The Cloud Data Engineer will be responsible for designing, implementing end to end solutions in Azure Enterprise Data Lake and optimizing data pipelines, ensuring the availability, performance, scalability and security of large-scale data processing systems. This role requires deep understanding of big data technologies, data architecture, infrastructure, CI/CD and data engineering best practices. Experience with Unity Catalog is a bonus. The Cloud Data Engineer will work closely with architects, leads and other stakeholders to support data driven decision making processes.
Experience Required
- Minimum 8+ years of experience with strong hands-on experience as a senior Data Engineer or related role
- 5+ years of demonstrated experience in developing Big Data solutions that support business analytics and data science teams
- 3-5 years of proficient Data ingestion end-to-end implementation of projects in Azure Enterprise Data Lake, Azure Functions, Databricks, Blob Storage, Cosmos DB, Azure stream analytics, Python, SQL
- Extensive hands-on experience implementing Lake house architecture using Data bricks Data Engineering platform, SQL, Unix shell scripting, SQL Analytics, Delta Lake, and Unity Catalog
- Good understanding of spark architecture with Databricks structured streaming, setting up Azure with Databricks, managing clusters in Databricks
- Experienced in DevOps and deployment automations with Azure DevOps - ARM, YAML, Terraform
- Ability to research the latest trends and propose advanced tooling/solutions for Cloud Data Lake & Data Science platforms
- Experience with business intelligence and analytics tools such as OBIEE, PowerBI or Tableau
- Collaborate applications teams/Business users to develop new pipelines with Cloud data migration methodologies and processes including tools like Azure Data Factory, Event Hub, etc
Roles & Responsibilities
- Drive and implement design of data schemas, drive cloud data lake platform design decisions and development standards and maintain data pipelines for data ingestion, processing, and transformation in Azure.
- Drive analysis, architecture, design, governance and development of data warehouse, data lake, and business intelligence solutions
- Using a combination of Azure Data factory, Azure Blob Storage, T-SQL, Pyspark and Azure Databricks should be able to Extract, transform and load from sources system to Azure Data Storage services
- By ensuring data quality, consistency and reliability, Integrate data from various sources,
- Define data requirements, gather and mine large scale structured and unstructured data, and validate data using various tools in a cloud environment.
- Manage and optimize Azure Enterprise data Lake to achieve efficient data storge and processing.
- Develop and optimize ETL processes using Databricks and related tools like Apache Spark
- Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
Similar Jobs
View All
10 Hours ago
MTS II - Software Engineer
Information Technology
- 4 - 7 Yrs
- Pune
MAJOR RESPONSIBILITIES
• Design, implement, integrate, and verify software applications and tools using JavaScript, NodeJS, and C++.
• Enhance, optimize, and improve the efficiency and robustness of current software, with a particular focus on OSS ...
1 Day ago
Business Advisory Analyst
Information Technology
- Bangalore, Karnataka, India
Skill required: Banking Services - Core BankingDesignation: Business Advisory AnalystQualifications:BBA/BCom/Master of Business AdministrationYears of Experience:3 to 5 yearsAbout AccentureAccenture is a global professional services company with lea...
1 Day ago
Front End Developer
Information Technology
- Bangalore, Karnataka, India
Position Title: Front End DeveloperCompany: Johnson Controls (JCI)Location: BangaloreJob Summary: We are seeking a talented Front End Developer with 4-7 years of experience to join our dynamic team. The ideal candidate will have a strong background ...
1 Day ago
Database Engineer III (Big Data)
Information Technology
- Bangalore, Karnataka, India
LivePerson (NASDAQ: LPSN) is the global leader in enterprise conversations. Hundreds of the world’s leading brands — including HSBC, Chipotle, and Virgin Media — use our award-winning Conversational Cloud platform to connect with millions of consume...
1 Day ago
Data Scientist Manager
Information Technology
- Bangalore, Karnataka, India
Job DescriptionLeads a team of people who design, develop and program methods, processes, and systems to consolidate and analyze unstructured, diverse “big data” sources to generate actionable insights and solutions for client services and product e...
1 Day ago
Data Scientist Manager
Information Technology
- Bangalore, Karnataka, India
Job DescriptionLeads a team of people who design, develop and program methods, processes, and systems to consolidate and analyze unstructured, diverse “big data” sources to generate actionable insights and solutions for client services and product e...
1 Day ago
Sr. QA Engineer
Information Technology
- Bangalore, Karnataka, India
Role Summary:Picarro is seeking an exceptional Sr. QA Engineer for functional testing of Picarro Analyzers. This role expects you to analyze requirements, create and execute test-plan, and record results in test-repo. This person is also expected to...
1 Day ago
C++ Graphics and Windowing System Software Engineer - Mir
Information Technology
- Bangalore, Karnataka, India
We build a high-performance, high-efficiency stack for window managers and display subsystems in C++, called Mir. We're growing the team and looking for new colleagues who share our passion for precision, performance and user experience.Our goal is ...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in