DataBricks - Data Engineer

About the role:

The Enterprise Document Delivery team is seeking a Sr ETL Software Engineer who will participate in the entire system development lifecycle of applications related to document generation processing, printing, electronic and postal delivery for the bank. The team supports high volume statement applications residing on different platforms. The position will work with various Lines of Business across the organization to understand their requirements and architecture to design and develop the required solution. The candidate should have a strong knowledge SDLC process as well as Agile experience , and ensure that all phases of our development are 100% Wells Fargo Technology SDLC compliant. This position requires someone that can be flexible, wants to be part of a dynamic team, and is able to manage multiple priorities and tasks simultaneously.

Responsibilities of the role include the following:

• ETL Design and Development: Develop, maintain, and optimize ETL processes for data ingestion, transformation, and data warehousing across multiple platforms, including both SQL and NoSQL databases.
• Data Pipelines: Design, build, and manage scalable data pipelines using technologies like Databricks, Apache Spark, Python, SQL, and NoSQL databases.
• NoSQL/MongoDB Expertise: Work with MongoDB to design efficient document schemas, implement query optimization, and handle large-scale unstructured data.
• Data Integration: Collaborate with cross-functional teams to ensure seamless data integration between different sources such as databases (both relational and NoSQL), APIs, and external files.
• Performance Optimization: Implement and monitor performance metrics, optimize data processing performance, and manage ETL job scheduling and dependencies.
• Data Quality: Ensure data quality and integrity across ETL pipelines, implementing processes for data validation, cleansing, and enrichment.
• Automation: Automate repeatable ETL tasks and data processing workflows to improve efficiency and accuracy.
• Collaboration: Work closely with data architects, analysts, and business stakeholders to gather and understand data requirements.
• Cloud Platforms: Leverage cloud services (Azure, GCP) for data storage, processing, and infrastructure management, ensuring scalability and cost efficiency.
• Best Practices: Maintain documentation, adhere to data governance, and best practices in data management, including security and compliance.
• Build Microservices APIS to expose ETL services .

ESSENTIAL QUALIFICATIONS

• Experience: Minimum of 5+ years of experience as a Data Engineer or ETL Developer in complex, large-scale data environments.
• SSIS, Databricks Expertise: Strong hands-on experience working with SSIS, Databricks, including using Apache Spark for data processing and optimization.
• ETL Tools: Proficient with various ETL tools and frameworks such as Informatica, Talend, or SSIS or DataBricks
• Big Data Technologies: In-depth knowledge of big data processing frameworks like Spark, Hadoop, Kafka, etc.
• NoSQL/MongoDB: Expertise in working with NoSQL databases, especially MongoDB, for large-scale data storage, retrieval, and optimization.
• Programming Skills: Proficient in SQL, Python or Java, Power shell for building data pipelines.
• SQL and NoSQL Proficiency: Strong knowledge of SQL and experience working with both relational databases and NoSQL databases like MongoDB.
• Data Modeling: Expertise in designing and implementing data models, including OLAP, OLTP, dimensional, and document-based models (NoSQL).
• Data warehousing: Data warehousing is a key part of ETL process, as it stores data from multiple sources in an organized manner and will be needed to build a repeatable ETL workflow to support many different data sources.
• Redesign and refactor legacy custom ETL processes to reusable ETL workflows that can ingest diverse data sources to normalize data to standard JSON / XML output.
• Data Governance: Knowledge of data governance, security standards, and best practices for managing sensitive data.
• Version Control: Experience with Git or other version control systems for code management.
• Certification: Databricks certification, MongoDB certification, or other relevant certifications in data engineering, cloud platforms, or big data technologies.
• Soft Skills: Strong problem-solving skills, excellent communication, and the ability to work in a collaborative team environment.
• Analytical Mindset: Ability to translate business requirements into scalable, efficient, and reliable ETL solutions.
• Solid understanding of legacy communication protocols and migration strategies.
• Experience with cloud platforms like AWS, Azure, or Google Cloud or TKGI

Project Details:

• Team(s) using SSIS to transform mainframe formatted files to standard JSON and XML file format
• Converting applications from hosted platform to distributed cloud hosted environment
• Evaluating and reengineering customed ETL workflows to reusable ETL microservice API
• Target migration from SSIS to Databricks

Role Purpose

The purpose of this role is to provide significant technical expertise in architecture planning and design of the concerned tower (platform, database, middleware, backup etc) as well as managing its day-to-day operations

Do

• Train direct reportees to make right recruitment and selection decisions

• Promote diversity in leadership positions

• Proactively challenge the team with larger and enriching projects/ initiatives for the organization or team

Stakeholder Interaction

Display

Lists the competencies required to perform this role effectively:

• Functional Competencies/ Skill
  • Technical Knowledge - Knowledge of own tower (platform, application, database etc) - Expert
  • Domain Knowledge - Understanding of IT industry and its trends - Competent to Expert
  
  

• Behavioral Competencies
  • Managing Complexity
  • Client centricity
  • Execution Excellence
  • Passion for Results
  • Team Management
  • Stakeholder Management
  
  

Deliver