Principal Software Engineer

Company: IT Services Organization

Key Skills: CICD, Apache Spark, Docker, Apache, Python, Azure Databricks, Airflow, Kubernetes, Terraform

Key Responsibilities:

Technical Leadership & Architecture:

• Act as a technical architect, ensuring alignment between the work of multiple development teams in data engineering and data science.
• Design scalable, high-performance data processing solutions within Azure and Databricks, considering factors such as governance, security, and maintainability.
• Drive the adoption of best practices in software development, including CI/CD, testing strategies, and cloud-native architecture.
• Work closely with Product Owners and Engineering Managers to translate business needs into technical solutions.

Hands-on Development & Technical Excellence:

• Lead by example through high-quality coding, code reviews, and proof-of-concept development.
• Solve complex engineering problems and contribute to critical design decisions.
• Ensure effective use of Azure services, Databricks, and data processing frameworks such as Apache Spark, Delta Lake, and Airflow.
• Develop and optimise data pipelines, data transformations, and ML workflows in a cloud environment.

Line Management & Team Development:

• Provide line management to engineers, ensuring their professional growth and development.
• Conduct performance reviews, set development goals, and mentor team members to enhance their skills.
• Foster a collaborative and high-performing engineering culture, promoting knowledge sharing and continuous improvement.
• Support hiring, onboarding, and career development initiatives within the engineering team.

Collaboration & Cross-Team Coordination:

• Act as the technical glue between data engineers, data scientists, and software developers, ensuring smooth integration of different components.
• Provide mentorship and guidance to developers, helping them level up their skills and technical understanding.
• Work with DevOps teams to improve deployment pipelines, observability, and infrastructure as code.
• Engage with stakeholders across the business, translating technical concepts into business-relevant insights.

Governance, Security & Data Best Practices:

• Champion data governance, lineage, and security across the platform.
• Advocate for and implement scalable data architecture patterns, such as Data Mesh, Lakehouse, or event-driven pipelines.
• Ensure compliance with industry standards, internal policies, and regulatory requirements.

Experience Requirements:

• Strong software engineering background with experience in designing and building production-grade applications in Python, Scala, Java, or similar languages.
• Proven experience with Azure-based data platforms, specifically Databricks, Azure Data Lake, Azure Synapse, and Azure Functions.
• Expertise in Apache Spark and Delta Lake, with experience building large-scale distributed data pipelines.
• Experience with workflow orchestration tools like Apache Airflow.
• Cloud experience in Azure, including containerisation (Docker, Kubernetes) and infrastructure as code (Terraform, Bicep).
• Strong knowledge of modern software architecture, including microservices, event-driven systems, and distributed computing.
• Experience leading teams in an agile environment, with a strong understanding of CI/CD pipelines, automated testing, and DevOps practices.
• Excellent problem-solving and communication skills, with the ability to engage with both technical and non-technical stakeholders.
• Previous experience in data engineering, data science, or analytics-heavy environments is highly desirable.
• Proven line management experience, including mentoring, career development, and performance management of engineering teams.

Qualifications: Any Post Graduation, Any Graduation