As a Data Engineer, with extensive experience in Databricks, you will be assigned key responsibilities concerning the design, development, and maintenance of our data environments. Your technical prowess will directly contribute to building scalable and robust data pipelines and enhancing our data platform's performance, availability, and scalability. You will be instrumental in optimizing data workflows, ensuring data quality, and integrating data from different sources.

Key responsibilites:

• Design, develop, and maintain scalable data pipelines using Databricks to support data ingestion, transformation, and loading (ETL/ELT) processes.

• Collaborate with and mentor other product teams and engineers to build and maintain the data platform that integrates data from multiple sources.

• Optimize data processing workflows and ensure they adhere to our architectural principles, performance and security.

• Implement and enforce data quality checks, monitoring, and alerting systems to ensure the integrity and reliability of the data.

• Leverage Databricks features such as Delta Lake and Databricks Workflows to enhance data pipeline performance and reliability.

• Work with cloud infrastructure teams to ensure the platform’s performance, availability, and scalability within the cloud environment.

Skill requirements

Knowledge:

• Hub-and-spoke or data mesh architectural principles.

• Azure native data services (Data Lake Storage Gen2, Synapse Analytics, Event Hubs).

• Modern data governance (Purview, data lineage, roles, and policies).

• Data modelling for analytics in a Lakehouse environment.

• Streaming and near-real-time ingestion design patterns.

• Best practices for securing data in cloud environments.

Experience:

• Built data platforms on Azure with Databricks and Delta Lake.

• Delivered reusable ingestion and transformation pipelines for multiple data sources.

• Implemented data governance controls in a hub-and-spoke or data mesh architecture.

• Migrated on-prem or siloed data solutions into a centralised cloud ecosystem.

• Tuned Spark workloads for performance and scalability.

• Supported self-service analytics teams with platform capabilities and curated datasets.

Skills:

• Proficient with Databricks (Notebooks, Delta Lake, MLflow, Jobs).

• Spark programming using Python (PySpark) or Scala.

• Data orchestration using Azure Data Factory, Databricks Workflows, or similar.

• CI/CD for data projects (Azure DevOps/GitHub Actions).

• Infrastructure as Code for Azure (Terraform, Bicep, ARM templates).

• Strong debugging, performance tuning, and problem-solving skills for large-scale data jobs.