Job Details:
Role Overview
We are looking for a Senior Data Engineer
with strong hands-on experience in Azure Databricks and the Microsoft Azure
data ecosystem. The role will design, build, optimise, and support scalable
data pipelines and lakehouse solutions that enable analytics, reporting, data
science, and enterprise decision-making.
What You'll Work On
• Design,
build, and optimise scalable batch and streaming data pipelines using Azure
Databricks, PySpark, Python, SQL, and Delta Lake.
• Develop
reusable ingestion, transformation, validation, and publishing frameworks for
large-scale enterprise datasets.
• Implement
medallion or layered lakehouse patterns across raw, curated, and
consumption-ready data zones.
• Integrate
Azure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure
Key Vault, Azure DevOps, and related Azure services.
• Develop
production-grade Databricks notebooks, jobs, workflows, libraries, and cluster
configurations.
• Create
complex SQL and PySpark transformations, data-quality checks, reconciliation
logic, and error-handling routines.
• Optimise
Spark workloads, storage layouts, partitioning, file sizes, caching, and query
performance while controlling platform cost.
• Implement
Delta Lake capabilities such as ACID transactions, schema enforcement, schema
evolution, time travel, and merge operations.
• Build
automated CI/CD pipelines for code promotion, configuration management,
testing, and deployment across environments.
• Apply
data security and governance controls using role-based access, secrets
management, data masking, auditability, and controlled access patterns.
• Monitor
data pipelines, troubleshoot failures, perform root-cause analysis, and deliver
sustainable production fixes.
• Partner
with Architects, Business Analysts, Data Analysts, Product Owners, and business
stakeholders to convert requirements into reliable data products.
• Provide
technical leadership, code reviews, engineering standards, and mentoring to
other data engineers.
Key
Skills and Experience
• 8+
years of experience in data engineering, data integration, ETL/ELT, or
large-scale data platform delivery.
• Strong
hands-on experience with Azure Databricks, including notebooks, jobs or
workflows, cluster configuration, debugging, and performance tuning.
• Advanced
proficiency in Python, PySpark, and SQL for production-grade data engineering.
• Strong
experience with Delta Lake and lakehouse architecture, including reliable
incremental processing and data-quality controls.
• Practical
experience with Azure Data Lake Storage Gen2 and Azure Data Factory.
• Experience
implementing CI/CD using Azure DevOps, Git, branching strategies, automated
testing, and environment-based deployment controls.
• Strong
understanding of dimensional modelling, data warehousing, data lake design,
metadata, lineage, partitioning, and file formats such as Parquet.
• Experience
with orchestration, monitoring, alerting, operational support, incident
analysis, and service-level expectations.
• Understanding
of Azure identity, Key Vault, managed identities, networking, private
endpoints, and secure data-platform patterns.
• Strong
communication, documentation, problem-solving, code-review, and stakeholder
collaboration skills.
Good to Have
• Experience
with Microsoft Purview, Unity Catalog, Azure Synapse Analytics, Event Hubs, or
Structured Streaming.
• Knowledge
of infrastructure as code using Terraform or Bicep.
• Experience
with dbt, Airflow, Kafka, Power BI, or machine-learning data pipelines.
• Experience
in telecommunications or another high-volume enterprise environment.
