Senior Data & BI Engineer

Role Title: Senior Data Engineer

(Azure | Databricks | End-to-End Delivery)

Experience: 8–12 years (with strong Databricks depth)

Location: Client location - Dubai

Employment Type: Contract - Full time

We are looking for a lead, hands-on Databricks Lead / Lakehouse Architect who can drive project kickoff , define the implementation strategy , and lead end-to-end delivery of an Azure Databricks Lakehouse along with Microsoft Fabric Power BI semantic modeling and reporting for hospitality analytics. This role will own architecture, technical decisions, delivery planning, stakeholder alignment, and initial build-out, and then help ramp the broader team for implementation and rollout.

Key Responsibilities

Project Kickoff & Strategy

• Lead discovery workshops, clarify business objectives, define scope, milestones, and delivery approach.
• Lead discovery workshops with hospitality business/tech stakeholders (Revenue Management, Operations, Finance, Loyalty, Digital).
• Define target-state lakehouse architecture and delivery roadmap (phased onboarding of domains: reservations, stays, POS, loyalty, digital).
• Create end-to-end implementation strategy : architecture, design standards, data onboarding plan, governance, and operating model.
• Define best practices, coding standards, branching strategy, CI/CD approach, and environments (dev/test/prod).

Architecture & Platform Build

• Design and implement Azure Databricks Lakehouse architecture using Delta Lake and Medallion (Bronze/Silver/Gold) .
• Design data ingestion patterns for batch (and streaming if needed) from varied sources (APIs, DBs, files, event streams).
• Build scalable transformation frameworks using PySpark / Spark SQL with performance and cost optimization.

Data Engineering Delivery

• Implement and review complex pipelines: incremental loads, CDC patterns, SCD Type 1/2, data quality rules, and reconciliation.
• Own performance tuning: partitioning, file sizing, Z-ORDER, caching, adaptive query execution, cluster sizing, autoscaling.
• Implement monitoring/alerting, job orchestration, dependency management, and restart/recovery strategies.

Security, Governance & Operations

• Define security model: RBAC/ACLs, service principals, Key Vault integration, secrets management.
• Implement Unity Catalog strategy (if applicable): catalogs, schemas, lineage, permissions, data sharing.
• Establish operational readiness: runbooks, SLAs, logging, audit, cost governance.

Collaboration & Leadership

• Act as primary technical point of contact for client stakeholders.
• Produce design documentation (HLD/LLD), implementation roadmap, and knowledge transfer plan.
• Mentor engineers, conduct code reviews, and ensure delivery quality.

Must-Have Skills

• Deep hands-on Databricks experience in end-to-end implementations (not only development).
• Strong Spark expertise: PySpark + Spark SQL , distributed processing concepts, optimization.
• Strong experience with Delta Lake (MERGE, OPTIMIZE, VACUUM, schema evolution, time travel).
• Experience designing lakehouse architecture (medallion layers, curated zones, semantic datasets).
• Strong Azure data platform exposure: ADLS Gen2 , Azure Data Factory (or alternative orchestration), Key Vault , Azure SQL/Synapse (any).
• Power BI development experience is mandatory.
• Strong understanding of:
• Incremental ingestion, CDC, SCD, data modeling for analytics
• Data quality frameworks, reconciliation, and observability

Nice-to-Have (Strong Plus)

• Unity Catalog implementation experience (access control, lineage, governance).
• CI/CD for Databricks: Azure DevOps/GitHub, Databricks Repos, asset bundles, Terraform.
• Streaming patterns (Kafka/Event Hubs), Auto Loader, DLT.
• Exposure to BI semantic layers (Microsoft Fabric Power BI)
• Databricks certifications (Data Engineer Associate/Professional).
• AI/Copilot Integration experience is plus