Senior Data Engineer

Senior Data Engineer - UAE Position Overview Location: Abu Dhabi, UAE

Work Requirement: Client requirement to have people onsite with office visits.

Key Responsibilities The successful candidate will be responsible for:

• Implementing Databricks pipelines using Delta Live Tables (declarative pipelines).
• Rewriting proprietary Foundry transforms into optimised PySpark on Databricks.
• Moving from Foundry's build system to modern CI/CD (IaC-managed Databricks workspaces).
• Performing performance tuning on Databricks Photon to match or exceed Foundry interactivity.
• Contributing to the reconciliation engine (Palantir vs Databricks output equivalence).
• Managing pipeline rebuild, validation, internal testing, and UAT preparation.

Required Skills & Experience Candidates must demonstrate strong hands-on experience in the following areas:

• Proven production-level Databricks experience.
• Strong hands-on coding capability.
• Experience building and optimizing modern data pipelines.
• Ability to work within pod-based delivery teams.
• Experience owning pipeline migration and rebuild activities from design through validation.

Databricks

• Databricks hands-on experience.
• Delta Live Tables.
• Unity Catalog.
• Workflows.
• Production experience required (not lab-based experience).

Data Engineering & Development

• PySpark.
• Advanced SQL.
• Data modelling.
• Schema design.
• Medallion architecture.
• Lakehouse architecture.

DevOps & Deployment

• Modern CI/CD for data pipelines.
• Infrastructure as Code (IaC).
• Git-based deployment.

Communication

• Strong written and verbal English skills.
• Ability to interact directly with Databricks Professional Services (DBX).

Preferred Qualifications The following experience is highly desirable:

Certifications

• Databricks Certified Data Engineer Professional.
• Associate certification at minimum.

Domain Experience

• Financial services.
• Audit.
• Asset management.

Platform Experience

• Prior Palantir Foundry exposure, including:

• Code Repo.
• Code Workbook.
• Pipeline Builder.

Migration Experience Experience With Migrations From Legacy Platforms Such As

• Informatica.
• Hyperion.
• Oracle.
• Palantir Foundry.
• Databricks.

Skills Candidates with any of the following experience will be advantageous:

• Azure Data Factory (ADF).
• Azure Synapse.
• dbt.
• Airflow.
• Equivalent orchestration tools.

Note: Client estate is Azure-heavy.