Senior Data Engineer

Data Pipeline Engineer

Role Overview

We are looking for a Data Pipeline Engineer with expertise in multi-source data integration and cloud-

native data platforms. The role focuses on building scalable ingestion frameworks, enabling cross-

agency data sharing, and implementing governance-driven data engineering solutions on AWS.

Key Responsibilities

Develop and maintain 15 data ingestion pipelines from diverse source systems, including REST

APIs, SFTP file transfers, and database extracts.

Design and implement ETL transformations based on business and technical data mapping

specifications.

Build secure cross-agency data-sharing mechanisms using AWS Lake Formation cross-account

permissions and resource links.

Implement data lineage and metadata tracking to support governance, auditability, and

compliance requirements.

Develop robust data quality frameworks capable of handling schema drift, delayed data arrivals,

and source system outages.

Build and manage Infrastructure as Code (IaC) assets using Terraform or AWS CDK.

Execute unit testing, integration testing, and cross-platform data validation activities.

Support User Acceptance Testing (UAT) with business stakeholders to validate data quality,

timeliness, and access controls.

Document source connectivity patterns, data pipelines, governance controls, and architecture

designs.

Participate in Agile ceremonies, code reviews, and technical design discussions.

Required Skills & Experience

5+ years of experience in data engineering and cloud data platforms.

Strong hands-on expertise with AWS Glue, AWS Lake Formation, Amazon S3, and Amazon

Athena.

Proficiency in Python, PySpark, and SQL.

Experience integrating data from APIs, SFTP systems, and relational databases.

Strong understanding of metadata management, data governance, and secure data sharing

models.

Experience with Terraform, AWS CDK, or similar IaC frameworks.

Knowledge of distributed data processing and scalable ETL architectures.

Good to Have

Experience with OpenLineage or data lineage/governance platforms.

Exposure to multi-tenant data platform architectures.

Experience delivering solutions for government agencies or public sector organisations.

1. Education

Bachelor’s degree in computer science, information technology, engineering, or a related discipline.

Pay: $5,000.00 - $12,000.00 per month

Work Location: In person