Data & Cloud Engineer · Lagos, Nigeria

Pipelines, and the
infrastructure under them.

Azure data platforms built on a foundation of real network and systems engineering — so the pipeline works, and so does everything it runs on.

DAG · daily_platform_load kafka.events adls.raw rest.api cdc.stream adf.pipeline dq.assert powerbi extract transform verify · serve

About

the short version

I came to data engineering through IT infrastructure, networking, and cloud engineering — which means I think about pipelines the way I used to think about networks: what happens when a link drops, where the bottleneck actually is, and who gets paged.

Today I build cloud data platforms on Azure — Data Factory for orchestration, Databricks and PySpark for transformation, Data Lake for zoned storage, SQL for the modelling layer. The goal is always the same: turn raw, inconsistent source data into datasets people are willing to make decisions on.

The infrastructure background still does work. I've built and run three-tier applications on AWS, designed VPC networking with private endpoints, and handled snapshot-and-AMI disaster recovery. Most data engineers can write the transformation; fewer can tell you why the subnet routing broke it.

I'm also working on where AI genuinely helps a data workflow — quality monitoring, documentation, anomaly explanation — with a firm line: models explain and draft, tests decide.

Certification Microsoft Certified: Azure Data Engineer AssociateDP-203 · passed Studying Fabric Data Engineer Associateexam not yet taken Education B.Sc. Computer ScienceLagos State University Based in Lagos, Nigeriaopen to remote

Azure data engineering

flagship + 4 case studies
azr-00 Nordic Sales Data Platform state · flagship

Four Nordic countries, four ERP exports, four incompatible formats — different delimiters, encodings, date formats, decimal separators, currencies and column names. A medallion platform reconciling all of it into one tested star schema, running end to end on a laptop with no cloud subscription required.

16,000
rows reconciled
42 / 42
dbt tests passing
0%
quality failures

One tool owns each layer

  • ADF orchestrates only — moves files, triggers compute, holds no transformation logic, so pipeline JSON stays diffable in git
  • PySpark owns Bronze→Silver — reconciling four schemas is genuinely distributed work
  • dbt owns Silver→Gold — dimensional modelling and tests living beside the models they guard

Decisions worth defending

  • dim_customer keys on (country, customer_no). 401 customer numbers appear in more than one country — keying on the number alone would have merged 401 pairs of unrelated businesses, with row counts still looking plausible.
  • Bad rows are flagged, never dropped. A dropped row is invisible; a flagged row is countable. Nine checks, one gate, applied in exactly one place.
  • Returns stay negative in the fact. Filtering them inflates revenue, and nobody notices until quarter close.

Two bugs found by running it, not reading it

  • A . separator passed to regexp_replace is a regex wildcard — it stripped every character, emptying all 4,000 Danish prices. Denmark would have silently contributed zero revenue had the quality layer not caught it.
  • A bare cast under Spark ANSI mode aborted the whole job on one malformed price instead of recording a quality failure — defeating the entire flag-don’t-drop design.
Azure Data FactoryDatabricksPySparkdbtDeltaStar schema
View source →
azr-01 Cloud Data Pipeline Modernization state · delivered

Migrated a batch ETL workflow to a cloud-native, event-driven pipeline on Azure. Airflow handled orchestration and dependency resolution; Data Lake Storage managed raw-to-curated zoning so reprocessing never hit source systems twice.

5 hrs → 65 min
processing time
Azure Data FactoryAirflowSparkADLS
View source →
azr-02 AI-Assisted Data Quality Monitoring state · delivered

An automated data quality layer where deterministic tests decide pass or fail, and an LLM generates plain-language summaries of what broke and where it came from — so stakeholders get an explanation instead of a red square, and engineers start triage with context.

80%
faster time-to-diagnosis

Checks implemented

  • Null and completeness validation on required fields
  • Duplicate detection against natural keys
  • Schema drift validation on ingest
  • Structured run logging and error handling
PythondbtLLM APIAzure
View source →
azr-03 Real-Time Analytics Dashboard state · delivered

Streaming pipeline ingesting live transaction data via Kafka into a dashboard the business team uses daily to track transaction trends. The design work was less about throughput than about defining "current" when late-arriving events and business-day boundaries disagree.

KafkaAzurePower BI
View source →
azr-04 Warehouse Cost & Performance Optimization state · delivered

Refactored a growing Snowflake warehouse — partitioning, query tuning, and workload isolation so competing workloads stopped contending for the same compute. Query times improved for the analytics team alongside the cost reduction.

90%
monthly compute cost
SnowflakeSQLDatabricks
View source →

AWS & infrastructure

one architecture, four stages

A complete serverless three-tier web application, built one tier at a time, plus an analytics layer over S3. Self-directed hands-on builds; each write-up documents the design decisions and the failures.

tier-01 Presentation — CloudFront + S3 state · built

Static front end delivered globally through CloudFront, with the origin bucket never public. Access flows through Origin Access Control, so the distribution is the only path to the objects.

What broke, and why it mattered

  • Access Denied — origin access was set to public against a private bucket. Switching to OAC was only half the fix; the generated bucket policy still had to be applied. Creating the identity does not grant it access.
  • 403 on S3 static hosting — unblocking public access alone was insufficient; the bucket policy needed an explicit public read grant.

Extension: S3 hosting vs CloudFront

  • S3 static hosting requires public objects. CloudFront with OAC restricts bucket access to the distribution — materially more secure.
  • Measured load times were faster through CloudFront, since content is cached at edge locations rather than fetched from one region.
S3CloudFrontOACBucket policy
Read the write-up →
tier-02 Logic — API Gateway + Lambda state · built

REST API fronting a Lambda function via proxy integration, deployed to a prod stage. API Gateway acts as the traffic manager — it reduces backend load, decides whether a request is authorised, and only then passes it through.

Design decisions

  • REST over HTTP or WebSocket — standard HTTP methods, broad language compatibility, no need for persistent bidirectional connections
  • Resources and methods — a GET method on the users resource, wired to the function
  • Stages — deployed to prod, each stage carrying its own invoke URL and its own API version

Extension: API documentation

  • Written in the console and published per stage, since different stages hold different API versions
  • Export combines hand-written descriptions with auto-generated output, loadable into Swagger or Redoc
API GatewayLambdaRESTOpenAPI
Read the write-up →
tier-03 Data — Lambda + DynamoDB state · built

DynamoDB table keyed on userId, with a Lambda function that extracts the ID from the triggering event, queries for the matching record, and handles errors. The access pattern is a single-partition key lookup, which is exactly what the application needs.

The hard part

  • Writing a custom inline IAM policy — managed policies grant more than a single-purpose function needs. Scoping the function's execution role to specific DynamoDB actions was new territory, and it resolved the permission error that had surfaced at the API in tier-02.
  • Schemaless storage — records can gain attributes independently, which is flexible but pushes consistency into the application layer.
DynamoDBLambdaIAMInline policy
Read the write-up →
anl-01 Analytics — QuickSight over S3 state · built

Dataset in S3 connected to QuickSight through a manifest file — the map describing where the files sit and how they are structured. Editing the manifest to carry the correct S3 URI is the step everything else depends on.

Built

  • Records by release year, grouped by content type
  • Genre distribution filtered to post-2015 titles — 171 records

Worth being direct about this one: the dataset arrived clean, so none of the failure modes that dominate real analytics work — schema drift, nulls in grouping columns, duplicates inflating counts — ever appeared. The transferable part is the connection pattern, not the charting.

QuickSightS3manifest.json
Read the write-up →

Working with AI

where it earns its place

I use AI as a working tool, not a demo. The distinction I hold to across every pipeline I build: models explain and draft, tests decide. An LLM is excellent at turning a failure into a readable explanation and at drafting the boring 80% of a transformation. It is not allowed to be the thing that determines whether data is correct — that stays with deterministic assertions I can reason about.

That line is not caution for its own sake. A model that silently passes bad data is worse than no monitoring, because it manufactures confidence. So the architecture is always the same shape: the tests gate, the model narrates.

Outside the pipeline, I use the same tooling to ship things end to end — sites, automation, and video — because a data engineer who can also build and publish is more useful than one who can only hand over a notebook.

The rule Tests gate.
Models narrate.never the other way round
Reviewed, always No generated code, SQL, or documentation ships without a read-throughspeed is not the same as trust

The stack I actually use

Engineering

Claude

Debugging, scaffolding pipelines, translating between SQL dialects, generating documentation from schema and lineage. Used in the desktop app and in Claude Code against real repositories.

Build

Lovable

Prompt-to-deployed web application — React and Tailwind on TanStack Start. Used to design, build, and ship this portfolio's predecessor to a custom domain.

Automation

n8n

Workflow automation connecting services without writing glue code for every integration — triggers, transforms, and scheduled runs.

Video

HeyGen

Avatar-led short-form video from purpose-written scripts. Script-driven rather than repurposed, with on-screen text timed as canvas elements.

Media

Higgsfield

Generative B-roll and cinematic footage to support narrated video, driven through an MCP connector rather than the web UI.

In pipelines

LLM APIs

Called directly inside data workflows for anomaly explanation and plain-language failure summaries — layered on top of dbt tests, never replacing them.

Stack

what I reach for

Data engineering

  • Azure Data Factory
  • Databricks
  • PySpark
  • SQL
  • Python
  • Azure Data Lake
  • ETL / batch processing

Cloud

  • Azure
  • AWS
  • EC2
  • VPC & networking
  • Storage

Infrastructure

  • Windows Server
  • Linux
  • Active Directory
  • Firewalls
  • VPN

DevOps

  • Git
  • GitHub
  • GitHub Actions
  • Docker learning
  • CI/CD learning

Proof of work

Bravo ’26 · click straight through

Assignments from the Bravo ’26 bootcamp, plus site builds, all public and linked. Nothing here is a screenshot standing in for something that does not exist.

bravo-01 Video — Five things I do with Claude state · published

Scripted short-form video on the practical uses of Claude that hold up in day-to-day engineering work — the ones that survive contact with a real repository, rather than demo material. Written for the format instead of cut down from something longer.

Watch on Instagram →
bravo-02 Video — UGC format state · published

Creator-style delivery to camera, working in the UGC format rather than a polished corporate register.

Watch on Instagram →
bravo-03 Automation — Naira vs. major currencies tracker state · running daily

A scheduled n8n workflow that logs the naira against a basket of seven major trade currencies, so the exchange-rate picture builds into a history rather than being checked ad hoc. Live and running on a timer.

Flow

N8N · 4 NODES · DAILY · UPSERT VERIFIED Schedule Trigger daily · 08:00 HTTP Request er-api · latest/NGN Code build basket · 7 ccy Google Sheets upsert · code+date

What each row carries

  • Currency code, name, and approximate share of global trade
  • Official rate — one unit expressed in NGN, computed live from the API response
  • Parallel-market rate — an explicit placeholder, because no free live feed exists
  • Ten-year context on both official and parallel market movement
  • A timestamp, and a provenance note on every single row

The design decision worth pointing at

The parallel-market rate is the number a Nigerian buyer actually cares about, and there is no free live source for it. The choice was between leaving the column blank, filling it with something that looked authoritative, or saying plainly what it is.

Every row therefore carries a provenance note stating which field is live from the API and which is static historical context, and the parallel-market field names the manual source it needs instead of holding a number. Output that tells you how much to trust each field is worth more than output that looks complete — the same reason a data quality layer explains a failure rather than just flagging one.

Revision: matching the schedule to the data

The first version fired every 30 minutes against an API that refreshes once a day — 48 runs producing one day’s worth of information, appended as duplicate rows. Rebuilt to run daily and upsert on currency_code + rate_date, so each currency holds exactly one row per day and the sheet becomes a rate history instead of a log.

The general lesson is one that applies to any scheduled pipeline: polling faster than the source updates buys nothing and costs storage. The schedule should follow the data’s refresh rate, not the impatience of whoever wrote it. The source confirms it directly — the API returns a time_next_update field, and it lands 24 hours out.

Verified rather than assumed. Migrating to upsert surfaced a second problem: Google Sheets cannot match columns on an empty tab, because the operation reads row one to resolve them. Seeding headers with a single append, then switching to upsert, fixed it. Two consecutive runs then produced seven rows rather than fourteen — which is the only real proof that the match key matches rather than merely resolving.

Known gaps

  • The parallel-market rate is still manual. The field names its source rather than holding a number, which is honest but means the decision input is incomplete.
  • The basket is hardcoded. Fine while the seven are stable, but the trade-share figures will drift and nothing prompts a review.
  • No failure path. If the API returns an error or a missing currency, the row writes a null rather than alerting.
bravo-04 Website — built and deployed with Lovable state · live

Prompt-to-production personal site: wrote the brief, designed it, built it, and shipped it to a custom domain. React and Tailwind on TanStack Start, with an animated hero graphic and sections for profile, work, stack, and contact.

The point of the assignment was speed from idea to live URL. The point of keeping it is that it works — and that going through the whole loop, including the custom domain and the DNS, is different from stopping at a preview link.

LovableReactTailwindTanStack StartCustom domain
Visit the site →
web Other builds state · live
  • This site — hand-built single-file static site, no framework, no dependencies, deployed to GitHub Pages

Certifications & CV

credentials

Earned

  • Microsoft Certified: Azure Data Engineer Associate DP-203
  • Microsoft Certified: Azure Fundamentals AZ-900
  • Microsoft Certified: Azure Data Fundamentals DP-900
  • Cisco Certified Network Associate CCNA

In progress

  • Microsoft Fabric Data Engineer Associate studying

Education

  • B.Sc. Computer Science
  • Lagos State University

Programme

  • Bravo ’26 bootcamp 4 assignments

In-progress certifications are listed as in progress. Download full CV →

Got a pipeline that keeps
waking someone up at 3am?

Available for data platform work, pipeline reliability reviews, and cloud infrastructure builds. Based in Lagos, working remote.