Camp Fire Academy
Learning path
Complete UI/UX Masterclass UI/UX Design Foundations UI/UX Design Intermediate Advanced UX Strategy
Specialisations
Fintech & Banking UI/UX UI/UX with AI Training UX for Leaders Design Thinking Figma Training
Most requested
AI for Workplace Productivity Excel for Business Power BI & Business Intelligence Cybersecurity Awareness PDPA Compliance Data Protection Officer Browse the full catalogue →
For teams
Corporate & In-House Training LMS & Self-Paced Learning Events
Company
About Contact Verify Certificate WhatsApp

Data Engineering & Pipelines

A reporting team loses credibility the first time a dashboard shows yesterday's numbers during a board meeting. Behind almost every case of that is a pipeline that failed quietly, a load that partially completed, or a source system that changed a column without telling anyone. This programme covers building pipelines that move data reliably, notice when they have not, and can be re-run over a period without producing duplicates.

Programme Agenda

Pipeline Architecture

Sources, landing, staging and serving layers. ETL against ELT and why the warehouse doing the transformation changed the default answer. Batch, incremental and streaming, and choosing honestly between them.

Ingestion Patterns

Database extracts, file drops, API pulls and change data capture. Handling pagination, rate limits and an API that returns different shapes on different days. Landing raw data unchanged and why that matters later.

Incremental Loading

Full refresh against incremental, watermark columns, late-arriving records, and idempotent loads so a re-run does not double the numbers. Handling deletes in the source.

Transformation and Modelling

Modelling for analysis: facts, dimensions, grain, and slowly changing dimensions. Keeping transformation logic in version control and testable rather than buried in a scheduled query.

Orchestration and Scheduling

Dependency graphs, retries with backoff, timeouts, and what should happen when an upstream task fails. Avoiding the schedule that assumes an upstream system is always on time.

Data Quality Testing

Row counts, uniqueness, referential integrity, freshness and value range tests placed at pipeline boundaries. Deciding which failures should stop a load and which should raise a warning.

Backfills and Schema Change

Re-running a pipeline over a historical period without corrupting existing data. Absorbing an upstream schema change, and the contract conversation with the source system owner.

Observability and Cost

Logging, run history, alerting on failure and on suspiciously successful runs. Warehouse cost drivers, and the query patterns that quietly multiply a monthly bill.

Learning Outcomes:
Design a layered pipeline architecture with a clear responsibility per layer
Choose between batch, incremental and streaming for a given source
Build ingestion that survives API limits, pagination and shape changes
Implement idempotent incremental loads that handle late and deleted records
Model warehouse tables at a defined grain with slowly changing dimensions
Orchestrate dependencies with retries, timeouts and sensible failure behaviour
Place data quality tests at boundaries and decide what blocks a load
Run a backfill and absorb a schema change without corrupting history

Duration: 2 Days (16 Hours)
Training Hours: 9:00 AM to 5:00 PM
Level: All levels
Training Mode: Physical, Online, or Hybrid
HRD Corp SBL-KHAS Claimable
Certificate of Completion included

Frequently Asked Questions

Data engineers, analytics engineers, BI developers, database administrators moving into analytics work, and senior analysts who have inherited responsibility for the pipelines feeding their reports.

Working SQL, and familiarity with at least one scripting language. Participants without SQL should take SQL for Data Analysis first.

Exercises use SQL, Python and a common open source orchestrator, with a cloud warehouse. The material stays on patterns rather than product features so it transfers to whatever stack you run.

No. Streaming is covered enough to decide whether you need it, which for most Malaysian reporting workloads is honestly less often than vendors suggest. Real-time architecture is a separate subject.

Yes, this programme is HRD Corp SBL-KHAS claimable. Our team can assist your HR department with the documentation required for the grant application.

Yes. Any programme can be booked as a team day. Everyone works the same brief together, so your people come away having built something and knowing each other better.

If you are claiming under HRD Corp, the session has to fall at least 14 days after your HRD Corp approval. If you are not claiming, the date is flexible and we work around your calendar.

Put them on the self-paced e-learning instead. Your team works through the modules on our LMS in their own time, sits the assessment, and earns the same certificate, so nobody has to clear a full day together.

More in Data Engineering and Machine Learning