What is a data pipeline: stages, patterns, and failure modes

A data pipeline is a series of automated steps that moves data from the system where it's created to the system or dashboard where someone uses it, reshaping the data on the way so it lands in a form the destination can use. It runs on a schedule or continuously, without a person moving the data by hand.

The reason it matters shows up downstream. A retailer's sales dashboard, a fraud model retrained overnight, a finance team closing the books at month end: each reads data that already passed through a pipeline, usually more than one stitched together. Nobody watches that handoff happen. The dashboard is just expected to already have yesterday's numbers on Monday morning, and the pipeline behind it only gets attention once that stops being true.

The four stages

Every pipeline moves through some version of the same four stages, even when the tooling behind them looks nothing alike.

Ingestion reads from source systems: application databases, third-party APIs, event streams, files someone still exports by hand. It's the same layer a system integration project connects, but a pipeline's ingestion is read-only, aimed at analysis rather than at triggering another system's process.

Transformation cleans, joins, and reshapes raw data into something usable: fixing types, standardizing units, folding a customer ID that looks different in every source system into one value. Whether this happens before or after the data lands is the ETL-versus-ELT question, covered next.

Orchestration coordinates the other three stages rather than touching the data itself. A scheduler such as Airflow, open source, decides what runs when, in what order, and what happens on failure: retry, alert, or stop everything downstream before a bad batch spreads.

Storage and serving is where reshaped data ends up and how it gets used: a warehouse table a BI tool queries, a feature store an application reads, an API another team calls. A lakehouse is mostly this stage changing shape: cheap object storage plus a table format such as Iceberg or Delta Lake, adding the transactional guarantees a warehouse always had, without a data lake's storage cost.

ETL versus ELT

ETL extracts data from a source, transforms it into its final shape, then loads the result into the destination. ELT flips the last two steps: load the raw data first, transform it inside the destination afterward.

Cheap warehouse and lakehouse compute is what pushed ELT to the front. Landing raw data first means transformation runs as SQL inside the destination itself, often through a tool such as dbt, and can change without touching ingestion at all. That's the appeal: a broken metric definition gets fixed downstream, not by re-extracting anything from the source.

ETL still fits when raw data can't legally land anywhere unfiltered, a compliance scrub of personal data before storage, for instance, or when a source can only export in a shape the destination can't take directly. Some pipelines still transform before loading for that reason, and most data teams run pipelines both ways rather than standardizing on one pattern.

Batch versus streaming

Batch runs a pipeline on a schedule: hourly, nightly, once a week, transforming a chunk of data each time it fires. It's cheaper and easier to reason about, since a failed run just restarts from a known point, and it fits anything that doesn't need to be current to the minute: a monthly finance report, a model retrained overnight on yesterday's data.

Streaming processes each event as it arrives, instead of waiting for a scheduled run. A fraud check on a live transaction, or an incident dashboard an on-call engineer is watching, loses most of its value once the answer is an hour old. A message broker such as Kafka carries the events; a stream processor transforms them as they pass through rather than after they've landed.

The cost gap is easy to underestimate. A batch job runs for minutes and stops billing; a streaming pipeline stays up continuously and has to handle events arriving late or out of order without corrupting a result that already looked done. Most teams that need even one metric live end up running both: streaming for the few metrics that justify it, batch for everything else.

Common pipeline failure modes

Pipelines rarely fail loudly. They keep running and quietly produce numbers that are wrong, usually for one of four reasons.

• Schema drift nobody caught. A source renames a field or changes its type, the pipeline keeps running without erroring, and the new field lands as null or gets silently truncated. The mistake surfaces weeks later, in a report that looks slightly off.
• No data quality checks. Nothing validates row counts, null rates, or value ranges between stages, so a partial extract from a source outage reaches a dashboard looking like an ordinary day.
• Backfills that double-count. Re-running a pipeline over a past date range to fix bad data, without deleting or overwriting the original run first, appends a second copy instead of replacing it, inflating every downstream total for that period.
• Retries that aren't idempotent. A step fails partway through, after writing some of its output, and an orchestrator retries it automatically. A step that can't detect what it already wrote duplicates or corrupts data while the job status still reports success.

A pipeline stops being a side project

A pipeline pulling one dataset into one dashboard is small enough for whoever needs the report to build and maintain alone, in a script or a scheduled query. That changes once a few things are true.

The source count grows. Past four or five systems, each with its own schema and its own way of renaming a field without telling anyone, keeping every downstream job correct stops being an afternoon fix.

Compliance changes the calculus. A transformation step can copy a customer's email or a patient ID into a table nobody classified as sensitive, and without column-level lineage there's no way to trace where it went or who can query it.

Decisions start leaning on the output. A dashboard nobody checks closely is a low-stakes place for a quiet bug to hide. A pricing model or a number a board reads quarterly is not, and that's when a pipeline earns the same design attention as the product it feeds.

A pipeline carrying that kind of weight usually falls to the same custom software development or product development team already building everything else around it. The exception is a pipeline feeding a retrieval system rather than a dashboard: that becomes AI integration services work.