ETL Lab and the case for stateless data pipelines

Software teams spend a lot of time cleaning up state. Cache invalidation, drift between source and destination, failed backfills, duplicate rows, and partial retries all come from one root issue. A system remembered something, then the world changed underneath it.

ETL Lab makes a blunt point: state that is never kept is state that cannot go stale. For a practitioner, this is more than a slogan. It is a design choice. You either persist intermediate truth and spend ongoing effort reconciling it, or you derive what you need from current inputs and accept the compute cost each time.

This matters anywhere you move data between systems. Pipelines often start simple. Then a queue appears, then a staging table, then a retry table, then an exceptions bucket. Each addition solves one local problem and creates a broader consistency problem. A small demo is a good place to inspect the opposite approach, where the pipeline avoids durable intermediate state and reduces the number of things you need to trust.

The problem ETL systems create for themselves

Classic ETL architecture often stores progress at multiple layers. You keep source offsets, transformed records, load batches, dedupe markers, and job metadata. Each piece exists for a reason. Each piece also becomes another thing to repair.

The failure modes are familiar:

• A transform changes, but old derived rows stay in place.
• A retry replays part of a batch and creates duplicates.
• A destination write succeeds, but the job tracker fails, so the next run reprocesses data.
• A source record is corrected upstream, but the pipeline never revisits it.
• A backfill uses a different code path from the daily run, so results diverge.

These are state problems before they are business problems. Once you save an intermediate representation, you own its lifecycle. You need rules for freshness, invalidation, replay, retention, and auditability. You also need operational discipline, because the pipeline now has memory in more than one place.

A stateless approach cuts into this complexity. If a run derives output from current inputs without depending on stored intermediate artifacts, you have fewer stale surfaces. The burden moves from state management to deterministic computation. That trade is often worth studying.

What to inspect in a stateless ETL design

When a system claims to avoid stale state, inspect where state still hides. “Stateless” is often used loosely. A scheduler, a destination table, and logs all hold information over time. The question is narrower: does correctness depend on durable intermediate state from prior runs.

Look for these design choices:

• Inputs are treated as the source of truth.
• Transform logic is deterministic for the same input set.
• Intermediate artifacts are ephemeral, or regenerated each run.
• Output writes are idempotent, or replaced as a complete materialization.
• Recovery uses rerun semantics, not manual patch-up of partial internal state.

The core engineering decision is whether to checkpoint transformation progress. If you do, reruns are cheaper but correctness depends on those checkpoints being accurate. If you do not, reruns are simpler to reason about but more expensive in compute and I/O.

A strong stateless pattern often looks like this:

1. Read an authoritative input set.
2. Apply pure transforms.
3. Write a full result, or an idempotent result.
4. If the run fails, discard partial work and rerun from step one.

This works best when the input set is bounded and accessible, or when the destination supports replace-by-version patterns. It gets harder when sources are unbounded streams, side effects are external, or the cost of full recomputation is high.

How you would verify the claim

You do not need private implementation details to test whether a demo supports its message. You need a few concrete checks.

First, inspect whether results depend on prior runs. Run the same input twice. The output should match. Change an input record, rerun, and confirm the output reflects the change without any manual cleanup. Remove an input record, rerun, and inspect whether the destination still carries a ghost of the old result.

Second, force failure in the middle of execution, if the demo gives you a way to do so. A stateless pipeline should recover through recomputation, not through editing a progress ledger. Partial work should not become a hidden dependency.

Third, inspect the write path. If the system writes incrementally, you want to see idempotency controls such as stable keys or replace semantics. If it writes full outputs, you want to see an atomic swap or versioned publication pattern, so readers do not observe half-built results.

A practical verification checklist looks like this:

• Repeated run, same inputs, same outputs.
• Corrected upstream input, corrected downstream result.
• Deleted upstream input, deleted or absent downstream result.
• Mid-run failure, clean rerun from inputs.
• No operator step to clear stale intermediate records.

If a system passes those checks, its “never kept” claim has substance. If it needs a manual reset, a hidden cache purge, or a special backfill mode, stored state still plays a correctness role.

Where teams get this wrong

The common mistake is to remove explicit state and keep implicit state. The code stops writing checkpoints, but it still relies on timestamps, mutable destination contents, or out-of-band assumptions about what has already been processed.

A few examples:

• Using updated_at > last_seen as the sole extract rule. Clock skew, late updates, and timestamp rewrites turn this into silent data loss or duplication.
• Reading from the destination to decide what to process next. The destination becomes a progress database in disguise.
• Writing side effects during transform steps. Once an email, webhook, or external mutation fires, a rerun is no longer equivalent.
• Depending on nondeterministic transforms such as unordered aggregation, ambient time, or random identifiers.
• Keeping temporary artifacts longer than intended, then relying on them during incident recovery.

Another mistake is to push statelessness into the wrong layer. You do not need every component to be stateless. You need the correctness boundary to be clear. For example, immutable raw inputs paired with reproducible transforms often matter more than whether a worker process holds in-memory state during one run.

The broader lesson is simple. Remove state where it multiplies reconciliation work. Keep state where it is authoritative and observable. Raw inputs qualify. Published outputs qualify. Hidden intermediate truth is where many ETL systems decay.

Signals worth reading in systems like this

A demo like this demonstrates an engineering posture. It favors recomputation over repair. When you see this pattern in production systems, read it as a signal to investigate a few operational properties.

Check recomputation cost first. Stateless designs shift effort into reruns, so you need to understand data volume, execution time, and destination write strategy. If a full rerun is too expensive, teams often reintroduce partial state under pressure.

Check determinism next. If transforms depend on external services, current time, unordered input traversal, or mutable reference data, reruns drift. A pipeline with no checkpoints still goes stale if the same input does not produce the same result.

Check publication semantics too. Readers need a stable view of output. Full recomputation is not enough if downstream consumers see half-populated tables or mixed versions.

Finally, check observability. Statelessness does not remove the need for evidence. You still need logs, run identifiers, input snapshots or references, and enough metadata to explain why an output exists. The difference is that this evidence supports diagnosis, not correctness.

What to watch next

The useful next step is to track where the boundary sits between derivation and storage. As systems grow, pressure builds for faster incremental runs, lower compute cost, and richer side effects. That is where stale state usually returns.

If you adopt ideas from ETL Lab, keep asking one question. If this run fails halfway through, do you repair internal memory, or do you rerun from trusted inputs. The second path is often slower on paper and simpler in practice. Over time, simple tends to age better.