ETL Lab

The whole pipeline is rebuilt from nothing every cycle, and everything it built is thrown away at the end of one. Poll, extract, transform, load, detect, execute, flush — then again, from an empty graph.

A pipeline that keeps its state has to keep it correct: invalidate what changed, reconcile what drifted, and be right about which of yesterday’s rows are still true. A pipeline that keeps nothing has none of those problems and exactly one requirement in their place — it has to be able to rebuild fast enough that starting over is cheaper than remembering. State that is never kept is state that cannot go stale. That is the whole design, and the counters below are the argument for it.

The production version of this design was load-tested at 37M+ token mappings per request cycle.

What this demonstrates

The pipeline, end to end and in this process: quotes parsed and rejected with the cause recorded for each, exponents normalized, symbol case and venue aliases resolved to one spelling, duplicate pairs across venues collapsed to the better rate, a directed mapping graph built both ways, and a Bellman–Ford search over negative logarithms that finds a profitable cycle or reports that there is none. Every figure on this page is read off that run.

The exchanges and their prices are synthetic and are generated on this server, from a seed you can set and repeat. Each token carries a value and every quote is the ratio of two of them skewed by a fraction of a percent, so the mispricings the search finds are the ones such a book actually carries. Every fill is simulated — the profit and loss is arithmetic over those generated rates against a fixed 1000-unit notional, with seeded slippage of up to 85% of the theoretical edge subtracted from it.

Nothing is retained between cycles, and that is the design rather than a limitation of it. The rows, the graph, the routes and the fills exist inside one cycle and are unreachable when it ends. What crosses the boundary is the cycle number and the running profit, and the simulation’s own test suite asserts that reflectively — a field added later fails that test instead of quietly becoming state that outlives a rebuild.

Live venue connectivity, order routing and execution against a real book are a separate subject that takes separate evidence. Nothing here reaches a network, holds a credential or places an order.

Where the work happens

The simulation runs on the server, in Go, and the frames reach this page over a server-sent event stream. A run starts real goroutines on a real box and holds them, so the server admits 4 runs at a time, 1 per address, and ends any run after 90 seconds.

Every control below has a hard ceiling applied in the simulation, not in the browser: at most 8 exchanges and 64 pairs each, so no cycle builds more than 512 raw rows. At most 8 routes are surfaced per cycle, none longer than 6 hops. Ask for more and the server runs what it will and tells you what it ran.

The throughput figure is derived from a cost the simulation charges — 40 nanoseconds per raw row — rather than from wall-clock time. A rate measured against the wall would describe whichever machine this happened to run on; a rate measured against an explicit charge is a property of the simulation, so the number you see is the number anyone else sees for the same seed.

The speed control changes pacing only. It divides the wait between phases and touches nothing else: the same settings at any speed generate the same rows, build the same graph, find the same routes and report the same throughput. At the top of the range the phases go by faster than they can be watched, which is what the counters are for.

The simulation is a separate module, imported at a version: https://github.com/pigfox/etl-lab. The code there is the code that produced what is below, not a copy of it.

Run it

Same seed, same run. 1×

Press start to open a run.

  1. Poll

    Synthetic venues emit raw quotes.

    — raw rows

  2. ETL

    Parse and reject, normalize, dedupe, build the graph.

    — mappings · — rejected · — collapsed

    — mappings/sec

  3. Detect

    Negative-log weights, Bellman–Ford, negative cycles.

    — routes found

  4. Execute

    Simulated fills with seeded slippage.

    — this cycle

  5. Flush

    Every structure discarded. Nothing carried forward.

    0 cycles complete

Cumulative simulated P&L across every cycle of this run: —. It and the cycle count are the only figures that survive a flush.

Trace

    Reading the counters

    Rejected counts rows the extract step refused and records why: an unparseable price, a price too large for its field, a decimals value no token uses, a missing side, or a symbol this universe does not contain. Each cause is counted separately, because “bad row” is not a thing anyone can act on.

    Collapsed counts pairs a second venue had already quoted. They are not discarded — the better rate wins — and the count is what makes the join visible. Two venues quoting one pair is also the condition a profitable cycle exists across, so this number and the routes below it move together.

    Routes found is a search result, not a recommendation. The search is exact about whether a negative cycle exists and hands back one; finding several is iterative, so there may be cycles it does not report. A cycle with no route is the ordinary case for a small book and is reported as such rather than left blank.