Applied AI, with the numbers attached.
Every claim on this page is a measurement, and every measurement links to the thing that produced it — a running demo, or a repository you can clone and re-run. Where a result went against the prediction, the prediction is still published.
What is measured, and where the number comes from
RAG Lab — retrieval, graded
93 pages become 825 chunks, and a question set locked before grading asks 20 questions of them. 16 of 20 retrieved the right page. All four failures are on the page with what came back instead, rather than summarized away.
The same 20 questions were then put to five vector stores — an in-process brute-force scan, Actian VectorAI DB, and Weaviate, the last two in both their approximate and exact modes — with the chunker, the model, the questions and k held constant and the embeddings computed once and reused. Every store returned the identical ranking, across all 200 ranked positions. That is a fact about an 825-vector corpus rather than a verdict on the stores: at this size an approximate index has nothing to get wrong, and the honest reading is that the corpus is too small to separate them on retrieval quality.
Actian and Weaviate run locally, in containers, for the measurement only. Nothing on this site queries a vector database, and pigfox.com is not backed by one. No credential was used for any measured row; the repository reproduces every figure from a clone with no key. Source: pigfox/rag-lab.
Bench Lab — three models, one fixed suite
A 40-task suite whose answers are checked by code, never judged by another model, scoring accuracy, first-token latency and dollars per correct answer. Six predictions were committed before the run existed; three were refuted, and the run exposed one of its own metrics as broken. Both facts are on the page.
Judge Lab — models grading a model
Two models grade a third against a gold set labeled entirely by code. Six predictions were committed beforehand and one held. The largest finding is about the harness rather than the models: a strict output contract discarded 815 of one judge's 1,032 replies over their punctuation — and the contract was deliberately not amended afterwards, because amending it after seeing the result is how a harness stops measuring anything.
Agent Gateway — a policy gate in front of agents
150 requested actions from six agents, replayed as allowed, held for a person, or refused. Change a threshold in the browser and diff the old policy against the new one; two rules read history, so an edit moves actions whose own numbers never crossed a line.
SBA Lab — a prediction graded against real data
A model was given industry and state names with no numbers in them and asked which were riskier. 2.19 million real SBA 7(a) and 504 loans, from the published FOIA extracts, were then aggregated and its answers graded by code. It nailed restaurants, and put farming and mining in the worst tier when they are the safest — the maximum possible error, twice.
The loan data is real and public. The prediction being graded is the demo. No client is behind this and it is not advice about lending.
Access Audit Lab — an access review you can re-run
An access review over a made-up 50-person company — over-broad access, conflicting permission pairs, dormant accounts — with every finding filed under the control it is evidence for. Move a threshold and it re-runs in the browser, then prints the evidence report.
The company, the people and the permissions are invented. It is a demonstration of the review, not a review of anyone, and no client is behind it.
llm-evalkit — the harness underneath
Predictions are pre-registered and hashed together with the graders and the cases, so neither can be changed once a result has been seen. Grading is deterministic functions only, and every case runs k times with all outputs kept — including the ones that disagree.
What these are, and what they are not
They are demos. They are built to be checked, not to be sold as case studies. Where a demo uses invented data it says so on this page and on its own; where it uses real public data it names the source.
The predictions come first. Bench Lab, Judge Lab and SBA Lab each committed what they expected before the run existed, and each published a result that went against part of it. A benchmark that only reports its wins is measuring the author.
Naming a vendor is not an endorsement. Actian VectorAI DB and Weaviate appear because they were measured. Neither is in production here, neither sponsors this, and nothing on pigfox.com sends a request to either.
Want this kind of measurement applied to your own system? Get in touch, or read how the AI work is built.