Applied AI, with the numbers attached.
Every claim on this page is a measurement, and every measurement links to the thing that produced it — a running demo, or a repository you can clone and re-run. Where a result went against the prediction, the prediction is still published.
What is measured, and where the number comes from
RAG Lab — retrieval, graded
93 pages become 825 chunks, and a question set locked before grading asks 20 questions of them. 16 of 20 retrieved the right page. All four failures are on the page with what came back instead, rather than summarized away.
The same 20 questions were then put to five vector stores in seven index modes — an in-process brute-force scan, Postgres with pgvector, Actian VectorAI DB and Weaviate (the last two in both their approximate and exact modes), and Pinecone — with the chunker, the model, the questions and k held constant and the embeddings computed once and reused.
Six of the seven returned an identical ranking across all 200 ranked positions. Pinecone did not, and it is the only one that differs: on one question, two adjacent results at positions 6 and 7 trade places. Nothing in the top 5 moves anywhere, so hit rate and recall are unchanged across every mode — but MRR and nDCG shift, because one of the two swapped pages is a correct answer for that question.
That swap moved a correct page up, and it would be dishonest to read it as Pinecone doing better. The two chunks sit 0.00038 apart in exact arithmetic, and Pinecone's returned scores differ from exact by about 0.0011 — roughly three times the gap it would need to resolve. At that boundary the order is not determined by what the index returns, so which of the two lands first is chance. It fell the favorable way here; it could as easily have fallen the other.
The wider reading is unchanged: an 825-vector corpus is too small to separate these stores on retrieval quality. At this size an approximate index has almost nothing to get wrong, and the single position-6 swap is the entire measured difference between a managed ANN service and a brute-force scan over the same vectors.
Actian and Weaviate run locally, in containers, for the measurement only. Pinecone is a hosted service, so that row alone crosses a network — which makes its timing figures not comparable with the local ones, and they are not presented as if they were. Nothing on this site queries a vector database, and pigfox.com is not backed by one.
Six of the seven modes need no credential and reproduce from a clone with no key. Pinecone is the exception — it needs an API key, so that one row cannot be reproduced without an account of your own. Every other figure on this page can. Source: pigfox/rag-lab.
Bench Lab — three models, one fixed suite
A 40-task suite whose answers are checked by code, never judged by another model, scoring accuracy, first-token latency and dollars per correct answer. Six predictions were committed before the run existed; three were refuted, and the run exposed one of its own metrics as broken. Both facts are on the page.
Judge Lab — models grading a model
Two models grade a third against a gold set labeled entirely by code. Six predictions were committed beforehand and one held. The largest finding is about the harness rather than the models: a strict output contract discarded 815 of one judge's 1,032 replies over their punctuation — and the contract was deliberately not amended afterwards, because amending it after seeing the result is how a harness stops measuring anything.
Agent Gateway — a policy gate in front of agents
150 requested actions from six agents, replayed as allowed, held for a person, or refused. Change a threshold in the browser and diff the old policy against the new one; two rules read history, so an edit moves actions whose own numbers never crossed a line.
SBA Lab — a prediction graded against real data
A model was given industry and state names with no numbers in them and asked which were riskier. 2.19 million real SBA 7(a) and 504 loans, from the published FOIA extracts, were then aggregated and its answers graded by code. It nailed restaurants, and put farming and mining in the worst tier when they are the safest — the maximum possible error, twice.
The loan data is real and public. The prediction being graded is the demo. No client is behind this and it is not advice about lending.
Agent Swarm Lab — a blind prediction of a simulation
A model was given the rules of a made-up drone-swarm simulation, and no results, and asked where mission success first reaches 50% in four scenarios. The prediction was committed before the simulation ran them. 8 of 8 predictions landed within two drones, and 6 were exact. The two that were off by one are on the page with the rest.
The parameter file stated every rule, so this measures reasoning about a fully described model, and the model that predicted is the same one that wrote the simulation. Both limits are stated beside the score.
Access Audit Lab — an access review you can re-run
An access review over a made-up 50-person company — over-broad access, conflicting permission pairs, dormant accounts — with every finding filed under the control it is evidence for. Move a threshold and it re-runs in the browser, then prints the evidence report.
The company, the people and the permissions are invented. It is a demonstration of the review, not a review of anyone, and no client is behind it.
llm-evalkit — the harness underneath
Predictions are pre-registered and hashed together with the graders and the cases, so neither can be changed once a result has been seen. Grading is deterministic functions only, and every case runs k times with all outputs kept — including the ones that disagree.
What these are, and what they are not
They are demos. They are built to be checked, not to be sold as case studies. Where a demo uses invented data it says so on this page and on its own; where it uses real public data it names the source.
The predictions come first. Bench Lab, Judge Lab and SBA Lab each committed what they expected before the run existed, and each published a result that went against part of it. A benchmark that only reports its wins is measuring the author.
Naming a vendor is not an endorsement. Actian VectorAI DB, Weaviate and Pinecone appear because they were measured. None is in production here, none sponsors this, and nothing on pigfox.com sends a request to any of them.
Want this kind of measurement applied to your own system? Get in touch, or read how the AI work is built.