Future goals
Future Goals: Where the Framework Is Headed
The direction in one line: from labels the AI writes to labels code can check. Today an AI labels its own claims and nothing checks whether those labels are right. The plan: first, run bigger and fairer tests. Then build a new filter (not built yet) that looks for the record behind each “checked” claim, so the labels can be checked by code instead of simply trusted.
Where it stands today: a labelling method for AI output, plus an optional checker that tests the labels’ format. The model writes every label itself. The checker makes sure tags balance, every sentence is labelled and every (m) names a source, and it holds actions that rest on a (g). It does not check whether anything is true. The evidence is small: single runs or a few runs per version, mostly on Claude models. The results so far.
Everything on this page is a plan, not a result. Nothing below has been run yet. The order may change as results come in.
Part 1 First: Turn Examples Into Numbers
Our results so far are stories with one or a few runs behind them. The next tests are designed to give rates, with the bias controls our reviews called for.
- The message board test (planned first). Five AI agents, a small “swarm”, coordinate only through a shared board, where everyone reads everyone. Each agent holds facts the others don’t, and the right answer needs all of them. Three traps are planted: a rumour that might come back as fact, a figure that changes partway through, and an outside “confirmed” post with no source. The design borrows the hidden profile method from group psychology, where groups tend to miss the facts only one member knows.
- The code hand-off test. A small chain of coding agents gets a task with one planted gap, such as a settings file that looks local but points at the live database. The main question: when a bug appears, can someone trace it back to the guess that caused it, faster and more often with labels? The gaps are modelled on real agent incidents.
- Fairer scoring. Three versions every time: no instructions; a placebo, meaning instructions of the same length that ask for care but use no labels; and the labels. At least 20 runs per version on two model families. Scoring done by script where possible, and otherwise by a judge from another model family who doesn’t know which version it’s reading. The plan, including how big a difference counts, will be fixed and published before the first run.
- A second, blind scorer for the incident scan, so its first-look numbers can be replaced.
Part 2 Second: A Format a Policy Filter Can Read
Today a person reads the labels and decides what to trust. The next step is text that a person can still read normally, but that a rule-based filter can read exactly.
- The same three labels. (u) given, (m) checked, (g) generated. No new letters.
- Short fields in the closing tag, mostly optional. For example, what a claim rests on, when it was true and when it was checked, whether an action can be undone, and where a fact applies:
(m)St Anne’s is booked on 18 July.(/m: lookup #12 bookings; checked 15 July 09:00)(g)Use the Central Library.(/g: rests on #3, #7, #12; act: reversible) - Plain rules the filter applies, written as may, must and must not, the way policy languages already write permissions, prohibitions and duties. For example: “must not take an irreversible step unless everything it rests on is a fresh check.”
- Checks against real records. A “checked” claim counts only if the tool log shows the lookup behind it. Times and lookup numbers come from the log, not from what the AI wrote. This is meant to address our weakest result so far: in one test, a fake “checked” claim fooled the next agent 4 times out of 4. It would confirm that a lookup happened, not that the answer it gave is right. Untested.
- Time rules. Facts have a shelf life; a newer record replaces an older one; and a check can’t be dated before the thing it checks.
The plan for a proof of concept:
- An offline demo with no AI calls: a few sample messages and a tool log, where the filter accepts a real check, catches a fake one, flags an out-of-date fact, and holds a decision built on a guess.
- The same filter running live inside the message board test, as a moderator between rounds, compared with labels alone.
- A simple rule check on the final decision: it stands only if every requirement rests on an accepted, up-to-date check.
Is this neuro-symbolic AI? No. Today the rule-based part only checks format. With a working filter, the shape would resemble one known design, a pipeline where a neural model writes and labels and a rule-based part checks the labels against records (Henry Kautz’s taxonomy calls this “Neuro;Symbolic”). We make no claim to that label unless the filter is built and tested.
Part 3 Alongside: Real Cases and Other People’s Runs
- More case studies, written and reviewed the same way, including one held back until its primary sources can be read.
- Other people’s runs. The test kit runs against any OpenAI-compatible model. Results from other models, other scenarios and other people are the strongest check on anything here.
- The open questions on the builders page: languages with built-in evidentials, label accuracy against logs, human and mixed teams, other swarm shapes.
Part 4 What Would Change the Plan
- If labels do no better than the same-length placebo in the board test, the benefit is coming from asking for care, not from the labels. We’d say so.
- If the filter flags too many honest claims, it’s too strict to use, and the format needs to be simpler.
- If labels rarely survive real hand-offs (pasting into documents, summaries, other tools), the work moves to keeping the record outside the text.
Results will be published either way, including the ones that don’t favour the labels.
Want to help? Take this and build something better, or run the test kit on a model we haven’t tried.