Measurements: what the model can be trusted with

Every delegated task wisp offers is measured against a small eval set on a real model, and the result ships with the tool catalogue so a calling harness knows which delegations are reliable before it spends a call (ADR 0026). A measurement is one eval run on one Mac on one day; it is evidence, not a certification.

Where to see them

• wisp tools --markdown and the MCP resource wisp://tools.md: a Measured: line under each tool that has one, with the task, passed/total, the model, the date, and what a pass was.
• wisp tools --json and wisp://tools: the same as measurements on each tool.
• wisp://measurements: every measurement, including tasks that are not one model tool (triage, the risk classifier, schema-shaped replies).

What is measured

Classifier measurements also carry p50Milliseconds and p95Milliseconds, the latency per verdict, since a classifier runs on every command and speed is half of what makes one fit (ADR 0038); wisp classifier measure reports the same for any classifier over any labelled file.

Most sets are small on purpose: they prove the pipeline and catch regressions. edit_file, system_info, and draft_change were widened in 0.23.0, because their floors stopped or passed a release on chance (below); widen a set when the model gets a case wrong in practice, keeping cases that do not resemble the prompt’s own examples.

How to run and record

scripts/check eval record

runs the evaluations (the ModelEvalTests target of the harness/Evals package) on the configured model with WISP_EVAL_RECORD pointing at harness/Sources/WispCore/Resources/measurements.json; each test merges its Measurement into that file, replacing the previous one for the same task, model, and input size. The suites run in parallel, so the merge holds a lock for the process and an advisory flock on the file’s directory for other processes, and replaces the file atomically: every suite’s measurement lands, which a plain eval record could not promise before 0.21.1 (two merges could interleave and one be lost). A file that exists but does not decode is refused and left as it is, with a not recorded line in the output, rather than replaced by the one new measurement.

Every run ends with its summary table and then exits non-zero when swift test failed: a floor missed, a strict case that threw, a build that failed. Before 0.21.1 the run’s status was grep’s, which matches the failure lines, so a failing eval, and the release’s floors with it, passed. compare goes on past a model whose run fails and names the failed models after its summary, without failing.

A case that fails because Apple’s model service refused it never reached the model, so it is counted apart, not as a failure of the model (EvalModels.serviceFaults, ModelServiceFault): its line says ERROR rather than FAIL, the summary table’s note counts the faults, and on a model that met one no floor is asserted. On the configured model the run then fails as not measured, naming the fault and the fix. The service is the Mac’s, so one fault puts every suite measured after it on that model in doubt. On 2026-10-13 modelmanagerd, 26 days up, ran out of open files part of the way through a run: system_info scored 17 of 54 with 34 cases refused, then 0 of 54, and both looked like scores. Restarting the service (sudo killall modelmanagerd) cleared it, and wisp doctor now asks the model for a reply, which catches the fault.

The evaluations are a Swift package of their own, harness/Evals, which depends on the harness package by path; the gate and the coverage runs never build it. Its tests @testable import WispCore, which works because SwiftPM builds a path dependency in debug with testing enabled (verified on 2026-09-30 with swift build --build-tests --package-path harness/Evals). The scripted model and the context evaluation’s fixtures are shared through the harness’s WispTestSupport library product. Building it without running a model: swift build --build-tests --package-path harness/Evals; listing the tests: swift test --package-path harness/Evals list.

A task that routes by input size (ADR 0037) records one measurement per size band, each with maxInputBytes, the largest input among its cases: the result is evidence for inputs up to that size and no further. WISP_EVAL_MODELS (comma-separated model spellings, such as system,ollama:qwen3.8:27b) measures each named model in turn in the suites that decide delegation (below), so a ladder’s rungs all have numbers; without it the eval measures the configured model only. A plain scripts/check eval, which is what a release runs, covers every suite but the context eval (scripts/check eval context runs that one, on purpose, for a design decision), asserts the floors, and records nothing: the sets are small, a rerun re-rolls the numbers (on 2026-09-22 two consecutive runs gave 5 and 6 of 6 for the same task), and the file should change only when someone means it to. The file is embedded at build time by the EmbedSystemPrompt plugin, so the numbers a binary reports are the numbers committed with it. Commit the file with the change that moved the numbers, and say so in the message. The release preflight runs the eval, so a release never ships with stale numbers.

The eval asserts only floors (half or three quarters recall, the classifier’s hard requirement, and for edit_file, system_info, and draft_change floors set from each suite’s history, below); everything else is reported and recorded. Run WISP_MODEL_TESTS=1 swift test --filter ToolEvalTests in harness/Evals for one suite.

Every run ends with a summary table, a row per model and a column per suite, each cell the cases passed of those run and the median time per case (per verdict for the classifier, per turn for the context scenario). scripts/check eval prints it and saves it beside the run’s whole output under harness/Evals/.build/evals (eval-<date>-<time>.log and .summary.txt). A cell with * has a note under the table (an unavailable model, a classifier’s fallbacks, a dangerous command rated safe); n/a is a suite the model could not run; - is a suite with no result, one that did not finish.

The case sets

edit_file, system_info, and draft_change have case sets large enough that a floor stops a release on a real regression rather than on chance (0.23.0). Before, ten files three times, eight questions twice, and five small diffs twice re-rolled from run to run (edit_file 16 to 25 of 30 over six runs), and a floor of half let a release pass or stop on luck. The sets are data files under harness/Evals/Tests/ModelEvalTests/Fixtures, read through EvalSets in WispTestSupport:

Claude wrote the new cases and labelled them on 2026-10-12, as the operator agreed for training and eval sets; none holds a real secret or personal data. The gate checks every set without a model (EvalSetsTests): each loads, its ids are unique, its counts match the counts it declares and the sizes above, every edit_file case is solved by exactly the calls its prompt asks for (applied with the real FileWriter, a named line being that line and the only one with its text, a stale number being wrong and find on exactly one other line), every whitespace case changes only one line’s whitespace, every file fits one read_file page at the smallest budget, every system_info topic and target is one the tool takes, every diff’s hunks match their headers and its size its band, no rubric term is contradicted by another, and the secret scanner finds nothing.

The draft rubric is explicit and checked in code, not by a model: a subject passes when one of the case’s must terms and none of its mustNot terms begins a word of it, ignoring case (fix is found in “Fix paging” and “hot-fix”, not in “prefix”). must names the gist, so a vague subject (“Update the uploader”) fails; mustNot names what would misdescribe the change, such as “reduce” for a timeout that went up or “drop” for an index added.

The floors

A floor is set from statistics: with n cases each passed at the configured model’s historical rate p, it is the largest k for which a run at that rate scores below k with a chance of at most 1% (the binomial distribution’s lower tail, EvalFloor.minimum), so a release stops on chance alone at most one time in a hundred. p is the configured model’s pooled score over the six eval runs saved from 2026-10-04 to 2026-10-12 and the full-set run of 2026-10-12 (the summaries under harness/Evals/.build/evals; the full run is the last figure of each row):

For example, edit_file on the whole set: at p = 0.662 a run of 86 scores 46 or fewer with a chance of 0.998% and 47 or fewer with more than 1%, so the floor is 47. The old floor of half of 30 stopped a run at that rate with a chance of 2.2% (more in practice, since three attempts at one file are not independent). A floor of 0 asserts nothing: the medium and large bands hold too few diffs at too low a rate for any floor to be safe, and stay evidence for routing. system_info‘s p was re-set on 2026-10-13 from the runs on its 54-case set alone. The 16-case set it replaced was easier (84 of 96 over six runs), and pooling those runs in gave 0.847 (127/150), well above the 0.72 every run on the new set scored; 0.24.1’s release eval stopped at 15 of 24 against that floor of 16 with no case erroring (below). The subset’s runs repeat cases of the full set, so the pool slightly overstates its independence. The other suites’ rates still pool their old and new sets. What a floor buys is shown by a fall of 20 points in the true rate: the whole sets would stop it with a chance of 93% (edit_file), 59% (whitespace), 85% (system_info), and 42% (small drafts); the release’s subsets 49%, 19%, 49%, and 22%. The rates and floors are in EvalFloor.Historical, and a gate test holds every figure in this table to the code. The new cases first ran on a model on 2026-10-12 (the whole sets, recorded); that run is pooled in, and the floors moved with it (edit_file 44 to 47 of 86 and 16 to 17 of 36, whitespace 19 to 18, system_info 41 to 39 and 17 to 16, small drafts 19 to 17). A later whole run pools in the same way: EvalFloor.Historical and this table move with it.

Run time, and the release’s subsets

At the configured model’s median times per case in the 2026-10-12 run (edit_file 32.4 s, whitespace 44.8 s, system_info 32.2 s, drafts 23.7 s small, 56.6 s medium, 156.7 s large), the three suites, one after another, would take:

The whole sets would add about 84 minutes to the release’s eval if the suites ran one after another, so the release runs a subset: a fixed number of each set (release in each file), chosen by EvalSample with a fixed seed and spread over the edit shapes and the system_info topics, so every release runs the same cases, about 7 minutes more than 0.22.0’s (distinct cases once each where it ran ten files three times). WISP_EVAL_FULL=1 runs the whole sets, with each floor computed for the cases run:

WISP_EVAL_FULL=1 scripts/check eval record

scripts/check eval without it, which the release preflight runs, and scripts/check eval compare run the subsets. Run the whole sets to record measurements, after a change to edit_file, system_info, draft_change, or the system prompt, and when a subset’s result is close to its floor.

The 128-minute figure assumed the suites ran one after another. They do not: the suites run in parallel, and the first full run, WISP_EVAL_FULL=1 scripts/check eval record on the on-device model on 2026-10-12 (11:02 to 11:43, at commit 7c93dcc), took 41 minutes for everything, the classifier and chat suites included.

system_info on its floor, 2026-10-13

0.24.0’s release eval passed system_info at exactly its floor, 16 of 24, so the whole set was run on the on-device model to tell chance from a decline, on the same code: 43 of 54 on 2026-10-12, then 36 and 39 of 54 on 2026-10-13 (the runs between those two met the service fault above and measured nothing). The whole set’s floor was then 39, from a rate of 0.85, at which a run of 54 averages about 46 with a spread of about 2.6, so 36 looked well outside chance. Nothing the eval exercises changed in the code between them (the system prompt and the tool are as they were at 0.23.0). 0.24.1’s release eval then scored 15 of 24 against the subset’s floor of 16, with no case erroring. Lined up, every run on the 54-case set, full or subset, scored about 0.72 (185/258), and only the runs of the old 16-case set scored higher: the rate was set too high by pooling the two sets, not lowered by a change in the model. The rate was re-set from the new set alone (The floors): floors of 31 of 54 and 12 of 24. No measurement was recorded from these runs.

The full run of 2026-10-12

On the on-device system model: edit_file.replace 61/86, edit_file.whitespace 20/24, system_info.topic 43/54, draft_change.commit 21/30 small, 4/8 medium, and 3/6 large; also triage 7/7, summarise_diff 5/5, respond.schema 6/6, redact.thorough 5/5, chat.unclear 24/24; and on the 392 labelled commands the default classifier with the rules on Core ML 357/392, the system model with the rules 318/392 (p50 5.3 s), and the system model alone 283/392 (2 dangerous commands rated safe). All are in measurements.json; the six edit_file, system_info, and draft figures are pooled into the floors above.

The verification suite ran afterwards on the same model at a874e36, not recorded, over 31 replies (17 turns of 2026-10-04, 9 of them fabricated; 14 hand-written, 7 fabricated):

Two of the model’s answers failed on the verdict’s then-required claim (now optional). The pair missed the floor it then had, which is why the floors moved to the readers, the default, and the model step stays off.

Comparing models

WISP_EVAL_MODELS=ollama:granite4.1:8b,ollama:llama3.2:3b scripts/check eval compare

runs the suites that decide delegation on each named model: tool calls (edit_file.replace) and schema replies (respond.schema) in ToolEvalTests, system_info.topic, triage, summarise_diff, draft_change.commit in its three bands, the classifier’s model fallback (classifier.system-model, the model alone, and classifier.system-model+rules, beside the rules as the gate runs it, with the named model in the on-device model’s place), and one context scenario. Add record to merge the measurements into measurements.json.

• One model at a time. compare runs swift test once per model, in the order named, with that model alone in WISP_EVAL_MODELS and its suites one after another (--no-parallel), so Ollama holds one model, evicts it once when the next starts, and a case’s time is that model’s alone. Setting WISP_EVAL_MODELS on a plain scripts/check eval also measures every named model, but its suites run in parallel, each looping over the models, so a local runtime switches between them and the times include the queue: use it for two or three quick models, compare for a comparison.

• The context scenario (ContextEvalTests/comparisonOnEachModel, which runs only when WISP_EVAL_MODELS is set): the baseline, the shortest scenario (14 turns, then six questions), through the whole stack (memory, on here though off by default since ADR 0057, facts, the summary, references, condensing to the target), at a window of 8,192 tokens, the on-device model’s, so every model condenses the same conversation; recorded as context.memory-target.window-8192.

• Floors are the release’s: they apply to the configured model only, ModelSelection.default (system), which is what a release’s eval has always measured, whatever model in config.json says. A compared model below a floor, or rating a dangerous command safe, is reported in its cell and fails nothing.

• Failures stay in their cell. A model that is missing or that the runtime refuses fails its suites with the reason as the cell’s note. A case that throws (a model without tool calling, a reply that does not parse) or that runs past five minutes (EvalModels.caseLimit, each classifier verdict too) counts as failed, and the run goes on (on the configured model an error in a suite that always failed on one still fails it); a context turn is bounded by ollama.timeoutSeconds and the scenario by an hour. A model resolves with ~/.wisp/config.json (its runtime’s address and timeout), which the eval only reads; the context scenario uses the defaults apart from its window. Since 0.21.1 RedactionEvalTests and ChatEvalTests go through EvalModels too: each case is bounded by the case limit, one that throws is a failed case rather than the end of the suite, each named model is measured, and each prints the eval result line the summary table reads (redaction, chat); they are not among compare’s suites.

• MLX models. When WISP_EVAL_MODELS names an mlx: model, the eval package is built with its own MLX trait, which turns on the harness’s, in a scratch path of its own (harness/Evals/.build/mlx), MLX’s Metal library is copied beside each test bundle’s binary as scripts/check mlx-live copies it (and removed before the build and when the run ends), and the eval tests register the MLX backend as wisp’s main does; a model then resolves from the MLX models directory with the capabilities ~/.wisp/config.json declares for it, so enable it (wisp models enable mlx:<name>) first, and an undeclared one fails its tool suites. The context scenario also holds an MLX model’s window at 8,192 tokens and keeps the operator’s MLX declarations. This needs the Metal toolchain, and the first build compiles MLX (minutes); a run naming no mlx: model builds and runs exactly as before. Proved on 2026-10-05 with mlx:Qwen3-1.7B-4bit alone on system_info (a copy of the script limited to that suite): 14/16, 1.2 s a case, the two misses topic: process with process: "all" for the busiest-CPU question. The Falcon comparison:

  WISP_EVAL_MODELS=mlx:Falcon-H1R-7B-4bit,mlx:Falcon-H1-7B-Instruct-4bit scripts/check eval compare
  

The local-model comparison, 2026-10-04

Seven Ollama models, scripts/check eval compare (not recorded into measurements.json), on this Mac with about 10 to 19 GB free, so every gemma4 and qwen window sat at the 8,192 floor. qwen3.8:27b was sampled on system_info and edit_file only, at 40 to 70 s a request; its draft scores are from 2026-09-24. Triage, summarise_diff, schema replies (5/6 or 6/6), and the context scenario (6/6, llama3.2:3b 5/6) did not tell the models apart and are left out.

Decided with the operator the same day: granite4.1:8b stays the delegation default; gemma4:12b takes complex work, with qwen3.8:27b for large diffs (AGENTS.md); llama3.2:3b and ministral-3:14b are not for tool work. ministral-3:14b is a parsing gap rather than the model: it writes read_file[ARGS]{…}, Mistral’s own call format, which its Ollama template leaves as text. Its 0/30 and 2/16 predate the fix of 2026-10-06, which reads those calls from the reply’s text (backends.md, “Ollama”); it has not been measured since. The gemma4 models’ 92% as a classifier, at seconds a verdict, makes them candidates for labelling training sets rather than for the gate. qwen3.8:27b ran 4 to 6 times slower than in September, likely because less memory was free.

granite4.1 against granite4.2, with and without thinking, 2026-10-09 and 2026-10-10

granite4.2:8b (IBM, September 2026; 8.8B, Q4_K_M, 5.35 GB, the same size as 4.1) against granite4.1:8b, with scripts/check eval compare on 2026-10-09 (17:02 to 23:17, the delegation suites and the model classifier). 4.2 can think and does by default; 4.1 cannot. On 2026-10-10 4.2 ran the delegation suites again, without the classifier, with ollama.think false in a scratch WISP_HOME (15:57 to 16:09). Cases passed, median time per case:

Thinking made 4.2 4 to 15 times slower and bought nothing in the delegation suites: not thinking, it matched 4.1 on the tools at the same speed (a median of 3.2 s a request, p90 9.0 s, over 300 requests) and was the only run with every draft right, at the cost of two system_info cases. Its classifier was the most accurate of the three with the rules (91%), at 14 s a verdict, too slow for approvals, which the Core ML classifier decides. granite4.1:8b stays the delegation default for now: ollama.think applies to every Ollama model, and turning it off would take thinking from gemma4:12b and qwen3.8:27b too. Choosing thinking per model or per call was planned for 0.22.0 and shipped in it: ollama.models.<name>.think, respond’s think, and no thinking for wisp’s own calls (ADR 0053, amended 2026-10-11).

qwen3.8:27b re-measured, 2026-10-09

system_info and edit_file on ollama:qwen3.8:27b with every other Ollama model unloaded (22.3 GB free) and the hybrid-layer sizing of ADR 0043’s 2026-10-05 refinement in place: 16/16 and 30/30, 144 requests at a median of 13.7 s (p90 25.1 s), the two suites taking 11 and 22 minutes against 8 and 25 on 2026-10-05. The 40 to 70 s a request recorded on 2026-10-04 came from the classifier suite, where qwen3.8 thinks before each verdict, not from its tool work; there was no slowdown. Its window stays at the 8,192 floor here: its 17.7 GB of weights take most of the memory budget, and it rises above the floor only with about 41 GB free (ollama.models.qwen3.8:27b.contextLength can hold it to a size).

The Falcon comparison, 2026-10-05

The two Falcon-H1 candidates through wisp’s own MLX executor (scripts/check eval compare, not recorded), after wisp models enable checked them: Falcon-H1R-7B passed the tool check, Falcon-H1-7B-Instruct did not (it names tools it was not offered), so it ran as text only. H1R took about 4 h 15 min, Instruct about 6 h 50 min, most of it the classifier’s 784 verdicts.

Neither earns a place: H1R trails granite4.1:8b on every suite at 4 to 15 times the time, and Instruct calls no tools, writes poor commit subjects, and stalls up to five minutes on some classifier verdicts. Both were disabled on 2026-10-06 (models.disabled), with llama3.2:3b, ministral-3:14b, nomic-embed-text, and deepseek-coder-v2; Falcon-H1-Tiny-Tool-Calling-90M is kept for the tool-choice experiment. The run showed hybrid models working end to end through wisp’s MLX executor, tool calls included.

MLX against Ollama, the same weights, 2026-10-06

mlx:Qwen3-1.7B-4bit through wisp’s MLX executor against ollama:qwen3:1.7b (Q4_K_M), on the suites that showed a gap, run with a copy of the script whose EVAL_COMPARE_SUITES named only edit_file, the schema replies, and the drafts (not recorded). Before is the full comparison of the morning; after is the fix that leaves thinking to the chat template and lets a schema reply think first (ADR 0052, refined 2026-10-06, which has the probes behind it).

Ollama’s two runs on the same code differ by five edit_file cases, so MLX after is within that spread; its drafts are close on content and two to three times slower. Without thinking the MLX conversion of the 1.7B weights fails the tool loop where Ollama’s quantisation does not, on the very same prompt; the 4B model does not have that weakness on MLX.

edit_file’s line rules, 2026-10-06

The two line-edit rules of ADR 0024 (refined 2026-10-06: a line that lost its indentation keeps it, and a stale number with find moves to the one line holding it) and the edited line shown in the result, measured before and after with a copy of the script whose EVAL_COMPARE_SUITES named only the two edit_file measurements (not recorded). Before is the code without the rules; llama3.2:3b ran with a scratch WISP_HOME whose config did not disable it; granite4.1:8b was run after only, to check a strong model is not harmed (28/30 on edit_file.replace on 2026-10-04).

What failed, from each failing case’s last call (edit_file.replace, then edit_file.whitespace):

The rules removed every failure they were for: no dropped indentation and no stale number with find remain, and no whitespace case was spoiled by the indentation rule. What is left is a wrong number without find (the Ollama 1.7B), which nothing can safely repair, and failures before the tool runs (llama3.2:3b’s text line, MLX’s empty sessions). Runs on the same code differ by up to five cases (“MLX against Ollama”, above), so the Ollama 1.7B’s gain and llama3.2:3b’s loss are within that spread; MLX’s 11 to 27 is not. granite’s four failures predate the rules: an empty find beside a right line has always changed nothing.

The first run, on 2026-10-04 with llama3.2:3b alone, took ten minutes, half of it the classifier’s 392 verdicts twice; a larger model takes longer per case, so allow an hour or more for each 26B or 27B model. The test output is buffered, so a model’s lines reach the log when its swift test ends.

Context checkpoint 2

WISP_EVAL_MODELS=system,ollama:granite4.1:8b scripts/check eval checkpoint

runs the second checkpoint of the layered-context design, whose plan, questions, and decision rules are proposals/2026-10-06-context-checkpoint-2.md. It is a measurement for design decisions, like eval context: the release’s eval, eval context, and eval compare never run it (ContextCheckpointTests runs only when WISP_CHECKPOINT names a part, which eval checkpoint sets). The cells, the scenario, and the table’s row are ContextCheckpoint and ContextEval.sustained() in WispTestSupport, tested in the gate without a model.

The parts run as one test each, one at a time. Each cell prints its turns and report as the context eval does, then a checkpoint row; the run ends with a table, a row per model and cell, saved as eval-<date>-<time>.summary.txt, and every row’s fields (the answers, the switches, the load) as .checkpoint.tsv beside the log. A cell the grid and the memory part share (the default target and headroom, t60-h8 since ADR 0057; t50-h8 when the checkpoint ran) runs once. The memory part’s cell with memory turns it on explicitly, though it is off by default. Add record to merge each run’s measurement into measurements.json. The main pass took 3 hours 40 minutes on the two models with the switches (2026-10-06), and gemma4:12b’s two memory cells 40 minutes more.

Results, 2026-10-06 to 2026-10-08

Run on this Mac without record, so nothing was merged into measurements.json; the logs and tables were kept outside the repository. The decisions they led to are ADR 0057. Answers are of 10 (sustained) or 7 (recalling); the main pass is one run a cell, the confirmation three, each listed, then the mean.

The main pass (2026-10-06 21:54 to 2026-10-07 02:13, load average 1 to 3), grid, answers of 10:

The rest of the main pass, one run each:

The confirmation, three runs (2026-10-08: assessment 09:09, memory on granite and gemma4 11:45, grid 14:24 to 18:09, memory on the on-device model 21:53 to 22:47; load average 1 to 5):

Beside the scores: the detail questions with memory 0, 1, and 4 of 6 (on-device, granite, gemma4) and without it 1, 1, and 6; gemma4’s 95th-percentile turn 188 to 358 s with memory and 27 to 190 s without; under any the task answer wrong in 2 of 3 on-device runs and 3 of 3 on granite, with 8 to 23 task changes, and under restated right in all six with 1 or 2; every default cell condensed at least once. Single on-device runs range from 1 to 8 of 10 on the same cell, so read differences of an answer as noise unless three runs agree. ADR 0057 has the per-cell condensations, floors, and times.

Reading a measurement

{ "task": "triage", "model": "system", "date": "2026-09-20", "passed": 7, "total": 7,
  "notes": "expected failures found across abridged swift build, swift test, cargo test, and pytest output, by test name or file:line" }

tool names the model tool the task exercises when it is one, so the catalogue can attach it. A task with no measurement for the configured model has not been measured there; that is not the same as failing.