TODO: on-device task assessment, routing and audit

Date: 2026-09-19, updated 2026-10-04. Status: a living backlog. Items are ticked only where the code and a measurement show them done; each track’s progress note links the decisions that did it.

Goal and scope

Evaluate on-device AI for four jobs: input classification and pre-processing, dynamic context assembly, tool approval escalation classification, and local generation with tool execution. Select useful model and task combinations based on measured speed and accuracy. A specialist can be useful without handling every job.

This extends the local-model evaluation design. The installation record describes the existing Python MLX smoke checks. Those checks do not establish Swift provider compatibility or workload quality.

This backlog does not change the objective, accepted ADRs or runtime behaviour, and does not introduce remote generation. Explicit model selection and modular backends are implemented (backends.md, ADR 0019). The current work and the wider backlog are listed in section 7.

1. Input classification and pre-processing

“Optimal” means matching the model type, capability and execution settings to the request’s complexity, urgency and nature. It does not mean always selecting the fastest or largest model.

Progress. wisp routes by what it can measure before anything runs. An assessment of each request exists but stays off (below), and it chooses tools, not models:

• By input size. A task’s measured size bands pick a model from routing.ladder (ADR 0037).

• By task. routing.tasks sets a default model per task, and wisp ships measured defaults, such as secrets: system (ADR 0037, amendment).

• By memory. An Ollama model’s context window is sized from its shape and the Mac’s free memory (ADR 0043).

• A measured catalogue. Every eval task records results per model (ADR 0026, measurements.md).

• The assessment per request, built 2026-10-01 and off by default (assessment.enabled): rules, or one model call, choose the tools a request registers, the facts repeated beside it, and in chat the task. At the phase-6 checkpoint it scored lower on both models and cost 2 to 4 s a call (ADR 0045).

• Define a request profile covering intent, task type, modalities, complexity, urgency, requested reasoning effort, required capabilities, constraints and uncertainty.

• Distinguish explicit user preferences from inferred preferences. Preserve the original prompt and record any extracted or transformed input separately.

• Define complexity labels using reasoning depth, ambiguity and task dependencies. Permit selection of stronger reasoning, higher requested effort, or a larger model where measurements support it.

• Define urgency labels or time budgets. Prefer fast models or faster models with equivalent relevant capability when the request calls for speed.

• Define capability requirements for text generation, image analysis, extraction, coding, tool use and mixed requests. Do not infer capability from parameter count alone.

• Build a versioned model capability catalogue using measured quality, latency, memory, supported inputs, context limits and available execution settings. Partly: measured quality and latency per task (ADR 0026), declared capabilities per backend (ADR 0019), and context limits read or sized per model (ADR 0043). No version, and no execution settings.

• Define routing policy for conflicting preferences, uncertain assessments, unavailable models and requests no eligible local model can handle. Record the reason for each fallback or override. Partly: an explicit model always wins, a rung that cannot open is passed over, and each choice is audited as model.routed with its reason.

• Evaluate request assessment separately from model selection. Use an explicit user choice where feasible and make any inability to honour it visible.

2. Dynamic context assembly

Progress. Built and recorded in ADR 0045 on 2026-10-01, from the layered-context proposal: a store that refers to the audit log; a context composed for each request from the instructions, facts and a running summary, literal recent turns with tool output as a reference after its turn, and the task; memory to recall (off by default since 0.20.0, ADR 0057); condensing to a token target with headroom; and display decoupled from context. At the default budget the design kept everything in view for 22 turns where dropping lost every early fact (6/7 on device and 7/7 on granite, against 0/7 and 1/7).

• Condensing works on the on-device model since 2026-09-29. It counts the transcript and recognises the overflow message (ADR 0025, amendment).

• The consequences were measured. A planted fact was lost with the first condensation, and the model confidently misreported what came first (context-management.md).

• The context can be inspected. /inspect context saves the exact context, and every condensation saves the transcript before and after it; /inspect context next|N|turns, wisp-tui’s panel, and wisp://threads/{thread_id}/context show it at no model cost.

• Evaluate selection of relevant history, files, passages and tool results for the assessed request.

• Compare deterministic retrieval, embedding models, rerankers and generative models where applicable.

• Evaluate selection and compression separately. Retain source references, constraints, contradictory evidence and uncertainty in assembled context.

• Fit context to the selected model’s actual tokenizer and budget, reserving space for tools and output. Partly: the window is known per model (framework, sized, or configured), and the on-device model’s transcript is counted with its own tokenizer. Tool output stays bounded at 4 KiB in its own turn and is a reference after it, which answered question 9 instead of sizing it to the window (ADR 0045).

• Record which evidence was selected, omitted or compressed, and the context assembler’s version. Partly: condensations save the context before and after, name both files in the audit, and record their steps; cuts, references, distillations, summaries, recalls, and assessments are audited (context.*).

• Measure evidence retention, irrelevant content, assembly latency, token reduction and downstream answer accuracy. Include cases where an omitted detail changes the correct answer. Partly: the proposal’s eval, ContextEvalTests (now in harness/Evals), measures recall of planted and changed facts, order, and return to the task, for each phase and at the phase-6 checkpoint (ADR 0045).

3. Tool approval escalation classification

Progress. This track is largely done:

• Fast specialised classifiers judge each command beside the rules (ADR 0038).

• The shipped Core ML classifier is the default (ADR 0041).

• Measured closely. Five-fold cross-validation over 403 labelled dangerous commands rates 6 safe (1.5%, at most 2.9%) (ADR 0038, amendment).

• Evaluate proposed tool requests for required human escalation, separately from request routing.

• Preserve ADR 0011: the model can raise the rules’ verdict but cannot lower it, bypass policy or weaken the sandbox.

• Define handling of uncertain, malformed and failed classifications. Test the resulting approval behaviour explicitly, including clients that cannot obtain approval. Partly: below 0.6 confidence, a verdict is raised to at least moderate, and a classifier that fails falls back to moderate, both tested. Clients that cannot show an approval are the paused escalations proposal (2026-09-20-escalations.md).

• Measure missed required escalations, unnecessary escalations and decision latency separately. Set acceptance criteria that give missed escalations greater weight. (Dangerous rated safe, safe commands that ask, and latency per verdict are recorded separately, and the measure exits 1 when a dangerous command is rated safe.)

• Include ambiguous commands, contextual risks and misleading text in labelled evaluation cases. Partly: real and drafted commands with look-alikes, reviewed adversarially. The frozen test set holds 29 dangerous commands (25 until 2026-10-12, when 13 confirmed rows from real use joined it).

• Link the assessment, policy verdict and human decision to the originating prompt and tool call (classifier.verdict, policy.decision, and approval.decided carry the turn and the call).

4. On-device generation and tool execution

Progress:

• Evals. On-device evals run with scripts/check eval and publish their results (measurements.md).

• Ollama. Ollama models drive the same tool loop through wisp’s executor (ADR 0016). Since 2026-09-29, a required text argument such as system_info’s process, when a model leaves it out, is filled empty instead of ending the turn.

• Visibility. Chat shows the gate’s decisions, each turn’s tokens, and a live working line. MCP callers get progress.

• Evaluate bounded text generation, evidence-based answers and complete tool interactions locally. Partly: the eval suites cover tool calls, schema output, drafting, triage, and redaction.

• Verify model-specific chat templates, tool-call parsing, argument validation, tool results and conversation continuation through the intended Swift provider and packaged CLI (the on-device model through the framework; Ollama through wisp’s executor, tested over a fake server and live).

• Keep model proposals distinct from Wisp’s validated execution. Score actual completion and supporting evidence, not merely a plausible tool request or a claim of success.

• Measure end-to-end latency, first-attempt success, retries, unsupported claims and peak memory. Report policy denials, approval waits and execution failures separately. Partly: turn time and tokens per turn, /stats, and Ollama’s memory at a sized window (ADR 0043).

• Verify offline operation and compare direct use, MCP replay and parent-harness delegation as described in the existing evaluation design.

5. Prompt-linked transcript and audit records

Audit events carry id, session, turn and an optional call. Agent records the prompt before invoking the model, and the conversation’s store links each entry to the audit events that recorded it, saved with a transcript as <name>.store (since 2026-09-29). See logging, Agent, AuditEvent and TranscriptStore.

Progress:

• model.resolved records the model, its capabilities, and its window with why.

• model.routed records a routing choice with why.

• context.condensation names the saved context before and after.

• wisp logs --follow and /audit sessions let a person follow any session, MCP threads included.

• Every event has an id, and a store entry, an MCP calls item, and memory’s recall point to the event that holds the prompt, reply, or output; session.start names the sessions a resumed one came from (carriedFrom).

• Define stable prompt identity and its mapping to the transcript prompt entry. Support assessment before generation and failures that occur before a framework prompt entry exists. Partly: each store entry refers to its audit event by id, including a failed turn’s entries.

• Link the request profile and routing decision to that prompt ID in the persisted transcript/audit representation. Choose the storage mechanism without assuming the framework transcript accepts custom entries or arbitrary metadata.

• Record task type, complexity, urgency, requested effort, capabilities, explicit preferences and uncertainty. Treat reported confidence as uncalibrated until evaluated.

• Record the selected model and exact revision, runtime, generation/reasoning settings, classifier identity, classifier instructions/schema version, routing-policy version, timing and a concise decision reason. A reason is a decision summary, not a request to retain hidden reasoning. Partly: model, backend, asset, window, the classifier version and its verdict’s timing, and routing reasons. No exact model revision and no generation settings.

• Record context provenance, user overrides, unavailable candidates, fallbacks and later routing changes as linked events. Preserve the original assessment and decision.

• Append outcomes under the same prompt ID: elapsed time, completion status, retries, tool-call links and separately measured quality results. Include evaluator identity/version when quality is scored; an unscored result must not imply correct completion.

• Preserve correlation through streaming, retries, context condensation, transcript save/resume and concurrent MCP threads. Distinguish one logical prompt from its separate execution attempts.

• Define schema compatibility, audit-disabled behaviour and retention so saved records remain interpretable. Update AuditEvent.Kind, docs/logging.md and relevant transcript documentation.

6. Common model controls

The model-controls proposal defines the draft contract for reasoning mode, effort, native speed mode, separate performance preferences and reasoning output. These are still proposed controls, not implemented settings. Partly, since 0.19.0 (ADR 0053): Ollama’s think is requested through ollama.think (true, false, or a level), and, since 0.20.0, an MLX model’s through mlx.think (true or false) for a chat template that takes enable_thinking; a reasoning model’s thinking, such as qwen3.8:27b’s, is relayed to the person and the caller (chat, wisp-tui, wisp://threads/{id}/reasoning), audited as model.reasoning, and never sent back to the model. Since 0.22.0 (ADR 0053, amended 2026-10-11): one model’s default through <runtime>.models.<name>.think, a respond thread’s through think, and think: false on wisp’s own side calls (classifier verdicts, schema replies, the distiller, the summary, the assessment, memory’s condense).

• Describe supported controls and values per model/runtime/adapter combination, with provenance.
• Translate common controls in each backend and reject unsupported explicit requests before use.
• Keep reasoning generation separate from whether reasoning text is exposed to the caller.
• Expose supported native standard/fast modes as explicit controls and reject unsupported requests. Keep routing preferences separate; preserve explicit model, reasoning, capability, context, quality and locality constraints.
• Validate combinations with tools, schema output and streaming, rather than independent flags alone.
• Audit requested and resolved controls against the prompt/attempt, and evaluate their actual effects.

7. Current work and backlog (2026-10-12)

Committed, not yet released (0.24.0)

CHANGELOG.md, “Unreleased”: the inquiry verb (ADR 0062): the model asks the person one question mid-turn with its ask tool, off by default (tools.ask), answered in chat and wisp-tui, and under wisp mcp through wisp questions, plain chat’s /questions, or wisp-tui, the turn parking after 60 s and the answer reaching the thread’s next turn; the client’s question form built behind inquiry.elicitation, off until the operator’s hand test. Plain wisp chat showing and answering commands, facts, and questions waiting in wisp mcp servers. The approval banner posted only once the client’s dialog has gone unanswered for approval.bannerDelaySeconds (ADR 0046, amended). wisp-tui’s interactive status bar (ADR 0029, amended). The frozen risk test set grown from real use to 29 dangerous commands (ADR 0038, amended).

0.23.1 (2026-10-12): a patch for the secret scanner, which read prose such as “per token: every …” as a credential assigned to a name; a credential word inside a sentence now counts only with a value that mixes letters and digits and is not a number with a unit (ADR 0031, amended).

0.23.0 (2026-10-12) carried a verification pass beside each reply, whose exact readers, on by default, check the results a reply reports against the turn’s complete call list from the audit log and flag one no call produced as unsupported:, returned as verification by respond; its model step (verification.mode: model) built and off by default, since on the on-device model it flagged 6 of 14 honest replies (ADR 0060). Chat’s /show and /last and wisp-tui’s Ctrl-O showing a stored output whole, through the protocol’s output request. Larger eval sets for edit_file, system_info, and draft_change, with floors set from statistics and WISP_EVAL_FULL=1 for the whole sets. A request to keep a permanent fact waiting with no time limit and outliving its thread and its server (ADR 0048, amended); the terminal-only answers reviewed and kept (ADR 0046); processes named at spawn (ADR 0061). edit_file on CRLF and mixed-ending files, with a guard against a wider change (ADR 0024, refined 2026-10-12).

0.22.0 (2026-10-12) carried one tool-output budget from the model’s window, an eighth of it (4 KiB on the on-device model, 64 KiB at most), with what does not fit kept whole in the conversation’s output store and reached through memory by page, filter, exact summary, or a condensing question; the store swept by age (context.outputDays) and reported by wisp doctor; inspect listing a turn’s calls and a repeated call noted; the rest of the 2026-10-09 review (paged audit resources, one audit session per MCP thread, close-on-exec, read_file and edit_file bounds) (ADR 0050). One set of model actions for the person and the model: /models pull, trust, and cancel in chat and wisp-tui, and a models tool, off by default, whose every change asks the person (ADR 0059). Thinking chosen per model and per call, wisp’s own side calls sending think: false and respond taking think (ADR 0053, amended 2026-10-11). Ctrl-C stopping a model’s turn and its command. Shell loops judged by what they run, inline programs and pipes into an interpreter judged, a line asked once, approvals for folders that are gone marked and cleared, and the risk rules reading search patterns and printed text as data.

0.21.2 (2026-10-11): a patch for one fault found after 0.21.1. Under load, Swift’s cooperative thread pool could be filled by synchronous bridges (Blocking.run, for Core ML compiles and model resolution) waiting on work that itself needed the pool, so the command watchdog and every Timeout deadline fired tens of seconds late: a 200 ms timeout let a 30-second command finish. The bridges now run off the pool and the bounds are dispatch timers (design.md, “Concurrency”).

0.21.1 (2026-10-09): a patch from a code and test review of the whole codebase on 2026-10-09, about sixty fixes, each with a test. The approval gate judges every command a line runs (a ; inside parentheses hid one), remembers approvals only at the level given and keywords and interpreters only by exact text, and sees through quotes, continuations, and git -C; commands and edit_file can no longer write ~/.wisp; approvals and permanent facts merge across processes under a lock. wisp-tui no longer answers a dialog with typed keys and quits cleanly; an MCP thread runs one turn at a time. scripts/check eval fails when the evaluation fails, which it never could. The rest of the review is 0.22.0’s.

0.21.0 (2026-10-09) carried llama.cpp and LM Studio as backends through one shared executor for OpenAI-compatible servers, the window read from the server and the key from the environment or config.json (ADR 0058), tested against a fake server per dialect and live against Ollama’s OpenAI-compatible endpoint, not yet against a real llama-server or LM Studio; a context window for one model (ollama.models.<name>.contextLength, mlx.models.<name>.contextLength); wisp models pull resuming a file cut off part-way and seeding the Hugging Face cache from a real directory, and Core AI’s listing following links (ADR 0052, refined 2026-10-09); token usage in respond’s receipt and the response audit event; shell completions (wisp completions). Measured, not shipped as code: qwen3.8:27b re-measured, with no slowdown in tool work (measurements.md).

0.20.0 (2026-10-09) carried context checkpoint 2 run and decided (ADR 0057): the model’s memory tool off by default and turned on by context.memory, context.target 0.6, and assessment.taskChanges restated; windows sized for gemma4, reported per layer, and for hybrid models by their attention layers (ADR 0043, refined); on MLX, thinking left to the chat template unless mlx.think is set, a reasoning model’s thinking shown, schema replies that think first, tool calls as a JSON array, ChatML’s end of turn, and null arguments rendered (ADR 0052 and ADR 0053, refined); Mistral’s text tool calls read from Ollama replies; edit_file’s line edits forgiving a lost indentation and a stale number, and an empty find beside line (ADR 0024, refined); mlx: models in scripts/check eval compare, and scripts/check eval checkpoint. It also measured the local-model and Falcon comparisons, which kept granite4.1:8b as the delegation default (measurements.md).

0.19.0 (2026-10-04) carried MLX on wisp’s own executor, on a par with Ollama, and wisp models pull into the Hugging Face cache (ADR 0052); Core AI’s window from its bundle; the model’s thinking shown and ollama.think (ADR 0053); the sandbox’s refusals checked and a nested wisp denied to the model (ADR 0054); cited entries checked (ADR 0055); models enabled and disabled, the models table, capability checks on enable, and chat’s fallback to system (ADR 0056); an Ollama reply cut short ending the turn. 0.18.1 (2026-10-04): wisp chat from the Homebrew install finds wisp-tui again (in a bin beside libexec), and wisp doctor says where it found it. 0.18.0 (2026-10-04) carried commands the person types in chat (ADR 0049), the running summary in a place of its own, what each turn ran beside its reply (ADR 0051), a fact’s value held once across sources, and smaller fixes. 0.17.0 (2026-10-04) carried permanent facts over MCP (ADR 0048) and MLX in the release (ADR 0047). Their sections of CHANGELOG.md have the detail.

Planned next

The roadmap ends with 0.24.0, the release being made; what is left is under its “Not scheduled”, among it the verification pass’s model step, waiting on a more capable model, and the hand test of the client’s question form (ADR 0062, D8). Still to measure from 0.20.0’s list: the rest of what ADR 0052 lists under “What 0.20.0 must measure” beyond the Qwen3-1.7B comparison.

Open from the layered context

• memory on a second scenario, one that edits, builds, and tests, or at a far larger window;
• a specialised distiller, once reviewed pairs exist.

Done 2026-10-09: the guard’s 50% variants re-run, the target and headroom tuned, the assessment’s task changed only when restated, and D10’s model switch evaluated, in context checkpoint 2 (ADR 0057). Done 2026-10-04: window sizing for Core AI and MLX, from their bundles’ metadata (ADR 0052).

Offered, not started

• a test for the singular form of /tokens.

Carried over

• Secrets:
  • the gap between the drafted training data and third-party test data (85% against 32% of secrets found);
  • personal data in redact, which needs the model to name values within flagged lines;
  • the classifier’s low confidence on git push, which is a candidate for training data.
• Classifiers:
  • growing the dangerous test set beyond 25;
  • wiring in the failures classifier, which beats KnownFailures on test;
  • new providers, such as embeddings or an external process.
• From the backlog:
  • condensers that summarise a file, answer over several files, and extract fields;
  • reverse delegation through MCP sampling;
  • client roots;
  • per-chunk progress in the condensing tools;
  • bulk classification;
  • offline use;
  • the notification helper app;
  • Private Cloud Compute signing.
• Open: the inquiry verb of the escalations proposal; its approval part was settled by ADR 0046.

Evaluation and completion criteria

• Adapt the existing workload cases into the four tracks, with separate development and held-out cases. Define expected results, latency budgets and accuracy thresholds before comparing candidates. Partly: the classifier tasks have train, dev, and frozen test sets (ADR 0038).
• Use deterministic baselines where they can solve the task. Compare model/settings combinations, including reasoning levels, and account for assessment and context-assembly overhead. Partly: rules baselines for risk, failures, log severity, and secrets (wisp classifier baseline), and exact condensers ahead of the model (ADR 0039).
• Report speed versus accuracy per workload. Measure cold/warm execution, short/long inputs, retries and fallbacks; include time to a correct result where correctness can be established.
• For recurrent/state-space candidates, test fact retention, corrections and continued state alongside memory use. Distinguish base-model limitations from architecture or runtime failures.
• Select initial candidates from the researched shortlist without declaring architecture or parameter count a winner in advance. Pin assets and runtimes before each comparison.
• Produce a suitability table identifying useful model/task/settings combinations, measured limits and fallback conditions for each track.
• Add model-independent tests for routing policy, identity correlation and audit persistence. Keep live-model evaluation separate from the unit-test gate, consistent with repository guidance. Partly: routing, sizing, and audit fields are tested without a model.
• Demonstrate that an auditor can trace a prompt through assessment, selection, context assembly, tool approvals, execution and outcome, including any changes to the original decision.