Prompt Version Control: Changing One Word Can Drop an Eval from 5/5 to 0/5

TL;DR Rivumi's prompt is now `m3-exact-edit-v4`: the version persists into artifacts; core/tool/interaction/runtime/instructions/skills/workspace/memory are composed as stable or dynamic sections; and positive/negative examples cover replace_text, unified diffs, and direct replies. Unit tests pin the structure, while live-eval coverage still needs expansion.

#coding-agent #prompt-engineering #versioning #eval #ab-testing

Table of Contents

The design problem: how do you review a prompt change?
What the five do
rivumi's choice and how it differs
Engineering evidence
Improvement roadmap
References

Series: 跟成熟 coding agent 學設計 (25 / 38)

Learning Agent Design from Mature Coding Agents (24): Testing a Moving Agent — fake-CLI Contracts, Recorded Streams, TUI Pilot Learning Design from Mature Coding Agents (26): Context Compression and Compaction — From Gap to Auditable Baseline

🌏 中文版

The design problem: how do you review a prompt change?

A prompt is the strangest kind of code in a coding agent: it has no types and no unit test catches it directly, yet changing one word can wreck task success rates. There is hard academic evidence for this — FormatSpread measured performance of semantically identical content across different prompt formats and found spreads of up to 76 accuracy points. A "just rewording" prompt diff therefore carries risk comparable to changing a core algorithm.

Standard code review is helpless here: a reviewer cannot tell whether changing "prefer X" to "always use X" will make the model refuse tool Y on certain tasks. The real question is: how do prompt changes get tracked, reviewed, and verified? Five mature projects give four different answers.

What the five do

codex treats prompts as assets shipped alongside models. The openai/codex repo keeps a row of prompt files named by model generation at codex/codex-rs/core/gpt_5_1_prompt.md (331 lines), gpt_5_codex_prompt.md, gpt_5_2_prompt.md, and so on — each model family gets its own complete system prompt rather than conditional branches inside one shared text. The runtime selection chain lives in codex/codex-rs/models-manager/src/model_info.rs#with_config_overrides: user config overrides > file loading > built-in fallback (the fallback constant BASE_INSTRUCTIONS is compiled in via include_str!("../prompt.md")); the official per-model prompt text ships with the model catalog (codex-rs/models-manager/models.json), where each model's instructions_template runs about 17K characters. Secondary prompts (compaction summaries, review, permission explanations) are centralized in template modules like codex/codex-rs/prompts/src/compact.rs#SUMMARIZATION_PROMPT.

opencode is the minimal file-as-version variant: sst/opencode keeps roughly a dozen per-model prompts in opencode/packages/opencode/src/session/prompt/ (anthropic.txt, gpt.txt, gemini.txt, kimi.txt, codex.txt, …), selected by string matching on the model id in opencode/packages/opencode/src/session/system.ts#provider. No fancy machinery, but "which model runs which prompt version" is legible from the filesystem alone.

claude-code is the most elaborate. In the decompiled source at anthropics/claude-code, claude-code-source/src/constants/prompts.ts#getSystemPrompt assembles the system prompt as an array of sections with two key designs. First, feature flags directly determine prompt content: gates like feature('KAIROS') and feature('EXPERIMENTAL_SKILL_SEARCH') decide whether an instruction block appears; internal employees (USER_TYPE === 'ant') additionally see sections annotated with their experimental purpose — comments literally read "un-gate once validated on external via A/B" alongside effectiveness numbers. That is live A/B testing on prompts. Second, a marker constant SYSTEM_PROMPT_DYNAMIC_BOUNDARY separates static sections (cacheable across users) from dynamic ones (session-specific, not cacheable), directly serving the prompt-cache hit rate.

omp (can1357/oh-my-pi) makes assembly a template render: oh-my-pi/packages/coding-agent/src/system-prompt.ts#buildSystemPrompt imports a dozen .md templates (main template, three personality presets, safety sections), gathers environment info, tool lists, and skills into template data, and even deduplicates rules at paragraph level so a user's AGENTS.md doesn't repeat built-in guidance.

pi (badlogic/pi-mono) is the minimal control group: pi-mono/packages/coding-agent/src/core/system-prompt.ts#buildSystemPrompt is plain string concatenation — customPrompt replaces the default wholesale, plus tool lists and context files. No version number, because upstream relies on git itself.

They share exactly one conviction: the prompt is not a string literal scattered through code but a first-class asset — standalone files, centralized management, explicit selection logic.

rivumi's choice and how it differs

rivumi takes the road none of the five go quite as far on: a semantic version constant in the prompt string, with every evolution bound to eval evidence.

rivumi/src/rivumi/prompts.py#CODING_AGENT_PROMPT_VERSION now reads "m3-exact-edit-v4". The version still persists into sessions and run.created, but the system prompt is no longer one bare string. #PromptSection, #render_prompt_sections, and #build_coding_agent_system_prompt compose ordered core policy, tool policy, interaction policy, runtime context, instructions, skills, workspace state, and memory, with explicit stable/dynamic cache metadata. This is an assembly baseline; it does not prove every provider uses the same cache protocol or achieves production trace hit rates.

The v1→v3 evolution is textbook observation-driven iteration:

v1 (M3): only the replace_text vs apply_patch division of labor and read-before-edit, added to rescue qwen3:4b after it found the correct fix twice but emitted malformed unified diffs. Running the real Ollama eval with v1 passed 5/5.
v2: interactive use showed the agent exploring the repo and running checks even for greetings. The diagnosis doc docs/diagnoses/conversational-turn-redesign.md explicitly records that the fix borrowed kimi.txt's conditional rule style and codex's chit-chat wording — not vague advice but a trigger→action branch: when no change was made, skip straight to the answer and don't touch the repository.
v3: tightened further — capability questions ("can you help me write a program?") also deserve a direct reply, without exploring the repo or enumerating interpretations to disambiguate.
v4: replaces some abstract advice with a compact examples section. A positive example demonstrates byte-for-byte read_file → replace_text, another shows unified-diff shape, a negative example forbids guessed old text, and a direct-reply example keeps greetings, small talk, and capability questions tool-free. tests/test_prompts.py pins the version, examples, and section ordering.

The contrast with the five is clear: codex and claude-code have eval infrastructure but don't publish per-change eval evidence alongside individual prompt edits; opencode and pi lean on git history. rivumi binds "version number → observed failure → eval result" into one commit chain. The cost is equally honest: the eval covers one small Python task on one local 4B model — 5/5 does not mean broadly reliable, as the stage doc itself states upfront.

Engineering evidence

OpenAI's GPT-4.1 prompting guide explicitly recommends iterating, evaluating, and iterating again for agentic prompts — treating them as tested programs, not one-off copy. Anthropic's prompt engineering docs likewise list "build an eval first" as a prerequisite for touching prompts. SWE-agent makes the deeper point: agent performance is extremely sensitive to interface design, including tool guidance in prompts — the interface is the engineering. And claude-code's cache boundary design maps onto Anthropic's prompt caching practice — static prefixes are cacheable, dynamic tails must not leak into them.

Improvement roadmap

Diversify the eval manifest: examples and v4 have unit tests, but live evaluation still centers on a tiny Python task. Add at least "pure Q&A calls no tools" and "bad old_text forces a reread" cases before claiming the examples changed model behavior.
Prompt diffs in CI: pinning clauses in tests/test_prompts.py is a good first step; next, require every prompt version bump to reference an eval summary path, following the M3 stage doc's evidence format.
Validate section/cache strategy with traces: named stable/dynamic sections have landed. The next step is confirming each provider payload preserves the stable prefix and cache traces explain hits and misses before changing default ordering again.
No catalog needed yet: codex's per-model prompt catalog serves dozens of models; rivumi only needs its provider adapter layer to record "which prompt version was evaluated against which models."