Skip to content
All tags

#eval

4 posts

Prompt Version Control: Changing One Word Can Drop an Eval from 5/5 to 0/5

Rivumi's prompt is now `m3-exact-edit-v4`: the version persists into artifacts; core/tool/interaction/runtime/instructions/skills/workspace/memory are composed as stable or dynamic sections; and positive/negative examples cover replace_text, unified diffs, and direct replies. Unit tests pin the structure, while live-eval coverage still needs expansion.

tech deep-dive

Building a Taiwan Stock Research Agent (Part 5): Walk-Forward Evaluation, Run Cards, and an Honest 50% Baseline

I do not measure whether the agent ‘feels accurate.’ I freeze parameters in walk-forward OOS tests, record a hash of every input in run cards, and keep the honest 5/10 = 50% golden-eval baseline so the agent has to admit that it is not accurate yet.

tech deep-dive

Building a Taiwan Stock Research Agent (Part 6): Making Every Number in an LLM Report Auditable

Numbers are the easiest part of an LLM report to hallucinate. I therefore put every trusted number into a SHA-256-addressed evidence manifest and let the LLM cite only {{fact.id}} placeholders. If it writes a bare number, the entire output is discarded and replaced with a deterministic template.

tech deep-dive

Building a Taiwan Stock Research Agent (Part 7): The Copilot Loop—Plan Contracts, Verifiable Sources, and Human Review

A research request first becomes a ResearchPlan that requires human approval. External documents must be fetched in full, and verbatim quotes must be verified before they can enter a report. Quant review is always append-only, and free-text feedback never flows back into a prompt. This is the complete M5 Copilot loop.