Debugging Ask AI in Production: SSE, Traces, Cache, Checkpoints, and Shadow Runs

🌏 中文版

Optional companion reading: Beginners can read this article directly. For extra context, see RAG Streaming, RAG Observability, and Semantic Caching.

When a user reports that an answer looked wrong, Ask AI has no single file that reconstructs the entire run. Public SSE records what the user received. Langfuse and D1 store execution summaries. A semantic-cache hit may prevent the pipeline from running at all. A checkpoint may inject a summary from an earlier turn.

This article follows one request over time. The implementation centers on /api/chat and the shared pipeline facade.

A request first resolves authority, cache, and history

After receiving message, the API verifies the session, applies visitor quota, and loads RAG settings, provider keys, and the thread checkpoint. traceScope and cacheMode pass through the request policy: only an authenticated admin request may select evaluation tracing or cacheMode: bypass.

Only then does the API query the semantic cache. A cache hit returns one token event containing the stored answer and a done event with cached: true. Planner, Research, Writer, Validation, and Critic do not run.

This distinction changes incident diagnosis. A cached answer without sources does not demonstrate that the new retriever missed them. It demonstrates that the user received an older stored response.

The pipeline emits only the accepted answer

On a cache miss, the pipeline facade delegates to the configured engine. The engine may produce several Writer drafts internally. The facade suppresses those intermediate onToken callbacks, normalizes the lifecycle output, and emits only final_response.

This boundary fixes a confusing UI failure. If three retries each stream a draft and the client appends token events, the browser displays three answers concatenated together. Agent-step events can still reveal repeated stages, but only the accepted response becomes answer text.

The public stream may include:

• agent_step: stage name, completed status, and a small summary.
• token: accepted answer text.
• related: related-reading items produced by the pipeline.
• sources: deduplicated sources after Validation and Critic gates.
• done: token usage, confidence, thread ID, cache status, and remaining quota.
• error: public error type and message.

It does not include raw ranked chunks, the full prompt context, every rejected draft, or the complete Critic JSON.

Langfuse and D1 preserve execution summaries

Ask AI creates a blog-rag trace with pipeline engine, thread ID, cache mode, trace scope, and request start time. Completion adds duration, confidence, answer relevance, intent alignment, drift, result count, model usage, and lifecycle trace steps.

Each step is also mapped to a span. If the mapping cannot be exact, it is marked as fallback or unobserved instead of pretending every timestamp was measured. Calls through the Langfuse adapter are queued without blocking chat availability when the external observability service fails.

D1 separately stores chat_logs and step records. These can answer which stage repeated, how long the request took, and what the final score was. They still are not a complete model trace. Public agent_step events and backend trace rows are projections of a run, not an engine-state dump.

A checkpoint can change the next query's context

The checkpoint implementation estimates tokens across messages, drafts, and retrieved evidence. Above the configured ratio, it stores a summary containing the latest question, answer, and coverage gaps. The next request with the same thread_id loads that summary.

A short follow-up such as “What about leveraged ETFs?” therefore cannot always be reproduced from its query string alone. The prior thread summary is also input. Use a new thread for a clean regression case; preserve the original thread ID when testing context contamination.

A shadow run is hidden comparison, not the production answer

In RAG settings, shadowModeEnabled defaults to false. When enabled, the completed primary request triggers a baseline configuration with HyDE, Multi-Query, reranking, Critic, and PageIndex disabled. Primary and shadow responses plus confidence values are written to shadow_runs.

All shadow callbacks are no-ops. Users never see shadow tokens, steps, or sources, and the shadow response never replaces the primary answer. A shadow_runs row alone does not prove that one configuration is better; that conclusion still requires a fixed dataset, scoring rules, and inspectable artifacts.

A reproducible observation flow

After starting Astro with its normal Workers bindings, preserve the public SSE stream:

curl -N -X POST http://127.0.0.1:4321/api/chat \
  -H 'Content-Type: application/json' \
  --data '{"message":"What course articles do you have?","thread_id":"obs-clean-01"}'

To exclude semantic cache, use an authenticated admin session. A public request should not obtain bypass authority:

curl -N -X POST http://127.0.0.1:4321/api/chat \
  -H 'Content-Type: application/json' \
  -H 'Cookie: session=admin-session-cookie' \
  --data '{"message":"What course articles do you have?","thread_id":"obs-clean-02","traceScope":"eval","cacheMode":"bypass"}'

Run policy and facade tests to protect bypass authorization and final-only output:

pnpm exec vitest run \
  src/lib/conversation/pipeline.test.ts \
  src/lib/conversation/request-policy.test.ts \
  src/pages/api/chat.policy.test.ts

Record cached, thread ID, agent-step sequence, sources, done or error payload, and backend trace ID. Without those fields, a later investigation can easily misread a cache hit as retrieval failure or a public stage sequence as a full prompt trace.

What each layer can prove

More telemetry does not erase evidence boundaries. State which layer you have before making a diagnosis; that is more reliable than calling every log a “trace.”

References

• Ask AI API lifecycle and SSE
• Shared pipeline final-response facade
• Semantic cache implementation
• Conversation checkpoint implementation
• RAG runtime settings and shadow baseline
• Ask AI evaluation evidence boundary