Skip to content
All tags

#llm-api

5 posts

Learning Design from Mature Coding Agents (6): The ModelProvider Abstraction — Why Wrapping an SDK Is Not Enough

Wrapping an SDK directly buys you three walls within months: usage fields that don't agree, error semantics tied to SDK exception types, and tool-call formats that change per provider. All five reference projects separate 'wire protocol' from 'provider identity' as independent dimensions. Rivumi goes further with pydantic canonical contracts (Message/ToolCall/Usage/ModelTurn) plus six protocol adapters, forces the OpenAI SDK's built-in retries to zero, and routes every failure through a classified ProviderErrorKind before any retry policy sees it. Its provider table is deliberately copied from pi's packages/ai — lineage, not coincidence.

Learning Design from Mature Coding Agents (7): Provider Retry Policy — From One 5xx to Bounded Retry and Fallback

Intermittent NVIDIA NIM 500s exposed Rivumi's early gap: classified errors with no retry consumer. SDK retries are now disabled; the harness gives each candidate up to five attempts with jittered exponential backoff and capped Retry-After handling, then can move to an explicitly configured fallback model. Both model.retry and model.fallback enter the event log.

ai deep-dive

BytePlus ModelArk Coding Plan: ByteDance's AI Coding Subscription

BytePlus ModelArk Coding Plan offers Lite ($10/month) and Pro ($50/month) subscriptions covering models such as DeepSeek-V4, GLM-5.2, and Seed-2.0 in tools including Claude Code and Cursor. Lite includes about 24,000 requests per month; Pro includes five times as many.

Learning Design from Mature Coding Agents (22): The Gateway Pattern — Turning Any Provider into an OpenAI-Compatible Endpoint

The ecosystem treats /v1/chat/completions as the lingua franca, but your providers don't all speak it. The five reference projects split into three camps: pi and OpenCode make the client speak every dialect natively so no gateway is needed; OMP builds a real protocol translator (foreign wire → neutral context → provider adapter, no raw passthrough); Codex and Claude Code run proxies that translate nothing and exist purely to force traffic through a controllable path. Rivumi copies OMP's boundary but narrows it to one wire in, one out: strictly parse OpenAI Chat into a canonical contract, then dispatch to any ModelProvider — and along the way hit a cross-event-loop client-close bug whose lesson is that provider lifecycles belong to the ASGI lifespan, not the signal handler.

ai deep-dive

Amazon Bedrock Deep Dive: Putting Model APIs Inside the AWS Governance Boundary

Amazon Bedrock is more than a reseller for multiple model APIs. It brings model invocation, IAM, Regions, Knowledge Bases, Guardrails, and CloudWatch into one AWS control plane. It fits teams already on AWS that value governance overhead more than the lowest token price.