Self-Hosting Open-Source LLMs: Framework Choice, Hardware Math, and When It Beats APIs

🌏 中文版

Ornith 35B-A3B scores 79.0 on SWE-bench, MiniMax M2.5 hits 80.2% — open-source models now match closed-source on coding tasks. But "the model is strong enough" and "I can actually run it" are separated by real decisions: which serving framework, what GPU, how much quantization, and when self-hosting beats API calls. This is the decision guide.

This site already has individual deep-dives on vLLM (in Chinese), Ollama (in Chinese), and llama.cpp (in Chinese). This post doesn't repeat their content — it covers the cross-framework comparison and cost math.

Four Frameworks, Four Use Cases

How to Choose

Per the beri.net 2026 inference framework guide, the current consensus is "default to vLLM, switch to SGLang for prefix-heavy workloads":

• Independent requests, high concurrency → vLLM. Largest community (89K+ GitHub stars), broadest model support, most debugging resources
• Multi-turn chat, RAG, shared system prompts → SGLang. Per PremAI benchmarks, SGLang hits 16,200 tokens/sec vs vLLM's 12,500 on prefix-heavy workloads — 29% faster. The gap narrows to 1-4% on independent requests
• Local dev, trying models → Ollama. ollama run ornith-1.5-9b handles download, quantization, and server in one command
• Extreme resource constraints (12GB GPU, CPU-only, mobile) → llama.cpp or Ollama (which uses llama.cpp under the hood)

Hardware Requirements

Running a model doesn't mean running it fast. Here are the minimum GPU memory requirements for popular open-source models (FP16 full precision vs Q4 quantized):

The MoE trap: Ornith 35B-A3B activates only 3B parameters per token (inference is fast), but all 35B parameters must be loaded into VRAM (memory requirements stay the same). Inference speed approaches a 3B model, but GPU memory needs approach a 35B model.

Quantization Format Selection

Rule of thumb: start with Q4_K_M (GGUF) or AWQ 4-bit, benchmark, and only upgrade precision if quality falls short. Most coding tasks see negligible quality loss at 4-bit quantization.

Cost Breakeven Analysis

Self-hosting costs more than GPU rental — there's also engineering time, ops overhead, and idle waste.

Cloud GPU Monthly Costs (August 2026)

API Cost Comparison (per million output tokens)

Per our vLLM self-hosting decision guide (in Chinese), the critical variable is GPU utilization. Above 50% utilization, self-hosting almost always wins. Below 10%, it's more expensive than most APIs.

Breakeven rule of thumb: if you consistently consume 100M+ tokens/month and can maintain GPU utilization above 30%, self-hosting starts to pay off. Below that volume, APIs (especially low-cost ones like DeepSeek and MiniMax) are more economical.

Plugging Into Agentic Coding CLIs

Any self-hosted model with an OpenAI-compatible API endpoint works with mainstream agentic coding tools:

# Start vLLM server
vllm serve ornith-ai/Ornith-1.5-35B-A3B --port 8000

# Connect Claude Code
export OPENAI_API_BASE=http://localhost:8000/v1
export OPENAI_API_KEY=EMPTY

# Connect OpenCode (~/.config/opencode/opencode.json)
{
  "provider": {
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://localhost:8000/v1" },
      "models": { "ornith-35b": { "name": "Ornith-1.5-35B-A3B" } }
    }
  }
}

vLLM and SGLang both natively support tool calling (function calling), which is essential for agentic workflows. Ollama supports it too, but with lower performance.

Watch Out For

1. Context length ≠ usable context: A model may claim 128K context support, but on consumer GPUs, KV cache memory limits may restrict actual usable context to 8-16K
2. Tool calling quality varies widely: Open-source models' tool calling stability still lags Claude / GPT — Ornith and MiniMax handle it relatively well, but general-purpose open models (Llama, Gemma) are prone to format errors
3. Batching matters: Single-request speed differences are small, but vLLM's continuous batching can multiply throughput several times in multi-user scenarios. Local dev doesn't need this
4. Quantization isn't free: Coding tasks are relatively precision-insensitive, but reasoning tasks (math, logic) degrade noticeably below Q3

References

• vLLM GitHub
• SGLang GitHub
• Ollama Website
• llama.cpp GitHub
• vLLM vs SGLang vs TensorRT-LLM 2026 Guide — beri.net
• SGLang vs vLLM Prefix-Heavy Throughput Benchmarks — GPU Insights
• SGLang 3.8% Gap Analysis — Spheron
• H100 / A100 Cloud Pricing — Thunder Compute
• GPU Rental Pricing Trends — Hyperstack
• H100 vs A100 Cost Efficiency — CloudZero
• vLLM Self-Hosting Decision Guide — this site (in Chinese)
• Ollama Complete Guide — this site (in Chinese)
• llama.cpp Inference Engine — this site (in Chinese)
• Ornith Model Family — this site
• MiniMax Model Family — this site