Tongyi DeepResearch: From Base Model to Agentic Foundation

🌏 中文版

Previous articles covered individual pieces of the puzzle: the landscape (order 0), training from scratch (order 2), long-horizon memory (order 3), and planning optimization (order 4).

This article zooms out to see a complete system that threads all these insights together: Tongyi DeepResearch (arXiv:2510.24701, Alibaba Tongyi Lab).

Its core question: can you take a general-purpose pre-trained model and, through a complete pipeline, build an agent specifically designed for deep research?

Answer: yes. And not just capable — open-source SOTA.

Core Innovation: Agentic CPT

Most LLM training pipelines follow: pre-training → post-training fine-tuning.

Tongyi DeepResearch finds a fundamental problem with this flow: general pre-trained models lack "agent bias" — they don't know when to search, how to use tools, or how to manage long-horizon tasks. You can "patch" behavior on through post-training, but the model is fundamentally still a chat model.

Their solution: insert a mid-training stage between pre-training and post-training:

Pre-training → Agentic CPT → Agentic SFT → Agentic RL → Tongyi DeepResearch

What is Agentic CPT?

Agentic Continual Pre-training's core idea: instead of continuing pre-training on text, continue pre-training on synthetic agent trajectories.

Training data includes:

• Structured research process trajectories (question → search → read → synthesize)
• Tool call logs (when to invoke which tools)
• Graph-structured knowledge (associations)

Stage 1 uses 32K context length; Stage 2 expands to 128K. Through this process, the model learns the innate bias of being an agent — not behavior pinned by prescriptive prompts, but genuine understanding of what agent behavior means.

Why "pre-aligned"?

The paper's terminology is precise: Agentic CPT builds a pre-aligned agentic foundation model — before any post-training, the model already possesses foundational agent capabilities. The subsequent SFT and RL merely refine this base, not "patch" behavior from scratch.

Three-Stage Training Pipeline

Data Synthesis: AgentFounder

The pipeline's data is fully automated — no human annotation:

AgentFounder converts raw text, graph-structured knowledge, and tool logs into structured QA pairs and action sequences. Think of it as building a "memory palace" for the model — internalizing the structure of the research process into its parameters.

RL Training: Data > Algorithm

The Tongyi team makes an interesting observation:

"The algorithm is important but not the only decisive factor. Data and stability of the training environment are likely the more critical components."

They experimented with training directly on the BrowseComp test set and got substantially worse results than with synthetic data. Hypothesis: seeing answers directly causes overfitting, while synthetic data maintains generalization.

They used customized GRPO with token-level gradients and negative sample filtering for stable training.

Architecture: MoE + 128K Context

MoE advantages:

• Inference cost ≈ small dense model
• Retains specialist capacity (only 3.3B activated per token)
• 128K context supports long-horizon browsing sessions and iterative synthesis

Dual-Mode Inference

ReAct mode (standard): Core capability testing, single agent loop.

Heavy mode (IterResearch): Parallel context-managed agents + synthesis, targeting the hardest tasks. This is why HLE improves from 32.9 to 38.3.

Benchmark Performance: Open-Source SOTA

*OpenAI o3 BrowseComp scores are known but based on different prompting strategies. Tongyi wins on efficiency — achieving comparable or better results with 3B activated parameters.

Key takeaway: with 30B total / 3B activated parameters, surpasses larger closed systems like o3.

System View: Threading Previous Articles

Tongyi DeepResearch isn't a single technical breakthrough — it's a system integration. It threads together the insights from previous articles:

It proves one thing: deep research agents aren't built on a single technical breakthrough, but on systematic design of the entire training pipeline.

Key Takeaways

From Tongyi's practice, the most memorable lessons:

1. Pre-training doesn't produce agents — Agentic CPT is needed as a bridge
2. Data quality > algorithm innovation — synthetic data quality determines the system's ceiling
3. Training environment stability > algorithm choice — stability matters more than which RL algorithm
4. MoE is key for cost efficiency — 30B total activating 3B makes the system deployable in practice

What's Next

Final article: future outlook — embodied research, scientific automation, agent swarms.

參考資料

• Tongyi DeepResearch Technical Report — Kuan Li et al., Tongyi Lab, Alibaba. 30.5B MoE, Agentic CPT, HLE 32.9.
• Tongyi DeepResearch Blog — Official introduction with complete system overview.
• Tongyi DeepResearch GitHub — Model, framework, and solutions fully open-sourced.
• Scaling Agents via Continual Pre-training — Companion paper detailing Agentic CPT methodology.
• deep-research-survey-overview — Previous article: three-phase landscape classification.