Skip to content

Modal: The Layer Your Inference Engine Runs On — and When the Premium Isn't Worth It

Aug 21, 2026 1 min
Home › ai
TL;DR Modal is a per-second-billed serverless GPU platform that also treats agent sandboxes as a first-class primitive (company-reported: over 1 billion sandboxes launched, more than a third of revenue). The selection question isn't how convenient it is — it's your GPU utilization. Verified 2026-08-21: Modal's A100 80GB works out to $2.50/hr against RunPod's $1.59/hr for the same card, so above 64% utilization renting your own is cheaper. But on the same day, H100 SXM is $3.95/hr on Modal against $3.99 on Lambda — on that card the premium is gone.
#modal#serverless-gpu#sandbox#gpu#ai-agent#cost
Table of Contents
  1. 1. What Modal is
  2. 2. Scale: revenue and adoption first, funding last
  3. 3. First criterion: does your utilization clear 60%
  4. 4. Second criterion: are you buying the price gap or the software layer
  5. 5. Cold starts: the actual technical battleground
  6. 6. Third criterion: your agent runs untrusted code
  7. 7. The honest limits
  8. Overall
  9. References
Series: Technology Choices in the AI Era (13 / 128)
Exa: Neural Search Built for Agents, Not People Composio: Who Holds Every User's Token When Your Agent Connects a Hundred SaaS Apps

🌏 中文版

This site has covered how to pick an inference engine — vLLM's internals, when self-hosting is worth it — but never the layer underneath: the machines the engine actually runs on. This post fills that gap, through the company currently making the most noise in that layer.

Every price, limit, and scale number here was verified on 2026-08-21, with vendor-reported figures marked as such.

1. What Modal is

Modal is a New York cloud computing company, and what it sells fits in one sentence: put a decorator on a Python function and it runs on a cloud GPU, billed by the second, scaled to zero when nothing is calling it.

import modal

image = modal.Image.debian_slim().pip_install("torch", "transformers")
app = modal.App(image=image)

@app.function(gpu="H100", scaledown_window=300)
def generate(prompt: str):
    ...

No Dockerfile, no Kubernetes manifest, no instance you have to remember to shut down. The Image is declared in code, gpu="H100" is an argument, and scaledown_window decides how long an idle container survives (per the official docs: 60 seconds by default, configurable from two seconds to twenty minutes).

It does a second thing too: Sandboxes — containers defined at runtime for executing untrusted code. The first use case listed in the official guide is "execute code generated by a language model." A Sandbox lives five minutes by default and can be stretched to 24 hours.

Both layers share one substrate. Isolation is Google's gVisor, stated in Modal's security documentation; the same page notes that the CLI and client library are open source, while the platform is not.

2. Scale: revenue and adoption first, funding last

On 2026-05-21 the company blog reported three numbers. In descending order of how much they tell you:

  • Annualized revenue above $300 million, a fivefold increase since September 2025
  • Over 1 billion sandboxes launched cumulatively, with sandboxes now "more than a third of our revenue"
  • A $355 million Series C at a $4.65 billion post-money valuation, led by General Catalyst and Redpoint

Reuters added something the blog post left out: the round came in two tranches, the first at a $2.5 billion valuation, with the second pushed to $4.65 billion as demand grew — Accel and Menlo came in on the second. The same report, citing the company, puts the revenue jump at roughly $60 million in September to roughly $300 million.

Revenue and sandbox counts are both self-reported with no third-party audit. They are still far more useful than the valuation: a valuation is a group of investors' view of the future, while revenue growth and sandbox volume are people paying to run things right now. Only the latter belongs in your selection table.

This site has already brushed against Modal: the post on internal AI coding agents noted that Ramp's Inspect runs in Modal containers, and Modal's Series C announcement says Inspect now authors 70% of that company's merged PRs.

3. First criterion: does your utilization clear 60%

Rent a GPU machine and the bill is unrelated to whether you use it; Modal bills only while the container is alive. So the decision collapses into one number: what fraction of the day your GPU is actually computing.

Verified on three vendor sites on 2026-08-21, normalized to dollars per GPU-hour (Modal lists per-second prices; multiplied by 3600 here):

GPUModal (per-second, as /hr)RunPod PodLambda on-demand
H100 SXM 80GB$3.95$3.29$3.99
A100 SXM 80GB$2.50$1.59$2.79
L40S 48GB$1.95$0.99—
B200 180GB$6.25$6.79$6.69

The point of this table isn't "Modal is more expensive." It's that the premium swings violently by card. A100 against RunPod is 1.57x, L40S is 1.97x — but H100 against Lambda is 0.99x. On the same day, Modal's H100 is cheaper than Lambda's.

Converted into break-even utilization (Modal price × utilization = the competitor's full-month price):

ComparisonBreak-even utilization
A100 vs RunPod64%
L40S vs RunPod51%
H100 vs Lambda101% (above 100%, meaning Modal always wins on this card)

What to do: pull up last month's GPU monitoring and compute "seconds with utilization > 0 ÷ seconds the machine was powered on." Below 50%, serverless almost certainly wins. Above 80% with steady traffic, you're paying a premium for idle time. In between, it depends on the card — the table above says the H100 carries no premium and the A100 does.

4. Second criterion: are you buying the price gap or the software layer

The math above hides an assumption: that both sides deliver the same thing. They don't.

A bare GPU provider hands you a booted machine and the rest is yours: building container images, caching model weights, writing the autoscaler, reclaiming idle capacity, scheduling across regions. That's exactly what Modal sells — the four autoscaler parameters (min_containers, max_containers, buffer_containers, scaledown_window) are a whole team's worth of work you would otherwise build yourself.

So the real question isn't "who's cheaper," it's: is the saved price gap enough to pay someone to maintain that layer? One A100 costs $654 more per month on Modal ($2.50 − $1.59, running all 720 hours). Three cards is roughly one engineer's monthly salary in Taiwan; thirty cards and building it yourself is obviously right. That is the only arithmetic this criterion needs.

5. Cold starts: the actual technical battleground

Serverless dies on the first request after scaling to zero. Loading a large model takes tens of seconds, which is why most people give up on scale-to-zero.

Modal's answer is to save the container's memory wholesale. CPU Memory Snapshots freeze the memory state once a container is warm and restore it on every subsequent cold boot; in July 2025 they added GPU Memory Snapshots (still alpha), which additionally capture GPU memory, CUDA context, and torch.compile artifacts. The company blog's example is vLLM serving Qwen2.5-0.5B-Instruct, where cold start drops from 45 seconds to 5 (Modal's own measurement, not third-party).

But the feature's limitations run longer than its benefits, and they're all in the official docs: multi-GPU code is generally incompatible, non-CUDA GPU code fails, and torch.compile can break snapshot creation (fixable in some cases by setting TORCHINDUCTOR_COMPILE_THREADS=1). The line worth memorizing is this one:

That means that if the majority of your initialization latency is spent loading weights, GPU Memory Snapshots will generally not improve your cold start times — and may even worsen them, by adding overhead.

What to do: before enabling snapshots, measure once and split cold-start time into "reading weights" and "everything else." If weights dominate, the fix is baking them into the Image or a Volume, not turning on snapshots.

6. Third criterion: your agent runs untrusted code

The first two criteria compared costs. This one doesn't — it's a question of whether you can do this at all.

Once your agent executes model-generated shell commands, running on your own machine stops being an option. Modal's own tutorial post is blunt about it, demonstrating two prompts that would wreck a dev machine: one running rm -rf /, one POSTing the keys in .env to an external site. What this layer buys isn't speed — it's that the model sees a sandbox rather than your filesystem.

Which is why sandboxes suddenly became a market with seven vendors in it. The OpenAI Agents SDK sandbox client docs list the natively supported hosted backends: Blaxel, Cloudflare, Daytona, E2B, Modal, Runloop, Vercel. What sets Modal apart in that document is that it gets its own section on resource sizing (cpu, memory), and ModalSandboxClientOptions in the example source does carry a gpu field accepting values like "A100" or "H100:8". Modal describes attaching GPUs as "a capability unique to Modal" — that's the vendor's claim; I did not read through the other six clients' source to confirm none of them support it.

On price, this line is not cheap. Modal charges three times the standard Function rate for sandbox CPU and memory, while GPUs cost the same as anywhere else on the platform. Run a 2 vCPU, 4 GiB sandbox for an hour and compare with the same shape on E2B: Modal $0.24, E2B $0.17. Modal costs 44% more, and what that buys is an H100 behind the same API. If your agent only runs tests and lint, that 44% is pure waste.

Competition in this layer is still pushing prices down. Cloudflare made Sandboxes GA on 2026-04-13 and switched to charging only for actively used CPU cycles — the stated reasoning is precise: agents spend most of their time waiting on an LLM, and that time shouldn't be billed. Their standard plan includes 15,000 concurrent lite instances.

7. The honest limits

Lock-in is real. @app.function(gpu="H100") is pleasant to write, but it isn't a standard. Your autoscaling, snapshots, Volumes, and Secrets all grow on Modal's abstractions, and switching providers means rewriting the deployment layer. By contrast, the vLLM route delivers an OpenAI-compatible HTTP server that runs anywhere.

There is no self-hosted version. The client and CLI are open source; the platform is not. For compliance you rely on SOC 2 Type 2 and an Enterprise-plan BAA, not on "it runs in my own datacenter." And the HIPAA BAA has explicit carve-outs: Volumes v1, Images, Memory Snapshots, and user code are all out of scope.

The concurrency ceiling is lower than the marketing. The official docs state a hard limit of 4,000 concurrent containers for a single Function; at the workspace level, Starter allows 100 containers and 10 concurrent GPUs, Team allows 5,000 containers and 50 GPUs, and anything beyond that is an Enterprise conversation. "Millions of sandboxes in parallel" is a future plan in the Series C announcement, not a quota you get on signup. Those two numbers are three orders of magnitude apart; keep them separate when reading.

Snapshots expire. Filesystem and Directory Snapshots default to 30 days; Memory Snapshots are fixed at 7 days and currently cannot be extended. Before v1.5 (Python), Filesystem Snapshots persisted indefinitely — a breaking change that will bite anyone treating snapshots as a long-lived environment library. Pass ttl=None explicitly if that's you.

Overall

Modal's core trade-off is: a proprietary abstraction and a 0–50% price premium in exchange for an entire GPU infrastructure team's work.

The cases where it fits are easy to recognize: spiky traffic, GPU utilization persistently below 50%, nobody on the team who wants to maintain an autoscaler, or you're building agents and need GPU sandboxes — that last one currently has few substitutes.

The cases where it doesn't fit are equally easy: GPUs saturated nearly around the clock (go get a long-term contract from RunPod or Lambda), sandboxes that only run tests and lint (E2B and Cloudflare Sandboxes are cheaper), or compliance that forbids code leaving your own datacenter (which rules Modal out outright).

One last note for anyone doing the selection: the price tables here will probably be wrong in three months. What's worth keeping isn't the numbers, it's the formula — your utilization times Modal's price, against the competitor's full-month price. That formula doesn't expire.

References

Related

CrewAI: Organizing Multi-Agent Collaboration Through Role-Playing 1 min LlamaIndex Is Not a RAG Framework Anymore, and Old Tutorials Won't Tell You 1 min Composio: Who Holds Every User's Token When Your Agent Connects a Hundred SaaS Apps 1 min