Modal vs Baseten in 2026: Serverless GPU, vLLM & H100 TCO
Engineering and 36-month TCO teardown of Modal vs Baseten, Replicate (Cloudflare), and Together AI in 2026: per-second Python H100/B200 containers and GPU memory snapshots vs Truss TensorRT-LLM, KV-cache prefix routing, and 3,200 H100-hr/mo math.

On this page

Modal vs Baseten is the choice most teams reach once open-weight large language models, multimodal pipelines or code-executing AI agents outgrow prototype APIs. Both are serverless GPU vendors, and both raised large rounds in 2026, but they sell different things. Modal sells per-second Python compute. Baseten sells dedicated, tuned inference. Searches for Baseten vs Modal usually pull in Replicate, Together AI and self-hosted vLLM too, so this guide covers all five.
Neighbouring decisions live elsewhere: Portkey vs LiteLLM covers the API proxy and token-budget routing, Langfuse vs LangSmith covers trace observability, and the LLM development guide and RAG and LLM systems guide cover model and retrieval design. This article stays on the layer where H100 and B200 seconds decide your gross margin: GPU compute, cold starts and the model-serving data plane.
At Idealogic, our AI software development, generative AI development, AI agent development, and dedicated development team engineers build custom inference planes for fintech, healthtech, and aviation and aerospace clients. Three things follow: the vendor comparison, a reference blueprint (prefix-aware router, vLLM FP8 with Run:ai S3 streaming, Karpenter GPU burst), and a worked 36-month TCO model at 3,200 H100 GPU-hours per month. The model rests on stated assumptions, so plug in your own quotes.
The short version
- Modal wins on Python velocity, Sandboxes and per-second economics. Modal pricing is billed per second: an H100 lists at
$0.001097/sec(about$3.95/hr) and a B200 at about$6.25/hr, with no idle charge. You write plain Python decorators and get memory snapshots and Modal Sandboxes. - Baseten wins on managed
TensorRT-LLMtuning and self-hosted deployment. Baseten pricing is listed per minute: H100$0.10833/min(about$6.50/hr) and B200 about$9.98/hr. It pairs open-sourceTrusswithTensorRT-LLMengines, runs on multiple clouds including AWS and GCP, and can deploy inside your own VPC. Our model assumes a committed rate of$5.20 to $5.80/hr, which isn't a list price. - Replicate and Together AI sit in the middle. After Cloudflare's Replicate acquisition, announced in November 2025, Replicate pricing lists an H100 at
$5.49/hrwith per-second billing andCogpackaging. Together AI pricing for a dedicated H100 endpoint is also$5.49/hron demand. Neither gives you Modal's Python runtime or Baseten's self-hosted option. - At 3,200 H100-hours a month (3 warm H100s around the clock plus 1,010 burst hours), the owned cluster is cheapest in our model. Owned
vLLMon EKS with reserved neocloud H100s comes to$363,600over 36 months with CEE labor. That's$314,400less than Baseten Dedicated Enterprise ($678,000),$199,200less than Replicate or Together ($562,800), and$159,300less than Modal Enterprise ($522,900), with payback in 3.42 to 5.94 months. Under 400 H100-hours a month Modal wins, at about$1,800/moor less.
Quick answer: if your GPU load is spiky, Python-heavy or needs code sandboxes, start on Modal. If you serve one latency-critical model all day and want managed TensorRT-LLM engines or a deployment in your own VPC, Baseten fits better. Honestly, most teams should start on Modal and revisit once three or more GPUs never switch off.
2026 serverless GPU internals: Modal vs Baseten vs Replicate & Together AI
Modal is a per-second serverless GPU cloud for arbitrary Python. Baseten is a dedicated inference service, priced per minute and built around TensorRT-LLM. The Modal vs Baseten comparison (or Baseten vs Modal, if you search it that way) comes down to four splits: billing granularity, packaging, cold starts, and where the data plane runs.
1. Modal pricing and architecture: per-second Python cloud, GPU memory snapshots and Sandboxes
Modal closed a $750M Series D led by Accel in October 2026 at a $15.75B post-money valuation, after a reported $87M Series B in September 2025 and a Series C earlier in 2026. In the Modal documentation, infrastructure is Python code.
- Python decorators instead of YAML. No Dockerfiles, Helm charts or Terraform: you annotate a class with
@app.cls(gpu="H100", scaledown_window=60)(Modal GPU compute) and Modal's own container runtime does the rest. Modal pricing is per second: H100$0.001097/sec(about$3.95/hr), H200 about$4.54/hr, B200 about$6.25/hr, A100 80GB about$2.50/hr, L40S about$1.95/hr, plus about$0.047per core-hour of CPU and$0.008per GiB-hour of RAM. Plans are Starter ($30/mofree credits), Team at$250/mo($100/mocredits included), and Enterprise with custom pricing. - Memory snapshots. A large LLM container spends most of its cold start importing PyTorch and moving weights into VRAM. Modal memory snapshots capture the state after your
@modal.enter(snap=True)hook.enable_memory_snapshot=Truecovers CPU memory. GPU memory also needsexperimental_options={"enable_gpu_snapshot": True}, which Modal still labels alpha. Modal's docs say init-heavy functions often start 3 to 10 times faster from a snapshot, so measure your own model. - Sandboxes and Volumes. Modal Sandboxes run untrusted code in isolated containers, on a gVisor runtime by default or in a VM with its own kernel. That's what generative AI coding agents need, alongside Modal Volumes for shared storage.
- Where Modal trades off. You bring and tune your own inference engine (
vLLM,SGLangorTensorRT-LLM) in Python. Modal also runs on Modal-managed capacity only. We found no self-hosted or customer-VPC data plane in its docs.
2. Baseten pricing and architecture: Truss, TensorRT-LLM Engine Builder and self-hosted deployment
Baseten raised a $1.5B Series F in June 2026 at a $13B valuation, after a $150M Series D led by BOND in September 2025. The Baseten documentation centers on low Time to First Token (TTFT) and high tokens per second per GPU.
Trussplus theTensorRT-LLMEngine Builder. You package a model with open-sourceTruss, and Baseten's Engine Builder compiles aTensorRT-LLMengine for the target GPU. A tuned engine can raise throughput over an untuned PyTorch baseline. How much depends on the model, batch size and sequence length.- Multiple clouds and faster weight loading. Baseten runs capacity on multiple clouds, including AWS and GCP, and its Network Accelerator speeds up model weight downloads on cold start.
- Per-minute pricing and self-hosting. Baseten pricing for dedicated replicas is listed per minute: H100
$0.10833/min(about$6.50/hr), H100 MIG 40GB$0.0625/min($3.75/hr), B200$0.16633/min(about$9.98/hr) and L4 about$0.85/hr. Committed enterprise rates are negotiated, and the$5.20 to $5.80/hrwe use later is our assumption. Baseten offers a HIPAA Security Rule BAA, SOC 2 Type II, and a self-hosted deployment that puts its serving plane inside your AWS or GCP VPC. - Where Baseten trades off. The hourly GPU rate is higher (
$6.50/hrlist against Modal's$3.95/hr). Per-minute pricing and scale-down delays penalize short bursts. And there's no general-purpose sandbox for untrusted code.
3. Replicate (Cloudflare) and Together AI pricing for dedicated endpoints
If you just need standard Llama, Qwen, DeepSeek or Flux weights behind an endpoint, the shortlist is Replicate and Together AI. Cloudflare announced the Replicate acquisition in November 2025. Replicate packages models with Cog, bills per second and lists an H100 at $5.49/hr. Together AI pricing for dedicated H100 endpoints is $5.49 per GPU-hour on demand, and reserved capacity is quoted by sales. You give up Modal's Python compute and Baseten's self-hosted deployment.
Production blueprint: prefix-aware router, vLLM FP8 and Karpenter burst
Moving sustained 70B FP8 inference off a managed GPU cloud is the usual way to cut vendor lock-in. With naive Kubernetes manifests it fails two ways. Cold starts take minutes, because 70 GB of Safetensors gets pulled over sequential HTTP. And the KV-cache hit rate collapses, because a ClusterIP service round-robins multi-turn agent requests across pods.
Our reference setup fixes both. It was written against vLLM v0.33.0, so check the flags against your version. These are the design decisions that matter:
The loading targets are planning figures. Confirm them with NVIDIA DCGM and pod startup metrics on your own hardware.
The router is the piece most teams skip, and it's where the money goes. It can run standalone in TypeScript or behind an Envoy ext_proc filter. The routing decision is bounded-load prefix hashing: hash the tenant and its stable prefix onto the ring, stay on the sticky replica when it has capacity, probe ring neighbours when it is full or unhealthy, and spill to a per-second Modal or Baseten endpoint only when every replica is saturated.
Pinning the tenant and its stable prefix to one replica means the pod doesn't recompute an 8,192-token prompt on every agent tool-call loop. In our worked model that takes p95 Time to First Token from 1,150 ms to 190 ms, though that figure is illustrative. The serverless fallback catches a sudden 5x traffic spike on per-second Modal containers while Karpenter and the Run:ai streamer bring a new H100 replica online.
Modal vs Baseten cost: worked 36-month TCO at 3,200 H100 GPU-hrs/mo
At 3,200 H100-hours a month, an owned vLLM cluster is the cheapest option in this model: $363,600 over 36 months, against $522,900 on Modal and $678,000 on Baseten.
The workload is a production AI service over 36 months using 3,200 H100 80GB GPU-hours a month. That's 3 baseline H100s kept warm around the clock (2,190 hours a month) plus 1,010 hours of daytime and agent burst autoscaling. It serves a 70B FP8 LLM with a vision and audio pipeline under SOC 2 Type II, HIPAA, and PCI DSS v4.0.1 controls.
Labor is priced at Idealogic's $75/hr senior CEE rate against a $165/hr US agency rate (2.2x). Those two rates are ours. For labor-cost context, see Eurostat's EU hourly labour cost dataset (lc_lci_lev) and the O*NET profile for Software Developers (15-1252.00). Infrastructure adds Amazon EKS ($0.10/hr) and reserved H100 capacity rented from a neocloud.
- Option A: Baseten Dedicated Enterprise (
Truss+TensorRT-LLM):- Setup (
160h@$75/hrCEE =$12,000;$165/hrUS =$26,400):Trusspackaging, FP8TensorRT-LLMengine builds, and HIPAA/SOC 2 SSO wiring. - 36-mo Baseten GPU Cloud (
$17,600/mo x 36 = $633,600):3,200 H100-hrs/moat an assumed committed enterprise rate of$5.50/hr(list is$6.50/hron demand), including the per-minute scale-down buffer and enterprise support. - 36-mo MLOps (
12h/mo x 36 = 432h=$32,400CEE /$71,280US) ->$678,000CEE ($731,280US). Year-2+ run-rate:$222,000/yrCEE ($18,500/mo).
- Setup (
- Option B: Replicate (
Cloudflare) / Together AI Dedicated Endpoints:- Setup (
160h=$12,000CEE /$26,400US). - 36-mo Dedicated GPU Cloud (
$14,400/mo x 36 = $518,400):3,200 H100-hrs/mo@$4.50/hr, an assumed committed contract rate (both vendors list$5.49/hron demand). - 36-mo MLOps (
12h/mo = 432h=$32,400CEE /$71,280US) ->$562,800CEE ($616,080US). Year-2+ run-rate:$183,600/yrCEE ($15,300/mo).
- Setup (
- Option C: Modal Enterprise (Per-Second Python Containers + Memory Snapshots):
- Setup (
180h@$75/hrCEE =$13,500;$165/hrUS =$29,700): PythonvLLM/SGLang@app.clswrappers, memory snapshot checkpoints (enable_memory_snapshot=Trueplus the alpha GPU snapshot option), and Volume caching. - 36-mo Modal Compute (
$13,100/mo x 36 = $471,600):3,200 H100-hrs x $3.95/hr = $12,640/mo, plus about$460/mofor host CPU and RAM, Volume I/O and the plan fee (Team is$250/mo, Enterprise is custom). Per-second billing means no idle cooldown cost. - 36-mo MLOps (
14h/mo x 36 = 504h=$37,800CEE /$83,160US) ->$522,900CEE ($584,460US). Year-2+ run-rate:$169,800/yrCEE ($14,150/mo).
- Setup (
- Option D: Owned
vLLM/SGLangKubernetes Cluster (EKScontrol plane, reserved neocloudH100+Karpenterburst +Run:aiS3 streamer):- Turnkey Build (
600h@$75/hrCEE =$45,000;$165/hrUS =$99,000). - 36-mo GPU & Cloud Infra (
$7,500/mo x 36 = $270,000):3reservedH100 80GBon 1-year terms (2,190h x $2.15/hr = $4,708.50/mo) +1,010hof on-demand or spotH100burst (1,010h x $2.35/hr = $2,373.50/mo) + EKS control plane, S3 VPC Gateway & Prometheus/DCGM telemetry ($418.00/mo). - 36-mo MLOps / SRE (
18h/mo x 36 = 648h=$48,600CEE /$106,920US) ->$363,600CEE ($475,920US). Year-2+ run-rate:$106,200/yrCEE ($8,850/mo).
- Turnkey Build (
Assumptions and sources. Modal and Baseten list prices come from their pricing pages. The $5.50/hr Baseten rate and the $4.50/hr Replicate or Together rate are our assumptions for negotiated contracts. So are the $2.15/hr reserved and $2.35/hr burst H100 rates, which reflect neocloud pricing, not an AWS p5 per-GPU price. A 1-year reservation also renews twice inside 36 months at prices nobody can promise today. Change any of these and the ranking can move.
Sensitivity check: sub-400 H100-hr/mo Modal floor and sub-6-month payback
- Under 400 H100-hours a month, or below a 15% duty cycle, pick Modal. Async batch embeddings, nightly evals and bursty Sandboxes at that volume cost about
$1,800/moor less on Modal ($250/moTeam base plus per-second GPU time), and you run no cluster. A dedicated setup can't compete there. - At 3,200 H100-hours a month, the owned cluster pays back inside 6 months at both labor rates. At
$75/hrCEE, Option D's recurring cost is$8,850/mo($7,500GPU infra +$1,350MLOps). The extra build cost pays back in 3.42 months against Baseten ($33,000 / $9,650/modelta, saving$314,400), 5.12 months against Replicate or Together ($33,000 / $6,450/mo, saving$199,200), and 5.94 months against Modal ($31,500 / $5,300/mo, saving$159,300). At$165/hrUS agency labor, Option D ($475,920) still beats Modal ($584,460) by$108,540and Baseten ($731,280) by$255,360. Keep Modal wired in as the burst target either way, since it costs nothing while idle.
2026 GPU inference & vLLM migration build costs (Senior CEE vs US)
Our AI software development, generative AI development, and dedicated development team squads build production GPU infrastructure at $75/hr senior CEE against a $165/hr US agency rate (2.2x):
Inside the worked 600-hour ($45,000 CEE vs $99,000 US) owned vLLM cluster
- 1. vLLM / SGLang Engine (25% / 150h /
$11,250CEE vs$24,750US): FP8 weight and KV-cache quantization, attention backend and speculative decoding tuning, and multi-LoRA adapter hot-swapping. - 2. Karpenter GPU Burst (25% / 150h /
$11,250CEE vs$24,750US): Reserved H100 baseline plus multi-AZ spot and on-demand Karpenter NodePools, the NVIDIA GPU Operator, and DCGM XID health eviction. - 3. Run:ai S3 Streamer (20% / 120h /
$9,000CEE vs$19,800US): S3 VPC Gateway Endpoints and a 32-threadrunai_streamerconfig, with a target of taking 70B pod cold boots from about 4 minutes to under 14 seconds. - 4. KV-Cache Prefix Route (20% / 120h /
$9,000CEE vs$19,800US): Bounded-load SHA-256 prefix-sticky routing (target APC hit rate above 78%) with Modal serverless burst spillover. - 5. Benchmark & Cutover (10% / 60h /
$4,500CEE vs$9,900US):guidellmandk6streaming TTFT and ITL load tests, golden-dataset FP8 accuracy checks, and a zero-downtime canary cutover.
Modal is the best developer experience ever built for spiky Python GPUs and AI Sandboxes, and Baseten squeezes impressive TensorRT-LLM throughput when you need turnkey multi-cloud serving. Yet once three or more H100s stay warm around the clock, paying $4.00 to $6.50 per GPU-hour is pure margin leakage. Run baseline traffic on reserved H100s with KV-sticky vLLM routing, and keep Modal purely as your zero-idle-cost overflow valve.
Modal vs Baseten vs Replicate: 5-step decision checklist and Idealogic proof
Idealogic ships low-latency, compliance-governed AI workloads in regulated industries:
- Traceable aviation & industrial AI (AirPart Chain & AeroParts): High-integrity document extraction and compliance verification across aviation and aerospace.
- Regulated clinical, fintech & B2B AI (AddMed, Fese & Pressmerit): HIPAA-safe clinical pipelines, PCI DSS v4.0.1 financial controls, and multi-tenant semantic inference across 26 industries.
Five questions settle Modal vs Baseten (or Baseten vs Modal), Replicate, Together and an owned vLLM cluster in 2026:
- Under 400 H100-hours a month, spiky Python jobs, or untrusted AI code execution? Modal. You get a
$3.95/hrH100 billed per second, memory snapshots and Sandboxes. - Very low
TTFTvoice or realtime inference, with managedTensorRT-LLMor a deployment in your own VPC, and no Kubernetes team? Baseten Dedicated or its self-hosted option. - A standard open-weight endpoint, ideally next to Cloudflare Workers? Replicate (
Cog) or Together AI Dedicated Endpoints. Budget on the$5.49/hrlist price until you have a contract. - Past 1,500 H100-hours a month? Go hybrid. Serve the baseline 80% on owned reserved H100s running
vLLM, and spill the bursty 20% onto per-second Modal. - Sustaining 2,000 to 10,000+ H100-hours a month under HIPAA or SOC 2? Run your own
vLLMorSGLangcluster withKarpenter, theRun:aiS3 streamer and the KV-prefix router. In our model that's$363.6kover 36 months with CEE labor,$159.3k to $314.4kless than the managed options. Fair warning: someone on your team has to own GPU drivers and node upgrades. Contact our team for a 48-hour GPU TCO audit.

CTO and Co-founder at Idealogic. Directing engineering, systems architecture, security, and full-stack delivery across web, mobile, and blockchain.
Frequently asked questions
Pick Modal if your team ships arbitrary Python pipelines, multimodal batch jobs, snapshot-backed APIs or isolated agent code sandboxes, and wants per-second H100, H200 and B200 billing with no idle charge. Pick Baseten if the main workload is latency-critical synchronous LLM, voice or diffusion inference that needs managed TensorRT-LLM engines, capacity across multiple clouds including AWS and GCP, or a self-hosted deployment inside your own cloud account. The Baseten vs Modal choice follows workload shape more than vendor quality.
Both raised large late-stage rounds. Modal's reported $87 million Series B came in September 2025, a Series C followed in 2026, and in October 2026 it closed a $750 million Series D led by Accel at a $15.75 billion post-money valuation. Baseten announced a $150 million Series D led by BOND in September 2025 and a $1.5 billion Series F in June 2026 at a $13 billion valuation, which it described as its fourth fundraise in 18 months.
Per-second billing stops the meter when a container exits. Per-minute billing with scale-down delays doesn't. Modal lists an H100 at $0.001097 per second, about $3.95 per hour, and charges nothing once the container is gone. Baseten Dedicated lists an H100 at $0.10833 per minute, about $6.50 per hour, and a replica stays billable through the scale-down delay after a traffic spike. Our model assumes a committed Baseten rate of $5.20 to $5.80 per hour, which is our assumption and isn't a list price. On bursty daytime traffic that idle tail adds up.
They attack different parts of the cold start. Modal checkpoints the container: enable_memory_snapshot saves CPU memory after the modal.enter(snap=True) hook, and an alpha option, enable_gpu_snapshot, extends that to GPU memory. Modal's docs say init-heavy functions often start 3 to 10 times faster from a snapshot. Baseten speeds up the weight download instead, pairing open-source Truss packaging with its Network Accelerator before TensorRT-LLM engines load. Neither vendor publishes one cold-start figure that holds for every model, so benchmark your own.
They're the simpler option for standard open-weight models. Cloudflare announced its Replicate acquisition in November 2025. Replicate pairs Cog packaging with per-second billing and lists an H100 at $5.49 per hour. Together AI pricing for dedicated H100 endpoints is also $5.49 per GPU-hour on demand, with reserved capacity quoted by sales. Both fit when you don't need Modal's Python jobs and Sandboxes or Baseten's self-hosted TensorRT-LLM tuning.
Because it throws away the prefix cache. Agentic and RAG applications resend thousands of tokens of shared system prompt, tool schemas and document context on every turn. Round-robin scatters consecutive turns from one workspace across vLLM pods, so each GPU recomputes the prefill from scratch. Bounded-load consistent hashing on a SHA-256 digest of the prefix pins a recurring prefix to one replica. Our blueprint targets a prefix cache hit rate above 78 percent that way, which cuts Time to First Token.
Once roughly three H100s stay warm around the clock. Below 400 H100 GPU-hours per month, or on spiky batch and sandbox workloads under a 15 percent duty cycle, Modal costs about $1,800 per month or less with no Kubernetes to run. In our worked model at 3,200 H100 GPU-hours per month under HIPAA and SOC 2 rules, an owned cluster on reserved neocloud H100s running vLLM, Karpenter and Run:ai S3 streaming totals $363,600 over 36 months with CEE engineering. That's $159,300 less than Modal and $314,400 less than Baseten. The GPU rates in that model are assumptions, so rerun it with your own quotes.
At Idealogic's senior CEE rate of $75 per hour, a 600-hour owned vLLM and SGLang Kubernetes build costs $45,000, versus $99,000 at a $165 per hour US agency rate. That covers Karpenter GPU bursting, Run:ai S3 streaming and KV-cache prefix routing, and in our worked model it pays back in 3.42 to 5.94 months. Smaller engagements cost less: a two to three week GPU serving, quantization and FinOps audit runs $11,250 to $18,000, and a Modal or Baseten enterprise production build runs $13,500 to $27,000.
More from the journal

SigNoz vs Datadog in 2026: OTel, ClickHouse & APM TCO
Engineering and 36-month TCO teardown of SigNoz vs Datadog and Grafana Cloud in 2026: OpenTelemetry Collector tail-sampling, unified ClickHouse telemetry, p99 host high-water-mark billing, PG17 FORCE RLS alert planes, and 150-host math.

Dagster vs Prefect in 2026: Assets, Dagster Credits & EKS TCO
Dagster vs Prefect compared for 2026, with AWS MWAA as the baseline: Software-Defined Assets vs @flow and @task, how Dagster Credits meter Hybrid runs on your own EKS, Prefect Cloud seat pricing, Iceberg S3 I/O, PG17 event partitioning, and a 36-month TCO model.

Deepgram vs AssemblyAI in 2026: Latency, VAD & ASR TCO
Deepgram vs AssemblyAI and OpenAI Realtime ASR in 2026, compared on engineering and 36-month TCO: Flux native End-of-Turn vs Universal-3.6 Pro Realtime, Nova-3 keyterm prompting, PII audio redaction, silence billing on streaming, Silero VAD, and L4 GPU math.