Skip to content

Modal vs Baseten in 2026: Serverless GPU, vLLM & H100 TCO

Engineering and 36-month TCO teardown of Modal vs Baseten, Replicate (Cloudflare), and Together AI in 2026: per-second Python H100/B200 containers and GPU memory snapshots vs Truss TensorRT-LLM, KV-cache prefix routing, and 3,200 H100-hr/mo math.

Artem Zaitsev
Artem ZaitsevCTO · Idealogic
Author: Artem ZaitsevPublished: Updated: Reading time: 20 minHow we write →
On this page
Idealogic 2026 engineering comparison of Modal vs Baseten, Replicate, and Together AI covering per-second serverless GPU containers, TensorRT-LLM dedicated endpoints, cold-start snapshots, and 36-month H100 TCO

Modal vs Baseten is the choice most teams reach once open-weight large language models, multimodal pipelines or code-executing AI agents outgrow prototype APIs. Both are serverless GPU vendors, and both raised large rounds in 2026, but they sell different things. Modal sells per-second Python compute. Baseten sells dedicated, tuned inference. Searches for Baseten vs Modal usually pull in Replicate, Together AI and self-hosted vLLM too, so this guide covers all five.

Neighbouring decisions live elsewhere: Portkey vs LiteLLM covers the API proxy and token-budget routing, Langfuse vs LangSmith covers trace observability, and the LLM development guide and RAG and LLM systems guide cover model and retrieval design. This article stays on the layer where H100 and B200 seconds decide your gross margin: GPU compute, cold starts and the model-serving data plane.

At Idealogic, our AI software development, generative AI development, AI agent development, and dedicated development team engineers build custom inference planes for fintech, healthtech, and aviation and aerospace clients. Three things follow: the vendor comparison, a reference blueprint (prefix-aware router, vLLM FP8 with Run:ai S3 streaming, Karpenter GPU burst), and a worked 36-month TCO model at 3,200 H100 GPU-hours per month. The model rests on stated assumptions, so plug in your own quotes.

The short version

  • Modal wins on Python velocity, Sandboxes and per-second economics. Modal pricing is billed per second: an H100 lists at $0.001097/sec (about $3.95/hr) and a B200 at about $6.25/hr, with no idle charge. You write plain Python decorators and get memory snapshots and Modal Sandboxes.
  • Baseten wins on managed TensorRT-LLM tuning and self-hosted deployment. Baseten pricing is listed per minute: H100 $0.10833/min (about $6.50/hr) and B200 about $9.98/hr. It pairs open-source Truss with TensorRT-LLM engines, runs on multiple clouds including AWS and GCP, and can deploy inside your own VPC. Our model assumes a committed rate of $5.20 to $5.80/hr, which isn't a list price.
  • Replicate and Together AI sit in the middle. After Cloudflare's Replicate acquisition, announced in November 2025, Replicate pricing lists an H100 at $5.49/hr with per-second billing and Cog packaging. Together AI pricing for a dedicated H100 endpoint is also $5.49/hr on demand. Neither gives you Modal's Python runtime or Baseten's self-hosted option.
  • At 3,200 H100-hours a month (3 warm H100s around the clock plus 1,010 burst hours), the owned cluster is cheapest in our model. Owned vLLM on EKS with reserved neocloud H100s comes to $363,600 over 36 months with CEE labor. That's $314,400 less than Baseten Dedicated Enterprise ($678,000), $199,200 less than Replicate or Together ($562,800), and $159,300 less than Modal Enterprise ($522,900), with payback in 3.42 to 5.94 months. Under 400 H100-hours a month Modal wins, at about $1,800/mo or less.

Quick answer: if your GPU load is spiky, Python-heavy or needs code sandboxes, start on Modal. If you serve one latency-critical model all day and want managed TensorRT-LLM engines or a deployment in your own VPC, Baseten fits better. Honestly, most teams should start on Modal and revisit once three or more GPUs never switch off.

2026 serverless GPU internals: Modal vs Baseten vs Replicate & Together AI

Modal is a per-second serverless GPU cloud for arbitrary Python. Baseten is a dedicated inference service, priced per minute and built around TensorRT-LLM. The Modal vs Baseten comparison (or Baseten vs Modal, if you search it that way) comes down to four splits: billing granularity, packaging, cold starts, and where the data plane runs.

Four billing, packaging, cold-start and deployment differences between Modal and Baseten for 2026 production AI workloads

1. Modal pricing and architecture: per-second Python cloud, GPU memory snapshots and Sandboxes

Modal closed a $750M Series D led by Accel in October 2026 at a $15.75B post-money valuation, after a reported $87M Series B in September 2025 and a Series C earlier in 2026. In the Modal documentation, infrastructure is Python code.

  1. Python decorators instead of YAML. No Dockerfiles, Helm charts or Terraform: you annotate a class with @app.cls(gpu="H100", scaledown_window=60) (Modal GPU compute) and Modal's own container runtime does the rest. Modal pricing is per second: H100 $0.001097/sec (about $3.95/hr), H200 about $4.54/hr, B200 about $6.25/hr, A100 80GB about $2.50/hr, L40S about $1.95/hr, plus about $0.047 per core-hour of CPU and $0.008 per GiB-hour of RAM. Plans are Starter ($30/mo free credits), Team at $250/mo ($100/mo credits included), and Enterprise with custom pricing.
  2. Memory snapshots. A large LLM container spends most of its cold start importing PyTorch and moving weights into VRAM. Modal memory snapshots capture the state after your @modal.enter(snap=True) hook. enable_memory_snapshot=True covers CPU memory. GPU memory also needs experimental_options={"enable_gpu_snapshot": True}, which Modal still labels alpha. Modal's docs say init-heavy functions often start 3 to 10 times faster from a snapshot, so measure your own model.
  3. Sandboxes and Volumes. Modal Sandboxes run untrusted code in isolated containers, on a gVisor runtime by default or in a VM with its own kernel. That's what generative AI coding agents need, alongside Modal Volumes for shared storage.
  4. Where Modal trades off. You bring and tune your own inference engine (vLLM, SGLang or TensorRT-LLM) in Python. Modal also runs on Modal-managed capacity only. We found no self-hosted or customer-VPC data plane in its docs.

2. Baseten pricing and architecture: Truss, TensorRT-LLM Engine Builder and self-hosted deployment

Baseten raised a $1.5B Series F in June 2026 at a $13B valuation, after a $150M Series D led by BOND in September 2025. The Baseten documentation centers on low Time to First Token (TTFT) and high tokens per second per GPU.

  1. Truss plus the TensorRT-LLM Engine Builder. You package a model with open-source Truss, and Baseten's Engine Builder compiles a TensorRT-LLM engine for the target GPU. A tuned engine can raise throughput over an untuned PyTorch baseline. How much depends on the model, batch size and sequence length.
  2. Multiple clouds and faster weight loading. Baseten runs capacity on multiple clouds, including AWS and GCP, and its Network Accelerator speeds up model weight downloads on cold start.
  3. Per-minute pricing and self-hosting. Baseten pricing for dedicated replicas is listed per minute: H100 $0.10833/min (about $6.50/hr), H100 MIG 40GB $0.0625/min ($3.75/hr), B200 $0.16633/min (about $9.98/hr) and L4 about $0.85/hr. Committed enterprise rates are negotiated, and the $5.20 to $5.80/hr we use later is our assumption. Baseten offers a HIPAA Security Rule BAA, SOC 2 Type II, and a self-hosted deployment that puts its serving plane inside your AWS or GCP VPC.
  4. Where Baseten trades off. The hourly GPU rate is higher ($6.50/hr list against Modal's $3.95/hr). Per-minute pricing and scale-down delays penalize short bursts. And there's no general-purpose sandbox for untrusted code.

3. Replicate (Cloudflare) and Together AI pricing for dedicated endpoints

If you just need standard Llama, Qwen, DeepSeek or Flux weights behind an endpoint, the shortlist is Replicate and Together AI. Cloudflare announced the Replicate acquisition in November 2025. Replicate packages models with Cog, bills per second and lists an H100 at $5.49/hr. Together AI pricing for dedicated H100 endpoints is $5.49 per GPU-hour on demand, and reserved capacity is quoted by sales. You give up Modal's Python compute and Baseten's self-hosted deployment.

Architectural dimensionModal (Python Serverless)Baseten (Dedicated / self-hosted)Replicate (Cloudflare) / TogetherOwned vLLM / SGLang (EKS)
2026 scale & backing$750M Series D ($15.75B, Oct 2026)$1.5B Series F ($13B, Jun 2026)Cloudflare M&A / Together CloudApache 2.0 OSS (0 vendor markup)
Billing unit & H100 80GBPer-second (~$3.95/hr, $0 idle)Per-minute (~$6.50 list, $5.50 assumed committed)Replicate per-second, Together on demand ($5.49/hr list each)~$2.15 reserved / $2.35 burst (assumed neocloud rates)
Blackwell B200 rate~$6.25/hr (per-second)~$9.98/hr ($0.16633/min)Custom enterprise quoteDirect wholesale / AWS P5/neocloud
Packaging & kernelsPure Python (vLLM / SGLang)Truss + TensorRT-LLM BuilderCog YAML / turnkey model catalogUpstream vLLM / SGLang / TRT-LLM
Cold-start accelerationCPU memory snapshots, GPU snapshots in alphaNetwork Accelerator for weightsCached Cog weightsRun:ai S3 tensor stream (our target: under 14s)
AI code SandboxesFirst-class Sandboxes (gVisor or VM)No (inference-only runtime)No (model containers only)Kata / gVisor K8s pods
Customer-VPC deploymentNone documented (Modal-managed cloud)Yes (self-hosted deployment)No (vendor cloud / Edge)100% customer VPC & KMS

Production blueprint: prefix-aware router, vLLM FP8 and Karpenter burst

Moving sustained 70B FP8 inference off a managed GPU cloud is the usual way to cut vendor lock-in. With naive Kubernetes manifests it fails two ways. Cold starts take minutes, because 70 GB of Safetensors gets pulled over sequential HTTP. And the KV-cache hit rate collapses, because a ClusterIP service round-robins multi-turn agent requests across pods.

Our reference setup fixes both. It was written against vLLM v0.33.0, so check the flags against your version. These are the design decisions that matter:

DecisionWhat we doWhy it matters for TCO
Weight loadingvLLM loads with the runai_streamer load format, streaming Safetensors concurrently from Amazon S3 through a free VPC Gateway EndpointOur design target is a 70B load in under 14 seconds, against roughly 4 minutes over sequential HTTP
Precision and memoryFP8 weights, FP8 KV cache, prefix caching on, a 16,384-token context capA 70B model at FP8 fills most of an 80 GB card, so lower the context cap if you hit out-of-memory errors
Burst capacityA Karpenter GPU NodePool takes spot or on-demand H100 nodes and drops empty ones after 45 secondsYou pay for burst nodes only while traffic needs them
Node shapeA p5.48xlarge carries eight H100s, so either pack eight serving pods per burst node or rent single-GPU neocloud nodesOur TCO model assumes the neocloud burst rate
Request routingA router hashes the first 1,024 characters of the system, tool and RAG prefix with SHA-256 onto a ring of 64 virtual nodes per replicaRepeat prefixes land on the same pod, which keeps the KV cache warm (target hit rate above 78%)
Overload handlingEach replica has an in-flight cap; when the sticky pod is full the router tries its ring neighbours, then spills to a per-second Modal or Baseten endpointA spike never queues behind a hot shard while Karpenter brings up a new H100

The loading targets are planning figures. Confirm them with NVIDIA DCGM and pod startup metrics on your own hardware.

The router is the piece most teams skip, and it's where the money goes. It can run standalone in TypeScript or behind an Envoy ext_proc filter. The routing decision is bounded-load prefix hashing: hash the tenant and its stable prefix onto the ring, stay on the sticky replica when it has capacity, probe ring neighbours when it is full or unhealthy, and spill to a per-second Modal or Baseten endpoint only when every replica is saturated.

Pinning the tenant and its stable prefix to one replica means the pod doesn't recompute an 8,192-token prompt on every agent tool-call loop. In our worked model that takes p95 Time to First Token from 1,150 ms to 190 ms, though that figure is illustrative. The serverless fallback catches a sudden 5x traffic spike on per-second Modal containers while Karpenter and the Run:ai streamer bring a new H100 replica online.

Burning $15k+/mo on Baseten, Modal, or Replicate H100 endpoints?
Share your monthly GPU hours, model weights, p95 TTFT target, and HIPAA/SOC 2 posture. Our AI infrastructure architects will model Modal vs Baseten vs Owned vLLM.
Audit your GPU inference TCO

Modal vs Baseten cost: worked 36-month TCO at 3,200 H100 GPU-hrs/mo

At 3,200 H100-hours a month, an owned vLLM cluster is the cheapest option in this model: $363,600 over 36 months, against $522,900 on Modal and $678,000 on Baseten.

The workload is a production AI service over 36 months using 3,200 H100 80GB GPU-hours a month. That's 3 baseline H100s kept warm around the clock (2,190 hours a month) plus 1,010 hours of daytime and agent burst autoscaling. It serves a 70B FP8 LLM with a vision and audio pipeline under SOC 2 Type II, HIPAA, and PCI DSS v4.0.1 controls.

Labor is priced at Idealogic's $75/hr senior CEE rate against a $165/hr US agency rate (2.2x). Those two rates are ours. For labor-cost context, see Eurostat's EU hourly labour cost dataset (lc_lci_lev) and the O*NET profile for Software Developers (15-1252.00). Infrastructure adds Amazon EKS ($0.10/hr) and reserved H100 capacity rented from a neocloud.

  1. Option A: Baseten Dedicated Enterprise (Truss + TensorRT-LLM):
    • Setup (160h @ $75/hr CEE = $12,000; $165/hr US = $26,400): Truss packaging, FP8 TensorRT-LLM engine builds, and HIPAA/SOC 2 SSO wiring.
    • 36-mo Baseten GPU Cloud ($17,600/mo x 36 = $633,600): 3,200 H100-hrs/mo at an assumed committed enterprise rate of $5.50/hr (list is $6.50/hr on demand), including the per-minute scale-down buffer and enterprise support.
    • 36-mo MLOps (12h/mo x 36 = 432h = $32,400 CEE / $71,280 US) -> $678,000 CEE ($731,280 US). Year-2+ run-rate: $222,000/yr CEE ($18,500/mo).
  2. Option B: Replicate (Cloudflare) / Together AI Dedicated Endpoints:
    • Setup (160h = $12,000 CEE / $26,400 US).
    • 36-mo Dedicated GPU Cloud ($14,400/mo x 36 = $518,400): 3,200 H100-hrs/mo @ $4.50/hr, an assumed committed contract rate (both vendors list $5.49/hr on demand).
    • 36-mo MLOps (12h/mo = 432h = $32,400 CEE / $71,280 US) -> $562,800 CEE ($616,080 US). Year-2+ run-rate: $183,600/yr CEE ($15,300/mo).
  3. Option C: Modal Enterprise (Per-Second Python Containers + Memory Snapshots):
    • Setup (180h @ $75/hr CEE = $13,500; $165/hr US = $29,700): Python vLLM/SGLang @app.cls wrappers, memory snapshot checkpoints (enable_memory_snapshot=True plus the alpha GPU snapshot option), and Volume caching.
    • 36-mo Modal Compute ($13,100/mo x 36 = $471,600): 3,200 H100-hrs x $3.95/hr = $12,640/mo, plus about $460/mo for host CPU and RAM, Volume I/O and the plan fee (Team is $250/mo, Enterprise is custom). Per-second billing means no idle cooldown cost.
    • 36-mo MLOps (14h/mo x 36 = 504h = $37,800 CEE / $83,160 US) -> $522,900 CEE ($584,460 US). Year-2+ run-rate: $169,800/yr CEE ($14,150/mo).
  4. Option D: Owned vLLM / SGLang Kubernetes Cluster (EKS control plane, reserved neocloud H100 + Karpenter burst + Run:ai S3 streamer):
    • Turnkey Build (600h @ $75/hr CEE = $45,000; $165/hr US = $99,000).
    • 36-mo GPU & Cloud Infra ($7,500/mo x 36 = $270,000): 3 reserved H100 80GB on 1-year terms (2,190h x $2.15/hr = $4,708.50/mo) + 1,010h of on-demand or spot H100 burst (1,010h x $2.35/hr = $2,373.50/mo) + EKS control plane, S3 VPC Gateway & Prometheus/DCGM telemetry ($418.00/mo).
    • 36-mo MLOps / SRE (18h/mo x 36 = 648h = $48,600 CEE / $106,920 US) -> $363,600 CEE ($475,920 US). Year-2+ run-rate: $106,200/yr CEE ($8,850/mo).
Worked 36-month GPU serving TCO ($k, CEE labor included) at 3,200 H100 80GB GPU-hrs/mo (3 warm 24/7 H100s + 1,010h burst): Owned vLLM + Karpenter saves $314.4k vs Baseten and $159.3k vs Modal
36-month line item (3,200 H100-hrs/mo)Option A: Baseten Ent.Option B: Replicate/TogetherOption C: Modal Ent.Option D: Owned vLLM H100Economic driver
1. Setup / build ($75/hr CEE)$12,000 (160h; $26.4k US)$12,000 (160h; $26.4k US)$13,500 (180h; $29.7k US)$45,000 (600h; $99.0k US)vLLM + Karpenter + S3 stream
2. 36-mo GPU SaaS / Infra$633,600 ($17,600/mo)$518,400 ($14,400/mo)$471,600 ($13,100/mo)$270,000 ($7,500/mo)$2.15 reserved H100 vs $3.95 to $5.50
3. 36-mo MLOps / SRE ($75/hr)$32,400 (12h/mo; 432h)$32,400 (12h/mo; 432h)$37,800 (14h/mo; 504h)$48,600 (18h/mo; 648h)K8s GPU node & driver GitOps
Total 36-Month TCO (CEE)$678,000$562,800$522,900$363,600 (CapEx incl.)Saves $159.3k to $314.4k
Total 36-Month TCO (US $165/hr)$731,280$616,080$584,460$475,920 (CapEx incl.)Wins US by $108.5k to $255.4k
Year-2+ Run-Rate (CEE)$222,000/yr ($18,500/mo)$183,600/yr ($15,300/mo)$169,800/yr ($14,150/mo)$106,200/yr ($8,850/mo)Saves $63.6k to $115.8k/yr

Assumptions and sources. Modal and Baseten list prices come from their pricing pages. The $5.50/hr Baseten rate and the $4.50/hr Replicate or Together rate are our assumptions for negotiated contracts. So are the $2.15/hr reserved and $2.35/hr burst H100 rates, which reflect neocloud pricing, not an AWS p5 per-GPU price. A 1-year reservation also renews twice inside 36 months at prices nobody can promise today. Change any of these and the ranking can move.

Sensitivity check: sub-400 H100-hr/mo Modal floor and sub-6-month payback

  • Under 400 H100-hours a month, or below a 15% duty cycle, pick Modal. Async batch embeddings, nightly evals and bursty Sandboxes at that volume cost about $1,800/mo or less on Modal ($250/mo Team base plus per-second GPU time), and you run no cluster. A dedicated setup can't compete there.
  • At 3,200 H100-hours a month, the owned cluster pays back inside 6 months at both labor rates. At $75/hr CEE, Option D's recurring cost is $8,850/mo ($7,500 GPU infra + $1,350 MLOps). The extra build cost pays back in 3.42 months against Baseten ($33,000 / $9,650/mo delta, saving $314,400), 5.12 months against Replicate or Together ($33,000 / $6,450/mo, saving $199,200), and 5.94 months against Modal ($31,500 / $5,300/mo, saving $159,300). At $165/hr US agency labor, Option D ($475,920) still beats Modal ($584,460) by $108,540 and Baseten ($731,280) by $255,360. Keep Modal wired in as the burst target either way, since it costs nothing while idle.

2026 GPU inference & vLLM migration build costs (Senior CEE vs US)

Our AI software development, generative AI development, and dedicated development team squads build production GPU infrastructure at $75/hr senior CEE against a $165/hr US agency rate (2.2x):

Engagement tierTimelineHoursSenior CEE ($75/hr)US Agency ($165/hr)Concrete deliverables
GPU Kernel, Quantization & TCO Audit2 to 3 wks150 to 240h$11,250 to $18,000$24,750 to $39,600FP8/AWQ eval parity, KV-cache benchmark & 36-mo TCO
Modal / Baseten Enterprise Launch3 to 5 wks180 to 360h$13,500 to $27,000$29,700 to $59,400Memory snapshots / Truss TRT-LLM, Sandboxes & HIPAA
Owned vLLM / SGLang EKS Cluster (600h)8 to 11 wks500 to 700h$37,500 to $52,500 ($45.0k)$82,500 to $115,500 ($99.0k)Reserved H100 + Karpenter burst, Run:ai S3 & KV router
Disaggregated Multi-Region B200 Mesh12 to 16 wks800 to 1,080h$60,000 to $81,000$132,000 to $178,200Prefill/decode split (nixl/RDMA), LoRA multi-tenancy & DR
Dedicated AI Infrastructure PodOngoing240h/mo (2)From $18,000/moFrom $39,600/moSenior CUDA/vLLM MLOps + Kubernetes SRE engineers

Inside the worked 600-hour ($45,000 CEE vs $99,000 US) owned vLLM cluster

Worked model: 600-hour ($45,000 at $75/hr CEE vs $99,000 at $165/hr US) owned vLLM / SGLang H100 Kubernetes cluster, Karpenter GPU burst, Run:ai S3 streamer & KV-cache prefix router
  • 1. vLLM / SGLang Engine (25% / 150h / $11,250 CEE vs $24,750 US): FP8 weight and KV-cache quantization, attention backend and speculative decoding tuning, and multi-LoRA adapter hot-swapping.
  • 2. Karpenter GPU Burst (25% / 150h / $11,250 CEE vs $24,750 US): Reserved H100 baseline plus multi-AZ spot and on-demand Karpenter NodePools, the NVIDIA GPU Operator, and DCGM XID health eviction.
  • 3. Run:ai S3 Streamer (20% / 120h / $9,000 CEE vs $19,800 US): S3 VPC Gateway Endpoints and a 32-thread runai_streamer config, with a target of taking 70B pod cold boots from about 4 minutes to under 14 seconds.
  • 4. KV-Cache Prefix Route (20% / 120h / $9,000 CEE vs $19,800 US): Bounded-load SHA-256 prefix-sticky routing (target APC hit rate above 78%) with Modal serverless burst spillover.
  • 5. Benchmark & Cutover (10% / 60h / $4,500 CEE vs $9,900 US): guidellm and k6 streaming TTFT and ITL load tests, golden-dataset FP8 accuracy checks, and a zero-downtime canary cutover.

Modal is the best developer experience ever built for spiky Python GPUs and AI Sandboxes, and Baseten squeezes impressive TensorRT-LLM throughput when you need turnkey multi-cloud serving. Yet once three or more H100s stay warm around the clock, paying $4.00 to $6.50 per GPU-hour is pure margin leakage. Run baseline traffic on reserved H100s with KV-sticky vLLM routing, and keep Modal purely as your zero-idle-cost overflow valve.

Artem Zaitsev · CTO at Idealogic

Modal vs Baseten vs Replicate: 5-step decision checklist and Idealogic proof

Idealogic ships low-latency, compliance-governed AI workloads in regulated industries:

  • Traceable aviation & industrial AI (AirPart Chain & AeroParts): High-integrity document extraction and compliance verification across aviation and aerospace.
  • Regulated clinical, fintech & B2B AI (AddMed, Fese & Pressmerit): HIPAA-safe clinical pipelines, PCI DSS v4.0.1 financial controls, and multi-tenant semantic inference across 26 industries.

Five questions settle Modal vs Baseten (or Baseten vs Modal), Replicate, Together and an owned vLLM cluster in 2026:

  1. Under 400 H100-hours a month, spiky Python jobs, or untrusted AI code execution? Modal. You get a $3.95/hr H100 billed per second, memory snapshots and Sandboxes.
  2. Very low TTFT voice or realtime inference, with managed TensorRT-LLM or a deployment in your own VPC, and no Kubernetes team? Baseten Dedicated or its self-hosted option.
  3. A standard open-weight endpoint, ideally next to Cloudflare Workers? Replicate (Cog) or Together AI Dedicated Endpoints. Budget on the $5.49/hr list price until you have a contract.
  4. Past 1,500 H100-hours a month? Go hybrid. Serve the baseline 80% on owned reserved H100s running vLLM, and spill the bursty 20% onto per-second Modal.
  5. Sustaining 2,000 to 10,000+ H100-hours a month under HIPAA or SOC 2? Run your own vLLM or SGLang cluster with Karpenter, the Run:ai S3 streamer and the KV-prefix router. In our model that's $363.6k over 36 months with CEE labor, $159.3k to $314.4k less than the managed options. Fair warning: someone on your team has to own GPU drivers and node upgrades. Contact our team for a 48-hour GPU TCO audit.
Comparing Modal, Baseten, Replicate, or an owned vLLM H100/B200 cluster?
Send your model sizes, monthly GPU hours, concurrency profile, and latency SLA. Our senior CEE AI infrastructure engineers will return a 36-month TCO model in 48 hours.
Request a 48-hour GPU TCO audit

Occasional field notes on building software, no spam

Protected by Cloudflare Turnstile · Privacy · Terms

Artem Zaitsev
Artem ZaitsevCTO · Idealogic Development OÜ

CTO and Co-founder at Idealogic. Directing engineering, systems architecture, security, and full-stack delivery across web, mobile, and blockchain.

Frequently asked questions

  • Pick Modal if your team ships arbitrary Python pipelines, multimodal batch jobs, snapshot-backed APIs or isolated agent code sandboxes, and wants per-second H100, H200 and B200 billing with no idle charge. Pick Baseten if the main workload is latency-critical synchronous LLM, voice or diffusion inference that needs managed TensorRT-LLM engines, capacity across multiple clouds including AWS and GCP, or a self-hosted deployment inside your own cloud account. The Baseten vs Modal choice follows workload shape more than vendor quality.

  • Both raised large late-stage rounds. Modal's reported $87 million Series B came in September 2025, a Series C followed in 2026, and in October 2026 it closed a $750 million Series D led by Accel at a $15.75 billion post-money valuation. Baseten announced a $150 million Series D led by BOND in September 2025 and a $1.5 billion Series F in June 2026 at a $13 billion valuation, which it described as its fourth fundraise in 18 months.

  • Per-second billing stops the meter when a container exits. Per-minute billing with scale-down delays doesn't. Modal lists an H100 at $0.001097 per second, about $3.95 per hour, and charges nothing once the container is gone. Baseten Dedicated lists an H100 at $0.10833 per minute, about $6.50 per hour, and a replica stays billable through the scale-down delay after a traffic spike. Our model assumes a committed Baseten rate of $5.20 to $5.80 per hour, which is our assumption and isn't a list price. On bursty daytime traffic that idle tail adds up.

  • They attack different parts of the cold start. Modal checkpoints the container: enable_memory_snapshot saves CPU memory after the modal.enter(snap=True) hook, and an alpha option, enable_gpu_snapshot, extends that to GPU memory. Modal's docs say init-heavy functions often start 3 to 10 times faster from a snapshot. Baseten speeds up the weight download instead, pairing open-source Truss packaging with its Network Accelerator before TensorRT-LLM engines load. Neither vendor publishes one cold-start figure that holds for every model, so benchmark your own.

  • They're the simpler option for standard open-weight models. Cloudflare announced its Replicate acquisition in November 2025. Replicate pairs Cog packaging with per-second billing and lists an H100 at $5.49 per hour. Together AI pricing for dedicated H100 endpoints is also $5.49 per GPU-hour on demand, with reserved capacity quoted by sales. Both fit when you don't need Modal's Python jobs and Sandboxes or Baseten's self-hosted TensorRT-LLM tuning.

  • Because it throws away the prefix cache. Agentic and RAG applications resend thousands of tokens of shared system prompt, tool schemas and document context on every turn. Round-robin scatters consecutive turns from one workspace across vLLM pods, so each GPU recomputes the prefill from scratch. Bounded-load consistent hashing on a SHA-256 digest of the prefix pins a recurring prefix to one replica. Our blueprint targets a prefix cache hit rate above 78 percent that way, which cuts Time to First Token.

  • Once roughly three H100s stay warm around the clock. Below 400 H100 GPU-hours per month, or on spiky batch and sandbox workloads under a 15 percent duty cycle, Modal costs about $1,800 per month or less with no Kubernetes to run. In our worked model at 3,200 H100 GPU-hours per month under HIPAA and SOC 2 rules, an owned cluster on reserved neocloud H100s running vLLM, Karpenter and Run:ai S3 streaming totals $363,600 over 36 months with CEE engineering. That's $159,300 less than Modal and $314,400 less than Baseten. The GPU rates in that model are assumptions, so rerun it with your own quotes.

  • At Idealogic's senior CEE rate of $75 per hour, a 600-hour owned vLLM and SGLang Kubernetes build costs $45,000, versus $99,000 at a $165 per hour US agency rate. That covers Karpenter GPU bursting, Run:ai S3 streaming and KV-cache prefix routing, and in our worked model it pays back in 3.42 to 5.94 months. Smaller engagements cost less: a two to three week GPU serving, quantization and FinOps audit runs $11,250 to $18,000, and a Modal or Baseten enterprise production build runs $13,500 to $27,000.

Still unanswered
Ask us directly

A senior engineer replies under 4 hours.

Related expertise