AI API Integration: From First Call to Production

AI API integration is the work of calling a hosted model over HTTP from inside a product you already run, and building the machinery around that call so it survives real traffic. The call itself is four lines of code. Everything that makes it safe to ship (key handling, rate limit backoff, a latency budget, retries that do not double-charge you, a cost ceiling, a fallback path, and evaluations that catch quality regressions) is the actual engineering.

That gap is where most AI features die. A prototype on a laptop sends one request at a time, has no concurrency, never sees a 429, never sees a 529, and never gets billed twice for the same user action. Production has all four in the first week.

This is the guide we use when we wire a model into a live product: what to build, in what order, and which failure each piece exists to prevent. It assumes you have picked a provider and can already get a response back. Everything after that is the subject.

The short version

• AI API integration is calling a hosted model over HTTP and hardening the path around it. The call is trivial; the hardening is the job.
• Keys never reach the client. Calls go through a server-side gateway that owns auth, quotas, timeouts, retries, cost accounting, and provider routing.
• Rate limits are multi-dimensional. Both major providers meter requests and tokens separately, and you hit whichever runs out first.
• Decide a latency budget before you write the call, then propagate one deadline down the chain instead of letting every layer invent its own timeout.
• Retrying a model call is neither free nor idempotent: you get a different answer and a second bill. Anything that writes needs an idempotency key.
• Streaming fixes perceived latency, not total latency. Batch APIs halve the bill when nobody is waiting for the answer.
• Evaluation in production separates an integration that holds from one that degrades quietly when the provider ships a new model.

What AI API integration actually involves

AI API integration is connecting your application to a model someone else hosts, through an HTTP API, and then owning the reliability, cost, and quality of that connection. You are not training anything. You are adding a network dependency that is slow, metered, priced per token, occasionally unavailable, and nondeterministic. Each of those five properties breaks an assumption your existing code already makes about the services it calls.

Compare it to a payment API. Both are third party, both are metered, both cost money per call. But a payment API returns the same answer for the same input, finishes in a few hundred milliseconds, and hands you an idempotency key so a retry cannot charge the customer twice. A model API does none of that by default. It can take ten seconds, it returns different text every time, and a naive retry bills you again for a different answer.

So the integration is not the call. It is the set of components that make an expensive, nondeterministic, sometimes-unavailable dependency behave like a well-behaved one. The rest of this article is those components, roughly in the order we build them.

One clarification before the detail: this article is about the mechanics. If you are working out which capability to add first and what the delivery arc looks like, our AI integration services page covers that side, and the AI implementation guide covers the organizational rollout.

Where an AI API call sits in your request path

Decide this before you write any code, because it determines everything downstream. A model call is one to ten seconds of latency you are inserting into a system tuned for milliseconds, and there are only three honest places to put it.

Inline and synchronous. The user waits, your request thread waits, and your p95 absorbs the full model latency. Acceptable only for short outputs behind a hard timeout. This is the option teams pick by default and regret at scale, because a four-second call in a handler that used to take 80 milliseconds changes the concurrency math for your whole service.

Inline and streamed. The user waits but sees tokens immediately. Total time is unchanged, perceived time drops sharply. The cost is that you now hold a long-lived connection per active user, which has real infrastructure consequences covered below.

Out of band. The request enqueues a job, a worker calls the model, and the result arrives by webhook, poll, or push. Nobody holds a connection. This is the right answer whenever a human is not actively waiting, and it is the only shape that lets you use batch pricing.

The component that makes all three manageable is a server-side gateway: one internal service, or one module, that every model call in your product goes through. Competing guides tell you to "keep the key on the server" and stop there. The gateway is worth more than that. It is where auth, per-tenant quotas, timeouts, the retry policy, cost accounting, provider routing, prompt versioning, and PII redaction live, in one place, instead of being reimplemented slightly differently in six call sites.

Authentication and API key management

A provider API key is a bearer credential with a direct line to your billing account. Treat it the way you treat a database password, not the way you treat a public analytics token.

The rules are short. The key lives in a secret manager, injected as an environment variable at runtime, never committed and never bundled. It is never shipped to a browser or a mobile binary, because anything in a client build is public no matter how it is obfuscated. Clients authenticate to your gateway with your own session or token; the gateway holds the provider key. Use separate keys per environment so you can revoke staging without touching production, and rotate on a schedule rather than only after an incident.

The part teams skip is authorization. Your gateway now holds a credential that can spend money and, if the feature does retrieval, can read data. So the gateway has to answer two questions on every request: is this caller allowed to use this feature, and is this caller allowed to see the context you are about to put in the prompt. Retrieval that ignores your existing permission model is a data leak with extra steps, and it fails in a way that is hard to spot because the model paraphrases what it saw rather than quoting it.

Two OWASP categories are worth reading before you ship. Prompt injection covers untrusted input steering the model into actions you did not intend, which matters the moment your prompt contains user content or retrieved documents. Sensitive information disclosure covers exactly the leak described above. Neither is exotic; both are the default state of an integration nobody hardened.

Rate limits and quotas on AI APIs

Rate limits are the first production surprise, because they do not exist at prototype scale and they are not one number.

OpenAI meters requests per minute, requests per day, tokens per minute, tokens per day, and images per minute, and whichever limit you exhaust first is the one that stops you. Limits are tied to a usage tier that advances automatically as cumulative spend grows. On exceeding one you get HTTP 429, plus headers that tell you exactly where you stand: x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, matching -limit and -reset variants, and retry-after.

Anthropic meters requests per minute, input tokens per minute, and output tokens per minute, separately per model, so you can run several models concurrently up to their respective limits. It also returns 429 with retry-after and a set of anthropic-ratelimit-* headers whose reset values are RFC 3339 timestamps.

A few details from those docs change how you write the client:

• Limits are enforced on shorter windows than the name suggests. Anthropic notes that 60 requests per minute may be enforced as one request per second, so a burst can trip a limit you are nowhere near on a per-minute average. Smooth your outbound traffic rather than firing a batch of parallel calls.
• Anthropic uses a token bucket, so capacity replenishes continuously instead of resetting on a fixed boundary. Waiting for "the top of the minute" is the wrong mental model.
• Cached input tokens do not count toward the input-token limit on most Anthropic models. Prompt caching therefore raises effective throughput, not just cost efficiency. Their docs give the example of a 2,000,000 ITPM limit with an 80 percent cache hit rate processing 10,000,000 total input tokens per minute.

There is also an acceleration limit worth knowing about: a sharp jump in usage can produce 429s even when you are inside your nominal ceiling. Ramp traffic up gradually instead of launching a migration at full volume.

Practically, read the remaining-quota headers on every response and shed load before you reach zero, rather than treating 429 as the signal. By the time you get a 429 you have already failed a user request.

Setting a latency budget for AI API calls

Almost nobody writes this down, which is why almost every integration has a timeout of "whatever the SDK does." A latency budget is a small artifact and it prevents a specific, expensive class of outage.

Start from the user-visible surface and work inward. Decide the p95 you will accept for the whole interaction, subtract what your own code needs, and what remains is the model's budget. Then set the client timeout below the point at which the user has given up, and make sure it is shorter than any upstream proxy or load balancer idle timeout, so the failure is one you control and can log.

The rule that matters most comes from Google's SRE book: propagate a deadline rather than letting each layer invent one. Set the deadline high in the stack, reduce it as the request fans out, and check the remaining budget at each stage before doing more work. Their point is blunt: a heavily loaded server can spend eleven seconds moving a request from a queue to a thread pool, by which time the client has already given up, and every cycle spent on that request is wasted while the queue grows. A model call is the most expensive place in your system to do work nobody is waiting for.

Two measurements, not one. Time to first token is what a streaming user experiences. Total latency is what your connection pool and your worker capacity experience. Track them separately, because streaming improves the first and does nothing for the second. OpenAI's production guidance is explicit that latency is driven mainly by the number of tokens generated rather than prompt size, and that stream: true improves perceived responsiveness even when total time is unchanged.

Also cap the long tail explicitly. Anthropic's SDKs validate that a non-streaming request is not expected to exceed a ten-minute timeout, and their docs recommend streaming or the batch API beyond that, because networks drop idle connections and a long silent request is a request you may never get an answer to.

Streaming versus batch

These are the two escapes from synchronous latency, and they solve opposite problems.

Streaming an AI API response

Streaming is for when a person is waiting. Providers deliver tokens over Server-Sent Events, defined in the WHATWG HTML Living Standard with the text/event-stream media type: data: lines, events terminated by a blank line. The [DONE] sentinel most LLM APIs send at the end is a provider convention layered on top, not part of the standard.

Streaming has infrastructure consequences that catch teams out:

• Reverse proxies, load balancers, and CDNs buffer responses by default, which defeats the entire point. Buffering must be explicitly disabled on the streaming route.
• Idle timeouts close long streams. Heartbeats or comment events keep them alive.
• Each stream holds a connection for its full duration, so concurrency is now bounded by connections, not by CPU.
• Errors can occur after a 200 has already been sent. Anthropic documents this directly: a mid-stream error does not follow normal error handling, because the status line is long gone. Your client needs a distinct path for a stream that starts fine and dies at token 400.

Batching AI API calls when nobody is waiting

Batch is for exactly that case, and the pricing is the reason to care. OpenAI's Batch API gives a 50 percent discount against synchronous calls with a 24-hour completion window, up to 50,000 requests per batch. Anthropic's Message Batches API also reduces cost by 50 percent, with most batches finishing in under an hour. Backfills, bulk classification, offline enrichment, and evaluation runs should all be batch. Paying synchronous prices for work with no waiting user is a straightforward way to double a bill for nothing.

Retries and idempotency for AI API calls

This is the part most integration guides skip, and it is the one that costs money: retrying a model call is neither idempotent nor free. A retried database read returns the same rows at no extra cost. A retried model call returns different text and generates a second invoice. If the first call actually succeeded and only the response got lost on the way back, you have paid twice and may have performed the user's action twice.

Three separate mechanisms handle this, and you need all of them.

Retry only what is retryable, with a budget

Connection errors, 429, 500, and 529 are worth retrying. A 400 or a 401 never is. The official SDKs from both providers already retry transient failures with exponential backoff, twice by default, honoring retry-after, and both let you configure or disable that. Do not stack your own retry loop on top without accounting for it, or three layers of two retries becomes eight calls.

Cap attempts. Google's SRE guidance is three attempts per request and a per-client retry budget that only permits retries while the retry-to-request ratio stays under 10 percent. Their numbers make the case: the three-attempt cap on its own still leaves a threefold increase in requests during an incident, and layering the 10 percent budget on top brings that down to 1.1x. So an uncapped retry loop takes a provider that is merely struggling and finishes the job.

Add jitter to the backoff

Synchronized backoff means every client comes back at the same instant and knocks the provider over a second time. Randomize the delay so the herd spreads out.

Attach an idempotency key to anything that writes

This is the piece almost every guide omits. The IETF's Idempotency-Key header draft specifies the pattern: the client generates a unique value, a UUID is recommended, and never reuses it with a different payload. The server returns the stored result of the original operation for a duplicate that already completed, returns 409 Conflict if the original is still in flight, and 422 if the same key arrives with a different payload.

Your gateway should implement this on its own surface even where the provider does not. Store the key with the completed response, so a client retry after a dropped connection returns the answer you already paid for instead of buying a second one.

// Gateway handler: one deadline, capped retries, idempotent writes.
async function callModel(req: ModelRequest, deadlineMs: number) {
  const cached = await store.get(req.idempotencyKey);
  if (cached?.status === 'done') return cached.response;
  if (cached?.status === 'in_flight') throw new ConflictError(409);

  await store.markInFlight(req.idempotencyKey);

  for (let attempt = 0; attempt < 3; attempt++) {
    const remaining = deadlineMs - Date.now();
    if (remaining <= 0) break;

    try {
      const res = await provider.create(req, { timeoutMs: remaining });
      await store.complete(req.idempotencyKey, res);
      return res;
    } catch (err) {
      if (!isRetryable(err)) throw err;
      const wait = retryAfter(err) ?? backoffWithJitter(attempt);
      if (Date.now() + wait > deadlineMs) break;
      await sleep(wait);
    }
  }
  return fallback(req);
}

Note what the loop does with the deadline. It recomputes what is left before every attempt, passes that remainder down as the provider timeout, and refuses to start a sleep it cannot finish inside the budget. That is deadline propagation, and it is what stops the retry loop from holding a connection open long past the point where anyone is still waiting for the answer.

Controlling AI API cost per request

Per-token billing is quiet right up until it is not. The control that matters is a ceiling on every individual request, applied before any clever optimization.

Cap max_tokens, because unbounded output is unbounded cost and unbounded latency at the same time. Cap retries, since three attempts at a large prompt is three times the input bill. Cap retrieved context, because a retrieval bug that returns 200 chunks instead of 5 is a large invoice arriving through a code path that looks like it is working. OWASP tracks this class of failure under unbounded consumption, and it is as much a denial-of-wallet risk as a denial-of-service one.

Then take the structural discounts, which are larger than anything you will achieve by trimming prompts:

Prompt caching deserves particular attention because the economics are lopsided. On Anthropic, a cache read costs a tenth of the base input price, while a write to the default five-minute cache costs 1.25x. Anything stable and repeated across requests (system instructions, tool definitions, a large reference document) pays for itself quickly. OpenAI's caching applies to prefixes of at least 1,024 tokens and keeps a cached prefix eligible for reuse for at least 30 minutes. In both cases the structural requirement is the same: put the stable part of your prompt first and the variable part last, or you get no cache hits at all.

Finally, attribute the spend. Cost per feature and per tenant, computed from the token counts in every response, is the only way to see which thing is growing. A monthly total tells you far too late. For the wider build-versus-run picture, our guide to AI development cost breaks down where the money actually goes.

The cheapest AI feature is not the one with the smallest model. It is the one with a hard ceiling on every request, because your bill tracks unbounded inputs far more closely than it tracks model choice.

Fallback when the provider degrades

Providers have bad days, and every AI API integration that runs long enough will be awake for one. Anthropic documents 529 overloaded_error for temporary saturation across all users and 500 api_error for internal failures, alongside 504 timeout_error for requests that ran too long. These are normal operating conditions at scale, not exceptional ones, and "add a fallback model" is not a plan.

What you need is a degradation ladder, decided in advance, with each rung cheaper and more certain than the last:

1. Retry with backoff, inside the deadline, capped at three attempts.
2. Downgrade the model. A faster, smaller model from the same provider usually has separate capacity and a different limit.
3. Switch providers. Only works if you built the adapter layer, pinned model IDs, and know from evaluations how much quality you lose.
4. Serve a cached or deterministic answer. A slightly stale response or a rules-based one beats an error for many features.
5. Turn the feature off cleanly. Hide the entry point, tell the user plainly, and keep the rest of the product working.

Put a circuit breaker in front of the ladder. When a provider is failing, continuing to send full traffic makes recovery slower for everyone and burns your retry budget on requests that cannot succeed. Trip the breaker, drop to the next rung, and probe occasionally.

Rung five is the one to design first, and the one most teams never build. The question is not whether the AI feature can fail. It is whether the checkout, the dashboard, or the ticket queue it lives inside still works when it does.

Evaluating an AI API integration in production

An integration that ships without evaluation does not stay working; it degrades silently, and you find out from a customer. This is the least-covered topic in every competing guide, and it is the one that decides whether the feature is still good in month six.

The machinery is not large, but all of it has to exist.

A golden set in CI

Thirty to two hundred fixed cases with known-good outputs, scored on every change to a prompt, a model ID, a retrieval configuration, or a parameter. It does not need to be large. It needs to be hard, and it needs to run automatically, because the whole point is to catch a regression before a user does. Run it through a batch API and it costs half as much.

Online sampling of live AI API traffic

Score a small percentage of real production traffic continuously. Golden sets tell you about the inputs you thought of; sampling tells you about the ones you did not. Track quality alongside cost and latency, per prompt version, so a regression is attributable.

Model version pinning and a deprecation watch

This is the incident nobody writes about and everybody eventually has: the prompts stopped working and no code changed. Pin explicit model IDs rather than floating aliases, so an upgrade is a deliberate act you evaluate first. Then watch the retirement schedule, because pinning does not exempt you from it. Anthropic gives at least 60 days' notice before retiring a publicly released model and requests to a retired model simply fail. Their published history bears the notice period out: claude-opus-4-1-20250805 was deprecated on 5 June 2026 and retired on 5 August 2026. Sixty days is enough time to evaluate a replacement, and nowhere near enough to discover the problem in production and then evaluate one.

Log enough to debug all of this. One span per model call, carrying the provider request ID, the model ID actually used, prompt version, input and output token counts, cache read and write tokens, time to first token, total latency, finish reason, retry count, and computed cost. Anthropic returns a request-id header on every response and asks for it on support tickets, so capture it rather than reconstructing it later.

Hosted AI API versus self-hosting

The honest framing is not cost. It is which problems you would rather own.

With a hosted API, the provider owns the hard parts: capacity, uptime, hardware, model updates, and scaling. In exchange you accept their rate limits, their retirement schedule, their 529s, and their data terms. You pay per token, which scales linearly forever and is excellent at low volume and unremarkable at very high steady volume.

Self-hosting an open-weights model inverts every one of those. Nobody rate-limits you and nobody retires your model out from under you, which for some regulated workloads is the entire argument. But you now own GPU capacity planning, inference serving, batching and throughput tuning, model upgrades, and being on call for all of it. The cost profile flips from per-token to fixed capacity, which is efficient at high, steady, predictable utilization and wasteful at spiky or low volume.

What actually decides it, in our experience, is rarely the spreadsheet:

• Data residency and compliance. If data cannot leave your infrastructure, or a specific contractual posture is required, that settles it regardless of the arithmetic. In regulated fintech and healthtech work this is usually the deciding factor, well ahead of cost.
• Volume shape, not volume size. High and flat favors owned capacity. Spiky favors per-token pricing, because idle GPUs bill the same as busy ones.
• Whether you want an inference team. Self-hosting is not a deployment; it is an ongoing operational commitment with a headcount attached.

Most product teams should start hosted and revisit when a specific constraint forces the question. The hybrid is also legitimate and common: a self-hosted small model for high-volume classification, a hosted frontier model for the work that needs the quality. If fine-tuning is the direction you are weighing, our guide to fine-tuning an LLM covers when it earns its keep.

Shipping an AI API integration without a rewrite

You do not add AI in one launch. You put it at the edge and grow it inward.

Pick one narrow job. Build the gateway first, even if it wraps a single call, because everything above lives there and retrofitting it across six call sites is far worse than building it once. Put the feature behind a flag, ship to a slice of traffic, and wire the evaluation set in before the first user sees it, so you are measuring rather than hoping.

Then widen on evidence, and stack the next capability on top. A plain completion first. Retrieval over your own data when the answer depends on facts the model does not have. An agent that takes actions only when the task genuinely requires multi-step tool use, since every action an agent can take is an action it can take wrong. Each step should earn the next.

What separates an AI API integration that holds from one that falls over is almost never prompt quality. It is whether someone treated the model as a slow, metered, nondeterministic dependency and engineered around it. Most of that work looks like any other critical third-party integration, which means your team has probably done it before, just never against a dependency that answers differently every time you ask.

For the wider practice, our AI development hub maps the full picture, and generative AI development covers the patterns that show up most in content-heavy features.