LLM Architecture: From Transformer Block to Production

LLM architecture means two different things, and almost every argument about it comes from people using the term in both senses at once. The first sense is the model itself: tokenizer, embeddings, a stack of transformer blocks, and a decoding step that picks the next token. The second sense is the system around the model: context assembly, retrieval, tools, memory, orchestration, evaluation, observability. Both are real architecture. Only one of them is architecture you get to design.

If you are shipping a product, the second sense of LLM architecture is where your quarter goes. The model arrives as a fixed artifact with a price per token and a context limit, and you build a system around it. That system is where latency, cost, accuracy and failure behavior actually get decided, and it is the layer that nobody publishes a clean diagram of.

That said, the internals are not decoration. A handful of them, roughly five, change decisions you will make about serving, cost and context budget. The rest are fascinating and will not alter a single line of your design. This guide covers both levels: enough of the transformer to know which internals matter, then the application architecture in full, with the trade-off attached to every choice.

The short version

• LLM architecture splits into model architecture (fixed when you pick a model) and application architecture (everything you control). Say which one you mean before the meeting starts.
• Inside the model, the internals that change your decisions are the context window, the KV cache, the attention variant, mixture-of-experts sparsity, and the tokenizer. The rest is background.
• An LLM application has eight components: model, context assembly, retrieval, tools, memory, orchestration, evaluation, observability. Skipping the last two is the single most common architectural failure.
• Capability sits in one of four layers: prompting, retrieval, fine-tuning, continued pre-training. Climb only on evidence, because each rung costs an order of magnitude more than the one below.
• Use retrieval when the model lacks knowledge. Use fine-tuning when it lacks behavior. Most teams who say they need fine-tuning have a retrieval problem.
• Routing between a cheap model and a strong one saves more money than any other single change, and published research puts the range between two-fold and near-total.
• The architecture that survives a prototype is not the architecture that survives production traffic. Caching, streaming, fallbacks and concurrency limits are design decisions, not patches.

What LLM architecture means at two levels

Model architecture is the shape of the network: how many layers, how attention is computed, how positions are encoded, how the vocabulary maps to vectors. It is decided by whoever trained the model, and you change it only by training. When a paper says "architecture", this is what it means.

Application architecture is the shape of the system: what goes into the context window on each call, what the model is allowed to invoke, how many times it runs before answering, what happens when it returns nonsense. It is decided by you, it changes weekly, and it is what a production incident is usually about.

The two levels connect at exactly three points, which is worth stating plainly because most diagrams blur it:

• The context window is the interface between them. Model architecture sets its size and its cost per token; application architecture decides what goes in it.
• The serving profile is set by model internals (parameter count, attention variant, sparsity) and consumed by your architecture as latency and dollars per request.
• The output contract is a probability distribution over tokens on the model side and a parsed, validated object on yours.

Everything else is separable. That separation is good news, because it means you can swap the model without redesigning the system, and most teams do exactly that two or three times a year.

Inside the model: the transformer architecture in brief

Every current large language model descends from one 2017 paper. Vaswani and colleagues proposed an architecture "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" (Vaswani et al., 2017). Nine years later the core loop is unchanged; what moved is efficiency, scale and the tricks around attention.

Here is the path a request takes through the model, which is the diagram every explainer draws and few explain in terms of consequences.

Stage	What happens	Why an engineer cares
Tokenization	Text is split into subword units and mapped to integer IDs	Token count is the billing unit and the context budget
Embedding	Each ID becomes a vector; position information is added	Determines how the model perceives order and distance
Transformer blocks	Attention mixes information across positions; a feed-forward network transforms each one	Depth and width set both quality and serving cost
Output layer	Hidden states become a probability over the whole vocabulary	Sampling settings here control determinism

Tokens and embeddings: how text enters the model

A tokenizer splits text into subword pieces. This is worth one minute of your attention because it decides your bill: the same paragraph can cost noticeably more tokens in one model's vocabulary than another's, and non-English text, code and long identifiers fragment much harder than plain English prose. Any cost model built on word counts rather than token counts will be wrong.

Each token ID becomes a dense vector, and position information is added so the model knows word order, since attention on its own is order-blind. Modern models mostly use rotary position embeddings, which encode position by rotating query and key vectors rather than adding a fixed signal (Su et al., 2021). The practical consequence is that position handling is now the thing being modified whenever a lab extends a context window, which is why long-context variants appear as separate releases.

Attention, and the architecture variants that cut serving cost

Attention lets every token look at every other token and decide what is relevant. Multi-head attention runs several of these comparisons in parallel over different learned subspaces. That is the part of the architecture that made the transformer work, and it is also the part that makes it expensive, because attention cost grows quadratically with sequence length.

Two engineering problems follow, and both produced architecture variants you will meet in model cards:

• Generation recomputes everything. Producing token 501 requires attention over the previous 500. Models cache the key and value tensors instead, which is the KV cache. It grows with context length and with concurrent users, and on long conversations it can consume more GPU memory than the weights.
• The cache is the bottleneck, so shrink it. Grouped-query attention gives several query heads a shared set of key-value heads. The paper reports that "uptrained GQA achieves quality close to multi-head attention with comparable speed to" multi-query attention, and that converting an existing checkpoint takes only "5% of original pre-training compute" (Ainslie et al., 2023). Multi-head latent attention goes further by compressing the cache into a latent vector: DeepSeek-V2 reports that it "reduces the KV cache by 93.3%" and "boosts the maximum generation throughput to 5.76 times" its predecessor (DeepSeek-AI, 2024).

If you never self-host, this is trivia. If you do, it is your capacity plan.

Feed-forward layers, normalization and depth

After attention mixes information across positions, a feed-forward network transforms each position independently. This is where most of a dense model's parameters live. Normalization and residual connections wrap both sublayers so that gradients survive a deep stack, which is unglamorous plumbing that made scaling past a few dozen layers possible at all.

Stack that block dozens of times and you have the model. Depth and width are the two dials, and their product is roughly what you pay to serve it.

Mixture of experts breaks that relationship, and it now dominates open releases as a result. Instead of every parameter running on every token, a router activates a small subset. DeepSeek-V3 has "671B total parameters with 37B activated for each token" (DeepSeek-AI, 2024). The architectural consequence is a model with the knowledge capacity of something enormous and the per-token compute of something mid-sized, at the price of needing all 671 billion parameters resident in memory. For a hosted API you see this only as unusually good quality per dollar. For self-hosting it changes the hardware conversation completely.

Decoding: how the next token is chosen

The final layer produces a score for every token in the vocabulary. Softmax turns those into probabilities, then a sampling strategy picks one: greedy selection takes the highest, temperature flattens or sharpens the distribution, top-k and top-p restrict the candidate pool.

This is the one internal that sits directly in your hands through the API, and it is routinely misused. Teams building extraction or classification pipelines leave temperature at a creative default and then file bug reports about inconsistency. Anything with a schema wants low temperature. Anything with a voice wants more. Nothing wants the default just because it is the default.

Types of LLM architecture: decoder-only, encoder-decoder and encoder-only

The original transformer had two halves, and the field split them apart. Three families resulted, and the difference is which tokens each position is allowed to see.

Family	How it reads	Representative models	Where it fits in your architecture
Encoder-only	Every token sees the whole sequence in both directions	BERT, and most embedding models	Embeddings, reranking, classification
Encoder-decoder	An encoder reads the input, a decoder writes the output	T5, and most translation models	Fixed input-to-output transformations
Decoder-only	Each token sees only what came before it	GPT, Claude, Llama, Mistral, Qwen	Generation, chat, tool use, agents

Decoder-only won the general-purpose slot for a structural reason rather than a benchmark one. Its training objective is next-token prediction, so any text at all is usable training data with no paired examples required, which makes pre-training data effectively unlimited. The same design also absorbs classification, extraction and summarization once you phrase them as generation, so one model replaces a shelf of task-specific ones.

Encoder-only models did not disappear, they moved. Nearly every retrieval layer in production runs one to produce embeddings and often a second to rerank results. A typical LLM architecture therefore contains at least two model families, which is easy to miss because only one of them ever produces text a user reads.

Which model internals actually change an architecture decision

Here is the filter promised at the top. Of everything above, five internals reach your design. The rest belongs to the people training models.

Internal	Decision it changes	What it costs you to ignore
Context window size	How much retrieved material you can send per call	Truncated context and silently dropped instructions
KV cache growth	Concurrent users per GPU, and long-conversation cost	Capacity planning that collapses under real traffic
Attention variant (GQA, MLA)	Whether a given model is affordable to self-host	Choosing on benchmarks, then discovering the serving bill
Mixture-of-experts sparsity	Memory footprint versus per-token compute	Sizing hardware from the headline parameter count
Tokenizer and vocabulary	Real cost per request, especially for non-English or code	Budgets modelled in words that miss by a wide margin

Everything else, normalization placement, activation functions, the exact positional scheme, expert routing strategy, is fascinating and irrelevant to your architecture in equal measure. You cannot change it, and it does not change what you build.

There is one more internal worth knowing that is a property of behavior rather than of the network: models do not use long contexts evenly. Liu and colleagues found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., 2023). That is an architecture instruction, not a curiosity. It means a bigger context window is not a substitute for good retrieval, and that where you place retrieved passages in the prompt is a design choice with measurable consequences.

A larger context window does not fix bad retrieval. It just moves the failure somewhere harder to see.

The components of an LLM application architecture

This is the level you design. Eight components show up in every serious LLM architecture, whether or not the team has named them.

Component	What it does	Most common failure
Model layer	Runs inference, often with more than one model	One expensive model doing work a cheap one could
Context assembly	Builds the prompt: instructions, history, retrieved material, schema	Silent truncation when inputs grow
Retrieval	Fetches relevant documents at query time	Chunking decided once and never measured
Tools	APIs the model can call to read or change the world	Unvalidated arguments on privileged actions
Memory	What persists across turns and sessions	Unbounded history that inflates every call
Orchestration	Sequences calls, retries, decides when the task is done	Agentic loops where a fixed sequence would do
Evaluation	Measures whether a change made things better	Added after launch, when it is far more expensive
Observability	Traces, token counts, latencies, failures in production	No way to answer why yesterday got worse

Three of these deserve more than a table row, because they are where architectures go wrong.

Context assembly is the real prompt engineering. By the time a request reaches the model, several sources have competed for room: system instructions, conversation history, retrieved passages, tool definitions, output schema. Something has to decide the budget and the order. Teams that leave this implicit find that adding a feature quietly pushed the instructions out of the window, and the symptom looks like the model getting dumber for no reason.

Memory is a design decision, not a feature. Short-term memory is the conversation you replay each turn, and replaying it in full means every turn costs more than the last. Long-term memory is what survives the session, which means storage, retrieval and a policy on what to forget. Both are cheap to bolt on and expensive to bolt on badly.

Orchestration is where complexity gets added for free and paid for later. Agent frameworks make elaborate multi-step loops tempting on day one. Anthropic's guidance draws the line clearly: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage", and the recommendation is to find "the simplest solution possible, and only increasing complexity when needed" (Anthropic, 2024). A fixed sequence can be tested, reasoned about and bounded in cost. Reach for autonomy only where the task branches in ways you cannot enumerate, and treat every tool call as a privileged action: validate arguments, scope permissions, log what happened, keep a human in the loop for anything irreversible.

Where to put capability in your LLM architecture

The most consequential architecture question is not which framework or which vector database. It is which layer holds the thing that makes your system yours. There are four candidates, and they differ by roughly five orders of magnitude in cost.

Layer	What it changes	Data you need	Time to first result
Prompting and context	Nothing in the model; everything in the input	A handful of good examples	Days
Retrieval	What the model knows at query time	Documents you already hold	Four to eight weeks
Fine-tuning	The model's default behavior	Hundreds to thousands of labeled examples	Two to six weeks after the eval set exists
Continued pre-training	The model's underlying domain knowledge	Billions of tokens of domain text	A quarter or more

The rule is boring and it holds: climb only when you have evidence the layer below cannot do the job, where evidence means a failing eval set rather than a hunch. Teams that skip layers usually end up back at retrieval, having spent a quarter learning that their fine-tune could not recall a policy document because the policy changed after training.

Layer one: prompting and context engineering

The first instinct is to treat prompting as copywriting. Pick better words, get better output. That works until the inputs vary, and then it does not. Real prompt design is closer to interface design: you are defining a contract that states the role, the constraints, the shape of the answer, and what to do when the answer is not available. Wording matters less than structure.

Three things hold up in production. Be explicit about failure, because a line like "if the contract is not in the provided context, say you cannot find it" prevents a whole class of confident wrong answers. Constrain the output format, and validate it, since free text that usually parses will eventually break a Friday deploy. Use examples sparingly, because two well-chosen ones anchor format and tone better than ten generic ones and cost far less context.

For internal tooling and well-scoped tasks, a sharp prompt over a strong base model is the entire architecture. Reaching past it is premature optimization dressed up as ambition.

Layer two: retrieval, when the custom part is your data

Retrieval-augmented generation is where most LLM architectures should stop. It exists for one reason: the model lacks knowledge. Your policies, your product docs, last quarter's numbers, a customer's ticket history. None of it sits in the weights, and it changes too fast to bake in.

RAG fetches the relevant pieces at query time and places them in context, so the knowledge updates when you edit a document rather than when you retrain. That property alone explains why retrieval beats fine-tuning for most enterprise work: an answer that has to be current cannot live in weights frozen last March. It also keeps the system auditable, because you can show which passage produced which sentence.

The work is unglamorous and it is where quality comes from. Chunking sets your ceiling before you write a line of prompt. Embedding choice and reranking decide whether the right passage reaches the model at all. Citation plumbing decides whether anyone trusts the output. We go deeper on all of it in our guide to building RAG systems on LLMs.

One test tells you whether you are on this layer: write down a failing answer, then ask whether a human expert could have answered correctly given only a folder of your documents. If yes, you have a retrieval problem, and no amount of weight adjustment will fix it.

Layer three: fine-tuning on your own examples

Fine-tuning is for when the model lacks behavior. It knows the facts but will not reliably produce the format, tone or classification pattern you need. Fine-tuning adjusts weights on your examples so that behavior becomes the default instead of a paragraph of instructions you pay for on every call.

Decision axis	Retrieval-augmented generation	Fine-tuning
What it fixes	Missing knowledge: private or current facts the model never saw	Missing behavior: a format, tone or classification it will not follow
How you update it	Edit or re-index a document, no retraining	Retrain the weights on new labeled examples
Cost to iterate	Low, mostly data plumbing	Higher, needs curated examples and training runs
Auditability	High, you can point to the sources it used	Lower, the behavior is baked into the weights
Reach for it when	The model needs facts it does not have	Prompting plus retrieval still fights you on format

Two techniques made this layer affordable. LoRA freezes the base model and trains small low-rank matrices instead, which the original paper reports can "reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" relative to fine-tuning GPT-3 175B with Adam (Hu et al., 2021). QLoRA adds 4-bit quantization of the frozen base and gets far enough that a 65B parameter model fits "on a single 48GB GPU while preserving full 16-bit finetuning task performance" (Dettmers et al., 2023).

The data floor is lower than most teams assume. OpenAI's fine-tuning documentation puts the minimum at 10 examples and recommends "starting with 50 well-crafted demonstrations and evaluating the results" (OpenAI). Consistency across that training data matters far more than volume. Five hundred clean, uniform demonstrations beat five thousand contradictory ones, and a contradictory dataset produces a model that is confidently inconsistent, which is worse than the base model you started with. Before any of it goes into a training run, check that you hold the rights to use it: customer text collected under a processing agreement written for support tickets rarely covers baking that text into weights you then serve to other customers.

There is a platform note worth carrying into any 2026 plan. OpenAI is winding down its fine-tuning platform: the documentation now states it "is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months" (OpenAI). If your architecture assumes tuning on a hosted API, check the API still offers it before the roadmap depends on it. Open-weight models carry no such risk, which is a quiet argument for them that had nothing to do with quality. Our practical walkthrough of fine-tuning an LLM covers the mechanics.

Layer four: continued pre-training

Continued pre-training, sometimes called domain-adaptive pre-training, keeps training an existing base model on a large corpus of domain text before any instruction tuning. This is the first layer where the model's underlying sense of a domain shifts, not just its surface behavior.

The canonical worked example is BloombergGPT, a 50 billion parameter model trained on "a 363 billion token dataset based on Bloomberg's extensive data sources" combined with "345 billion tokens from general purpose datasets" (Wu et al., 2023). Note the shape of that: roughly half the corpus is proprietary financial text Bloomberg had accumulated over decades, and it still needed the general half to remain a usable language model.

That ratio is the entry test. You need a domain corpus measured in hundreds of billions of tokens, and it has to be text open models have not already absorbed. Public regulatory filings do not qualify, because every frontier model has read them. Forty years of proprietary trading commentary does. If you cannot name a corpus of that scale that is genuinely yours, you are not on this layer, and no vendor proposal should put you there.

The honest counterpoint: BloombergGPT was published in 2023 and the frontier has moved. General models now perform strongly on many finance benchmarks without domain pre-training, which is why Andreessen Horowitz found enterprises drifting toward retrieval and fine-tuning instead, reporting that "most are opting not to train their own LLM from scratch and instead use retrieval-augmented generation (RAG) or fine-tune an open source model for their specific needs" (a16z, 2024). This layer narrows every year.

Why almost nobody should pre-train from scratch

Pre-training a foundation model is a fifth option in a different budget category. Two published figures make the scale concrete.

Meta's Llama 3.1 405B was pre-trained on 15.6 trillion tokens using up to 16,000 H100 GPUs, and Meta's model card puts the 405B run at 30.84 million H100-80GB GPU-hours, out of 39.3 million cumulative across the whole Llama 3.1 herd (Meta). DeepSeek-V3, a deliberately efficiency-focused effort, required 2.788 million H800 GPU-hours for full training on 14.8 trillion tokens, which the team priced at $5.576 million assuming $2 per GPU-hour (DeepSeek-AI, 2024).

The DeepSeek number is the one people quote as proof that pre-training got cheap. Read the paper's own caveat first: "the aforementioned costs include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data." That excluded work is where the expertise lives, and it is not a rounding error. The headline figure is the cost of the final run once a research team already knows exactly what to run.

The real exceptions share a shape. A proprietary corpus at pre-training scale that you have legal right to train on. A non-text modality such as protein sequences, chip layouts or telemetry, where general language pre-training gives you little. A sovereignty requirement that forces you to account for every token in the training set. Or the model itself is the product, in which case its weights are your differentiator and the economics change completely. Note what is absent: "our industry is very specialised" is answered by fine-tuning, and "our data is very sensitive" is answered by where you deploy, not by how you train.

Single model versus routing in an LLM architecture

Almost every system starts with one model behind one endpoint, and almost every system that reaches real traffic ends up with more than one. It is the highest-leverage cost decision in an LLM architecture, and it is usually made by accident.

The logic is simple. Requests are not uniform. Some are trivial classification, some need genuine reasoning, and paying frontier prices for both means subsidising the easy ones. A router sends each request to the cheapest model that can handle it, with the strong model as fallback.

Published research puts real numbers on the gain. FrugalGPT, which cascades from cheap models to expensive ones and stops when an answer is good enough, reports that it "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost" (Chen et al., 2023). RouteLLM, which learns to route from preference data, "significantly reduces costs by over 2 times in certain cases without compromising the quality of responses" and keeps working "even when the strong and weak models are changed at test time" (Ong et al., 2024).

Treat the 98 percent figure as a ceiling from a favourable benchmark rather than a forecast. The two-fold saving is the number to plan against, and it is still enormous at scale.

Routing carries a cost of its own, and that is what makes it an architecture decision instead of an optimization. You now have two quality bars to maintain, a classifier that can itself be wrong, and a debugging story that starts with "which model answered this". Three rules keep it manageable:

• Route on evidence, not intuition. Build the eval set first, run it against both models, and route the categories where the cheap model measurably holds up.
• Make the routing decision observable. Every trace records which model ran and why, or you will never diagnose a quality regression.
• Keep the escape hatch. A confidence threshold or a validation failure should escalate to the strong model rather than returning a bad answer.

The related shift worth watching is downward. A distilled model in the 3B to 8B range, fine-tuned on one narrow job, now does work that needed a frontier model two years ago, at a fraction of the serving cost and inside a boundary you control. For a large share of routing decisions, the cheap branch is now genuinely good.

Choosing a base model for your LLM architecture

Once you know which layer holds your capability, the base model follows from two questions rather than from a leaderboard: where the model has to run, and how much of the licence you can live with.

Base model choice	Where it can run	Customization available	Main constraint
Hosted frontier API (GPT, Claude, Gemini)	Provider cloud only	Prompting, retrieval, sometimes tuning	Weights never leave the provider
Permissively licensed open weights (Apache-2.0, such as Mistral or Qwen)	Anywhere you have GPUs	Every layer, including continued pre-training	You operate the serving stack
Conditionally licensed open weights (Llama)	Anywhere you have GPUs	Every layer	Naming, attribution and scale clauses apply
A small model distilled for one task	Edge, CPU, or a single GPU	Fine-tuning, distillation	Narrow capability by design

Data residency usually settles this before quality does. A clinical or defence workload that cannot send text to a third-party cloud has already chosen open weights, whatever the benchmarks say. Everyone else should start on a hosted API and move only when a measured reason appears: an eval the API fails, a latency budget it misses, or an inference bill a smaller tuned model would halve.

The internals from earlier come back here. If you are self-hosting, the attention variant and the sparsity pattern decide whether a model fits your hardware at your concurrency, and two models with identical benchmark scores can differ several-fold in what they cost you to serve. Our AI development team runs this comparison on serving cost at target concurrency rather than on leaderboard position, because that is the number that shows up in the invoice.

Evaluation and observability belong in the architecture

Most LLM architecture diagrams stop at the model and the data stores. That omission is why so many systems ship and then quietly degrade: the components that tell you whether the thing works are treated as tooling rather than as architecture.

Evaluation is what makes a system safe to change. The model is non-deterministic, the inputs are open-ended, and "correct" is often a judgment call, so you cannot test it like a normal function and "it looked good when I tried it" is not a test. Teams that ship reliable systems keep a real eval set of a few dozen to a few hundred representative inputs with known-good answers, grade outputs with a mix of exact checks and model-as-judge rubrics, and treat that set as a versioned asset. It is also the only honest way to measure hallucination on a domain-specific task, since a general benchmark tells you nothing about whether the model invents clauses in your contracts.

Observability is the production half of the same job. An LLM call is a distributed-systems span with unusual attributes: which model, how many input and output tokens, how long, which tools fired, what came back. OpenTelemetry now maintains dedicated conventions for exactly this, covering "spans, metrics, and events for GenAI clients, MCP (Model Context Protocol), and provider-specific conventions" (OpenTelemetry). Adopting a standard schema early costs an afternoon and means your traces survive a change of vendor.

If you can't measure whether a change made the system better, you're not developing. You're redecorating.

The pairing matters more than either half. Offline evals catch the problems you already know about; production traces show you the ones users found first. A healthy loop moves failures from the second into the first, so every incident permanently enlarges the eval set.

How an LLM architecture changes from prototype to production traffic

The LLM architecture that gets a demo applauded is not the one that survives a Monday. Four things change, and each is a design decision rather than a patch.

Caching stops being optional. Long system prompts and stable retrieved context get re-sent on every call, and providers now bill that differently. Anthropic's documentation states that "cache read tokens are 0.1 times the base input tokens price", with 5-minute cache writes at 1.25 times and 1-hour writes at 2 times (Anthropic). OpenAI applies caching to "prefixes containing at least 1,024 tokens", which it calls "a strict minimum" (OpenAI). Both facts have the same architectural implication: put the stable parts of your prompt first and the variable parts last, or you will pay full price for content that never changed. Prompt layout is now a cost decision.

Latency becomes a structure problem. A multi-step chain that takes ten seconds feels broken regardless of answer quality. Streaming the first token early, running independent retrieval and tool calls concurrently rather than in sequence, and routing easy queries to a faster model are all architectural responses. Adding a spinner is not.

Failure modes multiply. Provider APIs rate-limit, time out and have incidents. Production architecture needs timeouts, retries with backoff, a circuit breaker, and ideally a fallback model on a different provider. The last one is only cheap if you built the model layer as an interface rather than as a hard-coded client, which is a decision made on day one and regretted in month six.

Concurrency exposes the internals. If you self-host, the KV cache from earlier is now your capacity ceiling, and it grows with both context length and simultaneous users. Teams that sized hardware from a single-user benchmark discover this in the worst possible week. Cap conversation history, summarize instead of replaying, and measure memory at your target concurrency before you commit to a GPU count.

Under all four sits one habit: know your unit economics per request before traffic arrives. Token costs are small per call and large in aggregate, and the difference between a healthy margin and a bad one is usually a cache hit rate and a routing rule.

What each layer of an LLM architecture costs

Cost in an LLM architecture tracks the layer, and the shape of the spend changes as you climb. On the lower layers almost all of it is engineering time. On the upper ones compute takes over, and by continued pre-training the compute line dwarfs everything else.

Layer	Data cost	Compute cost	Dominant expense
Prompting and context	Days of curation	Inference tokens only	Engineering time
Retrieval	Weeks of cleaning and chunking	Embedding runs plus a vector store	Data plumbing
LoRA or QLoRA fine-tune	Labeling and eval-set construction	Tens to hundreds of dollars of GPU time per run	Data curation and evaluation
Continued pre-training	Corpus acquisition and licensing	Thousands to millions of GPU-hours	Compute, then compute again
Pre-training from scratch	A corpus you probably do not have	Millions of GPU-hours	A research team you definitely need

Published prices give you an anchor for fine-tuning. OpenAI lists supervised fine-tuning at $25 per million training tokens on gpt-4.1 and $5 on gpt-4.1-mini (OpenAI), so a dataset of a few thousand demonstrations is a rounding error against a single engineer-week. Self-hosted LoRA runs land in the same territory once you price GPU time near the $2 per hour DeepSeek assumed for H800s. The training run is never the expensive part.

Two cost traps deserve naming. The first is treating that run as the project: a LoRA run on a mid-size open model is cheap, but the eval set that tells you whether it worked is not, and it usually takes longer to build than the model takes to train. The second is forgetting that custom weights are an asset you maintain. A fine-tuned model needs hosting, monitoring and periodic retraining as the world moves; a prompt does not. That ongoing line is what turns a one-quarter project into a permanent team commitment, and it belongs in the business case from day one. For the wider picture, our breakdown of what AI development actually costs puts these numbers alongside the rest of a build.

What you own at each layer of the architecture

This question gets asked last and should be asked first, because ownership does not scale with spend the way people assume.

Prompting and retrieval are cleanly yours. Prompts, chunking logic, embeddings, the retrieval index and the evaluation set are all your artifacts, and they port between model providers in an afternoon. That portability is an underrated asset: teams who stayed at the retrieval layer swapped model providers three times over the past two years without rewriting anything of substance.

Fine-tuning depends entirely on the base licence. A LoRA adapter you trained over an open-weight model is yours to host, but the base model's terms travel with it. Llama 3.1 is instructive because it is widely used and its Community Licence carries real conditions. Distribute a model derived from it and you "shall also include 'Llama' at the beginning of any such AI model name", and you must "prominently display 'Built with Llama' on a related website, user interface, blogpost, about page, or product documentation". There is a scale clause too: cross 700 million monthly active users on the release date and you must request a separate licence from Meta (Meta). Apache-2.0 models carry none of this, which is a reason to check the licence before the benchmark.

A fine-tune on a hosted API is something you rent. You cannot export the weights, the provider controls the lifecycle, and as OpenAI's wind-down shows, the platform can close to new users while a competitor's open-weight adapter keeps running on hardware they rent by the hour.

Continued pre-training can make you a regulated provider. Under the European Commission's guidelines for general-purpose AI models, a downstream modifier becomes the provider of the modified model when "the training compute used for the modification is greater than a third of the training compute of the original model". Where the original figure is unknown, the threshold falls back to a third of 10^23 FLOP for an ordinary GPAI model, or a third of 10^25 FLOP for one with systemic risk (European Commission, 2025). Cross that line and the transparency and documentation obligations of a model provider attach to you, not to the lab whose weights you started from. Very few teams pricing a continued pre-training run have this on the slide.

How to design your LLM architecture: a decision sequence

Run this in order and stop at the first answer. It takes an afternoon and saves quarters.

1. Write twenty failing cases. Real inputs where a strong base model with a decent prompt gets it wrong. If you cannot produce twenty, you do not yet have evidence to justify any architecture beyond an API call.
2. Sort each failure into facts or behavior. Was the model missing something it never knew, or refusing to do something the way you need? Two piles, no third.
3. If the facts pile is larger, build retrieval. Nothing you do to the weights will help, because weights cannot contain something that changes weekly.
4. If the behavior pile is larger, try harder at prompting first. A serious pass with explicit failure handling and a constrained output schema resolves a large share of cases that started as "we need to fine-tune".
5. If prompting plus retrieval still fails on behavior, fine-tune. You now have a labeling target that is specific and defensible: the cases that survived steps three and four.
6. Only consider continued pre-training if you can name the corpus. Hundreds of billions of tokens, proprietary, legally yours, in one sentence. If you cannot, the answer is no.
7. Decide the model layer separately. Single model or router, hosted or self-hosted, based on the eval results and the serving cost at your target concurrency.
8. Build evaluation and observability alongside, not afterwards. Every step above produces test cases. Keep them.

The one honest shortcut past this sequence is latency and unit cost. If a small fine-tuned model can do at 8B parameters what you currently pay a frontier model to do, the fine-tune pays for itself in inference savings regardless of quality gain. That is a finance decision with an engineering method, and it belongs in a spreadsheet before it reaches a roadmap.

A last word on where to start. Resist the urge to pick a framework first. Start with the job: one narrow, valuable task you can describe precisely. Write twenty examples of good output for it. That is your first eval set, and it will tell you more about feasibility than any architecture diagram, including the ones above. Then get a prompt working, add retrieval when the model needs facts it does not have, add orchestration when one call is not enough, and measure at every step, because in a probabilistic system the only thing standing between "it works" and "it used to work" is the number you are watching.

If you are weighing whether to build your LLM architecture in-house or partner on it, that decision is mostly about people. Our guide to AI staffing covers the roles this work needs, what they cost in the US market, and when bringing in a team that has already shipped one beats a six-month search.