Large language models (LLMs) demystified

A large language model (LLM) is a neural network trained on very large text corpora to predict the next token in a sequence. That one objective, guessing what comes next, is enough to produce models that summarize, translate, write code, answer questions, and follow instructions. Almost every AI chat product you have used is an LLM doing exactly that, one token at a time.

How an LLM produces text

The model converts your prompt into tokens, runs them through a transformer, and outputs a probability distribution over the next token. It samples one token, appends it, and repeats until it hits a stop signal. Everything the model "knows" lives in its weights plus whatever you put in the prompt. It has no live view of your data unless you hand it tools or context.

Here's the part people miss: there is no lookup, no database, no reasoning engine underneath. Fluency is a side effect of scale. Train a big enough network on enough text and next-token prediction alone starts to look like it can reason, plan, and explain. It cannot, not really, but the output is often good enough that the distinction stops mattering for practical work.

Base vs instruction-tuned models

• Base models are trained purely to predict text. They complete patterns well but do not reliably follow instructions, so a raw base model is awkward to talk to.
• Instruction-tuned models take a base model further, training it on instruction and response pairs plus human feedback until it behaves like an assistant. This is the version you actually ship against.

Almost every product-facing model you touch through an API is instruction-tuned. The base model is the raw capability; the instruction tuning is the layer that makes it usable. Worth knowing the difference, because "why won't the base model answer my question" is a real question people hit when they grab a model off an open-weight repo.

Tokens and context windows

Two numbers shape most of what you can build: how the model chunks text, and how much of it the model can hold at once.

A token is the unit a model reads and writes. In English, one token runs about three-quarters of a word, so a 4,000-word document is roughly 5,000 tokens. Non-English text and code tokenize differently, usually less efficiently. The context window is the hard ceiling on tokens the model can attend to in a single request, prompt and response combined.

As of 2026, frontier models from OpenAI, Anthropic, and Google commonly offer context windows on the order of hundreds of thousands of tokens, with top models reaching around a million. A handful of specialized systems push well past that. Open-weight models you self-host tend to sit lower, often in the 64K to 256K range. Bigger windows let you drop whole documents into the prompt, but they cost more and run slower, and models still lose track of detail buried in the middle of a very long context.

What LLMs are good and bad at

Good at: drafting and rewriting text, summarizing long inputs, translating, generating and explaining code, extracting structure from messy input, and answering questions when you supply the source material. Anything where fluent language plus broad pattern knowledge does the job, an LLM handles well.

Bad at: exact arithmetic, anything needing current facts it wasn't trained on, and tasks that demand a guarantee of correctness. It will produce fluent, confident text even when it is wrong, and it has no reliable sense of its own uncertainty. For high-stakes work, treat the output as a strong first draft that a human or a second system checks, not as a final answer.

The honest framing: an LLM is a fast, knowledgeable, slightly unreliable assistant. Design around the unreliability instead of pretending it isn't there, and you will build something that holds up.

How teams actually deploy LLMs

Most production systems start with the smallest intervention that works and add machinery only when they need it.

• Prompting first. Careful instructions, examples, and a good system prompt cover a surprising share of use cases without touching model weights. It is the fastest thing to change, so it is where you start.
• Retrieval-augmented generation. Pull relevant passages from your own data and inject them into the prompt so the model answers from current, grounded material. Choose retrieval-augmented-generation when the challenge is access to a large or changing knowledge base.
• Tool and function calling. Give the model a schema of tools, such as search, code execution, or database queries, and it returns structured calls instead of plain text. This is the foundation of ai-agents that orchestrate real systems.
• Fine-tuning. Adjust the weights on task-specific data when you need a stable behavior prompting can't enforce, like a fixed format or a narrow classification task at volume. Reach for it last, because it is the most expensive to build and change.

Rule of thumb worth remembering: retrieval adds knowledge, fine-tuning changes behavior. Plenty of real systems use all four at once, a frontier model with retrieval and tools, plus a small fine-tune for one niche task.

Limits you design around

• Hallucination. The model can state plausible, confident falsehoods, especially about specifics outside its training data. Ground answers in retrieved sources and verify anything high-stakes.
• Knowledge cutoff. Weights freeze at training time, so the model doesn't know events after its cutoff unless you feed it fresh context through a prompt or a tool.
• Cost and latency at scale. Big contexts, long outputs, and multi-tool workflows get expensive and slow. Teams route simple work to cheaper models and cache aggressively to keep the bill sane.
• Prompt injection. Untrusted text in a document or web page can smuggle in instructions that hijack the model. Any system that reads outside content and can act on it needs input filtering and output checks.

None of these are fatal. They are the reason production LLM work is real engineering, not a prompt in a text box. LLMs sit under most modern AI features, including generative-ai products and retrieval systems, and the teams that ship well are the ones that respect the failure modes and build guardrails around them.