Large Language Models, Explained Without the Hype

What an LLM is, structurally

A large language model is a neural network with billions of internal parameters that has been trained to do one extremely narrow task: predict the next token in a sequence. A token is roughly a fragment of a word, around three or four characters on average. When you type a prompt into a chatbot, the system breaks your prompt into tokens, feeds them into the model, and the model returns a probability distribution over every token in its vocabulary describing what is likely to come next. The system picks one, appends it to the running sequence, and repeats. That loop, run a few hundred times, is what produces a paragraph of fluent text.

That is the whole trick at the lowest level. Everything else, every emergent capability, every failure mode, every debate about whether these systems "understand" anything, sits on top of next-token prediction trained at sufficient scale.

The transformer, briefly

The architecture that made modern LLMs possible is the transformer, introduced in a 2017 paper called *Attention Is All You Need* (Vaswani et al., 2017). The headline innovation was the attention mechanism, which lets the model weight every token in the input against every other token when deciding what to predict next. Before transformers, models processed text strictly left to right and struggled to keep track of context across long passages. Attention let a model look at the whole context at once and decide, dynamically, which earlier words mattered most to the word it was currently generating.

Stack enough attention layers, train on enough text, and the model starts to do things its designers did not explicitly program. It writes code. It translates between languages it was barely shown. It answers questions in domains it was never specifically trained for. None of that was hand-coded. All of it fell out of next-token prediction at scale.

How an LLM is actually trained

Training a frontier LLM happens in three phases.

The first is pretraining. The model is shown a few trillion tokens of text and asked, over and over, to predict the next token in each sequence. The training corpus typically includes a filtered slice of the public web, a large set of books, code repositories, scholarly articles, and licensed datasets. This phase costs tens of millions of dollars and consumes the electricity of a small town for weeks. By the end, the model can produce fluent text on essentially any topic represented in its training data.

The second is supervised fine-tuning. Pretrained models are good at completing text but bad at following instructions. To make them useful as assistants, engineers show them tens of thousands of high-quality examples of a prompt followed by an ideal response. This teaches the model the format of being helpful: answer the question, do not produce another question, stop when you are done.

The third is reinforcement learning from human feedback (RLHF) or its variants, formalised in *Training language models to follow instructions with human feedback* (Ouyang et al., 2022). Human annotators rank multiple model responses to the same prompt, and a smaller reward model is trained on those rankings. The main model is then nudged to produce outputs the reward model would rate highly. This is what makes a model refuse to write instructions for synthesising nerve agents, adopt a particular tone, or cite its sources. RLHF is also where most of the model's measurable bias is shaped, for better and for worse.

Context windows and what they really mean

The context window is the number of tokens the model can attend to at once. Early models had context windows of 2,000 tokens, about three pages of text. Frontier models in 2026 routinely handle a million tokens, the equivalent of a long novel or a year of internal email. This sounds magical but the practical reality is more limited than the marketing suggests. Performance degrades noticeably toward the end of very long contexts. Models are demonstrably better at finding information in the first and last sections of a long document than in the middle, a phenomenon researchers call the "lost in the middle" effect (Liu et al., 2023).

For our work this matters concretely. When we feed a Humanitarian Needs Overview to a model and ask for a summary, we deliberately chunk the document and prompt the model to summarise each chunk separately, then summarise the summaries. Trusting the model to ingest a 200-page PDF in one pass and produce an accurate condensed version is a recipe for the middle of the document being silently dropped.

Why LLMs hallucinate

Hallucination is the term of art for when a model produces a confident statement that is not supported by reality. It is not a bug that will be fixed in the next version. It is a direct consequence of how the model works.

An LLM does not have a database of facts. It has a statistical compression of the text it was trained on. When you ask it a question, it is sampling from the distribution of answers that look like answers to that question, weighted by the patterns in its training data. If the question is well-represented in training and has a stable answer, the sample will probably be correct. If the question is rare, time-sensitive, or has an answer the model never saw, the sample will still come out fluent and confident, because the model has no internal signal for the difference between "I am pattern-matching from solid ground" and "I am pattern-matching from nothing in particular."

There are partial defences. Retrieval augmented generation, where the model is forced to read a specific set of trusted documents before answering, reduces hallucination dramatically for any question whose answer is in the supplied documents. Chain-of-thought prompting, where the model is asked to reason step by step, improves accuracy on tasks that decompose well. Verifier models, which read the main model's output and flag inconsistencies, catch some errors. None of these techniques eliminate hallucination. They reduce its frequency and shift its character.

What LLMs are good at right now

The five capabilities that are reliably useful at production quality:

• Drafting structured prose from a clear brief. The model produces a first pass; a human edits and verifies. Faster than writing from scratch, slower than no-quality-control automation.
• Translating between widely spoken languages. Quality is high for major language pairs, weaker for under-resourced languages, and unreliable for dialects, idioms, and specialised vocabulary.
• Summarising long documents into shorter ones, with the caveats about long contexts above.
• Extracting structured information from unstructured text. Pull dates, locations, casualty figures, and actor names out of a news article into a clean schema. This works extremely well when the schema is tight and the documents are short.
• Writing and reviewing code. Frontier models in 2026 produce working code for routine tasks and competent code review on most pull requests. Production deployment without human review is still a bad idea.

What LLMs are bad at, and dangerous around

They are bad at arithmetic beyond the simplest cases, although tool use (giving the model a calculator) fixes most of this. They are bad at counting and at any task that requires tracking exact quantities through a long context. They are bad at admitting they do not know something, because their training rewards confident answers. They are bad at tasks where the right answer is rare in the training data, which means they are bad at frontier research questions and at any subject under-represented online, which in practice means most of the Global South and most of what humanitarian responders need to know.

They are dangerous around any task where a confident wrong answer will be acted on without review. The classic failure is the lawyer who filed a brief full of cases the model invented, but the humanitarian-sector equivalents are worse. A model that confidently states the wrong governorate for a displaced population, the wrong agency to contact in an emergency, or the wrong eligibility criteria for a protection programme is producing a fact that can route resources away from people who need them.

How we use LLMs on this site

Every AI-assisted page is built with retrieval augmented generation. The model is given a closed set of verified primary sources, asked to synthesise, and the output is then checked against those sources before publication. We do not let the model browse the web freely and stitch together a briefing. We do not let it answer from its pretraining memory on questions where the underlying figures change. We publish an inline citation for every numerical claim, so readers can verify the figure against the source we used. AI-assisted pages carry an amber disclosure banner so readers know what they are looking at. For the broader pattern this sits inside, see our foundations explainer on what AI actually is and the longer-form essays in our research and analysis library.

If you are evaluating an AI vendor for humanitarian use, the four questions to ask are: what is your source-grounding architecture, what is your hallucination rate on your reference benchmark, what is the human-review step before output reaches a user, and what is the path for correcting an error after it has been published. A vendor that cannot answer those is selling you a chatbot, not a humanitarian tool.

Where the technology is heading

Three trends are worth watching. First, smaller specialised models. A seven-billion-parameter model fine-tuned on a domain-specific corpus now outperforms much larger general models on its narrow task. This matters for humanitarian deployment because small models can run on field hardware, on a sanctioned country's local infrastructure, or behind a firewall without sending sensitive case data to a third-party server.

Second, agentic systems, where a model is allowed to plan a multi-step task, call tools, browse the web, and check its own work. Agents are powerful and brittle in roughly equal measure. They unlock genuinely new use cases and they fail in ways that are harder to debug because the failure can be six steps deep in a chain the model decided to run.

Third, regulation. The EU AI Act came into force in 2025 and is now shaping how high-risk AI systems can be deployed inside the bloc, including for migration and border management. Humanitarian organisations that operate inside Europe or that handle data on people who will eventually reach Europe need to read the Act, not the press coverage of it.

A short reading list

If you want the foundational paper, read *Attention Is All You Need* (Vaswani et al., 2017). If you want to understand how RLHF works in practice, read *Training language models to follow instructions with human feedback* (Ouyang et al., 2022). If you want a critical perspective on what these models can and cannot do, *On the Dangers of Stochastic Parrots* (Bender et al., 2021) is the most-cited starting point and remains relevant. None of those require advanced mathematics to follow.

The honest summary is that LLMs are a transformative tool that has been over-marketed and under-disciplined. Used inside a tight evidentiary loop, with primary sources at the centre and human review at the edges, they can do real work. Used as oracles, they manufacture plausible nonsense fluently enough that nobody catches it until it is in print.

Sources

• Vaswani et al., *Attention Is All You Need*, 2017. The transformer architecture paper underpinning every modern LLM.
• Ouyang et al., *Training language models to follow instructions with human feedback*, 2022. The reference paper for RLHF.
• Liu et al., *Lost in the Middle: How Language Models Use Long Contexts*, 2023. Empirical evidence for the long-context degradation discussed above.
• Bender, Gebru, McMillan-Major and Mitchell, *On the Dangers of Stochastic Parrots*, FAccT 2021. The most-cited critique of large-language-model scaling.
• Lewis et al., *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*, 2020. The original RAG paper.
• European Commission, EU AI Act. Official regulatory framework cited in the regulation section.