How do large language models generate text?
Large language models generate text one token at a time by calculating the probability of the most likely next word based on the preceding context. They use a transformer architecture to process input text into numerical embeddings, apply self-attention to understand the relationships between words, and output a probability distribution for the next token. The model then samples from this distribution, appends the chosen token to the sequence, and repeats the process autoregressively.
When you sit down to revise for your machine learning exams, it is easy to get distracted by media narratives about artificial intelligence reasoning or thinking. As a practitioner and examiner, I need you to strip away the anthropomorphism. A language model is fundamentally a highly complex statistical engine performing sequence-to-sequence mapping. Your examiners are not looking for philosophical debates about consciousness. They want to see that you understand the linear algebra, the probability theory, and the specific architectural choices that allow these models to predict the next word with such high accuracy.
To secure top marks, you must be able to break the text generation process down into its component mathematical steps. You need to explain how raw text becomes numbers, how the model weights the importance of different words, how it scores potential future words, and how it finally selects the text the user sees.
Breaking text down into tokens and embeddings
Before a model can generate anything, it must first understand the prompt. Language models cannot read raw text. Instead, they rely on a process called tokenisation to chop sentences into smaller units called tokens. Modern models typically use algorithms like Byte Pair Encoding to build a vocabulary of sub-word units. For example, the word “unbelievable” might not exist as a single token. The tokeniser might split it into “un”, “believ”, and “able”. This approach allows the model to handle rare words and misspellings by assembling them from common building blocks.
Once the text is tokenised, each token is mapped to a high-dimensional continuous vector known as an embedding. You can think of this embedding space as a vast mathematical map where words with similar meanings are positioned close together. However, a standard embedding only represents the isolated meaning of a token. Because transformers process all tokens simultaneously rather than reading left to right, the model has no inherent way to know the order of the words.
To solve this, the architecture adds positional encodings to the token embeddings. These are mathematical patterns added to the vector values, giving the model a way to distinguish between “the dog chased the cat” and “the cat chased the dog”. In an exam, failing to mention positional encodings when discussing embeddings is a common way students drop marks. The combination of the semantic embedding and the positional encoding forms the foundational input that is passed up into the transformer layers.
Understanding context through self-attention
The defining feature of modern large language models is the transformer architecture, specifically the self-attention mechanism. When the sequence of embeddings enters a transformer block, self-attention allows the model to look at the surrounding words to build a context-rich representation for every single token. If the model is processing the word “bank”, self-attention helps it determine whether the surrounding tokens relate to a river or a financial institution.
Mechanically, this is achieved through three learned matrices: Query, Key, and Value. For every token, the model projects its embedding into a Query vector (what the token is looking for), a Key vector (what the token contains), and a Value vector (the actual semantic content). The model computes the dot product between the Query vector of the current token and the Key vectors of all other tokens in the sequence. A high dot product indicates a strong relationship. These scores are scaled and passed through a softmax function to create attention weights, which are then multiplied by the Value vectors to produce a final, contextualised embedding.
For text generation, models use a specific variant called masked self-attention. Because generation happens autoregressively (one step at a time), a token cannot be allowed to attend to words that appear after it in the sequence. Masking artificially sets the attention scores for future tokens to negative infinity before the softmax step. This ensures that when predicting the next word, the model only bases its calculations on the words that have already been generated. Mentioning the necessity of masked self-attention in decoder-only models is a highly rewarded point in university mark schemes.
Calculating probabilities and the softmax function
After the tokens have passed through multiple transformer blocks, the model reaches the final output layer. The goal here is to map the contextualised representation of the final token back into the vocabulary space. The model multiplies the final hidden state by an un-embedding matrix, resulting in a vector of raw, unnormalised scores called logits. There is one logit for every single token in the model’s entire vocabulary.
Because logits are raw numbers that can range from negative to positive infinity, they are not immediately useful for statistical sampling. The model must convert them into a valid probability distribution where all values sit between 0 and 1, and the entire set sums to exactly 1. It achieves this using the softmax function. The softmax function exponentiates each logit and divides it by the sum of all exponentiated logits.
Consider a short worked example. Imagine a simplified vocabulary of just three words: “mat”, “rug”, and “dog”. Given the input context “The cat sat on the”, the final layer might output logits of 5.0 for “mat”, 4.0 for “rug”, and -1.0 for “dog”. Applying the exponential function gives us roughly 148.4 for “mat”, 54.6 for “rug”, and 0.37 for “dog”. The sum of these values is 203.37. Dividing each exponential by the total sum yields probabilities of 73% for “mat”, 27% for “rug”, and 0.1% for “dog”. The model now has a clear statistical basis for choosing the next word.
Decoding strategies and temperature scaling
Generating the final text requires a decoding strategy to select a specific token from the probability distribution we just calculated. The simplest approach is greedy decoding, where the model simply selects the token with the highest probability. While this is computationally cheap, it often leads to repetitive and highly predictable text. For tasks requiring natural-sounding human language, models rely on probabilistic sampling instead.
To control the randomness of this sampling, practitioners apply a mathematical adjustment called temperature scaling. Temperature is a hyperparameter applied directly to the logits before the softmax function. The logits are divided by the temperature value. A temperature of 1.0 leaves the distribution unchanged. A temperature less than 1.0 makes the higher logits stand out even more, creating a sharper distribution and more deterministic text. A temperature greater than 1.0 flattens the distribution, giving lower-probability words a higher chance of being selected, which increases the perceived creativity of the output.
Even with temperature adjustments, sampling from the entire vocabulary can occasionally result in the model picking completely nonsensical words from the extreme tail of the distribution. To prevent this, models use truncation strategies like Top-k or Top-p sampling. Top-k sampling restricts the selection to the k most likely tokens. Top-p, also known as nucleus sampling, restricts the selection to the smallest set of tokens whose cumulative probability exceeds a threshold p (such as 0.9). Once a token is selected using these methods, it is appended to the input sequence, and the entire massive calculation begins again to predict the word after that.
How to answer this in an exam
When faced with an exam question asking you to explain text generation, your structure and precision dictate your grade. Mark schemes at the university level heavily penalise students who rely on vague metaphors about the AI “thinking” or “reading” the prompt. You must demonstrate mechanical understanding.
Start by defining the autoregressive nature of the generation process. Clearly state that the model processes the entire context to predict a single next token, appends it, and loops. Next, ensure you accurately define the difference between logits and probabilities, explicitly naming the softmax function as the bridge between the two. Examiners actively look for students who understand that the transformer outputs raw scores, not percentages.
Finally, demonstrate applied judgement by comparing decoding strategies. Explain why greedy decoding might be suitable for a deterministic task like writing code, whereas Top-p sampling with a moderate temperature is better for writing an essay. If you want to see exactly how these concepts are tested in practice, the Full Marks Press Natural Language Processing guide covers this topic extensively with complete worked exam questions and examiner commentary. By aligning your answers with the mathematical reality of the architecture, you will comfortably secure top marks.
Frequently asked questions
What is the difference between greedy decoding and nucleus sampling? Greedy decoding always selects the single token with the highest mathematical probability, which can cause repetitive text loops. Nucleus sampling (Top-p) calculates a dynamic pool of likely tokens whose probabilities sum to a target threshold, then randomly selects one token from that specific pool.
Why do language models hallucinate facts? Language models do not possess databases of true facts. They are trained purely to predict the most statistically probable next token based on their training data. If a sequence of factually incorrect tokens has a high probability score in a specific context, the model will generate it.
How does temperature affect text generation? Temperature is a scaling factor applied to the logits before the softmax calculation. A low temperature makes the probability distribution sharper (favouring highly probable words), while a high temperature flattens the distribution (increasing the chance of selecting less probable words).
What does autoregressive mean in machine learning? Autoregressive means that a model uses its own past outputs as inputs for future predictions. In text generation, the model predicts one token, adds that token to the original prompt, and then uses the new, longer sequence to predict the subsequent token.
Sotiris Spyrou is an AI practitioner and author for Full Marks Press. You can get a free digital copy of our latest exam-revision guides, including the Natural Language Processing module guide, at fullmarkspress.com/free.