What is the transformer architecture in NLP?
The transformer architecture in natural language processing is a neural network model that relies entirely on self-attention mechanisms to weigh the importance of different words in a sequence, dispensing completely with recurrence and convolutions. Introduced in 2017 by Google researchers, it allows for massive parallel processing of text, which drastically reduces training times compared to older sequence models. By processing entire sequences at once rather than word by word, transformers capture long-range dependencies in language far more effectively and serve as the foundational architecture for modern large language models.
Before the transformer arrived on the scene, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) were the standard for text processing. These older architectures processed data sequentially. If you wanted to translate a sentence, an RNN had to read the first word, update its hidden state, read the second word, and so on. This created a severe computational bottleneck. You could not speed up the process by throwing more computer processors at it because step two always had to wait for step one.
The sequential nature also meant that by the time the network reached the end of a long paragraph, it had essentially forgotten the beginning, leading to poor performance on long documents. The transformer solved both problems simultaneously. It looks at all words in a sequence at exactly the same time. This parallel approach changed the entire trajectory of artificial intelligence, allowing models to scale up to hundreds of billions of parameters.
The mechanism of self-attention
Self-attention is the mathematical heart of the transformer. When the model processes a given word, self-attention allows it to look at every other word in the input sequence to gather context. If you read the sentence “The animal did not cross the street because it was too tired”, your brain instantly knows “it” refers to the animal, not the street. Self-attention calculates association scores between “it” and every other word in the sentence, assigning the highest mathematical weight to the word “animal”.
To achieve this, the transformer creates three distinct vectors for each word: a Query, a Key, and a Value. Think of this like a retrieval system in a library database. The Query represents what the current word is looking for contextually. The Key represents what other words offer. The Value is the actual informational content of the word. The model computes a dot product between the Query of the current word and the Keys of all other words. This mathematical operation produces a raw score determining how much focus to place on those other words.
Once the raw scores are calculated, they are scaled down by dividing by the square root of the dimension of the key vectors. This scaling prevents the gradients from becoming vanishingly small during the training phase. The scaled scores are then passed through a softmax function, which normalises them into probabilities that sum to one. Finally, these probabilities are multiplied by the Value vectors. The result is a newly weighted representation of the original word, heavily influenced by the context of its surrounding words.
Multi-head attention and capturing different contexts
A single self-attention calculation provides one specific perspective on a sentence. However, human language is highly ambiguous and words relate to each other in multiple ways simultaneously. Two words might be grammatically linked as subject and verb, whilst also being semantically linked through tone, sentiment, or irony. To capture all these different relationships, the transformer architecture uses multi-head attention. Instead of computing a single set of Query, Key, and Value matrices, the model computes multiple parallel sets, known as heads.
Imagine a model configured with an eight-head attention layer. Head one might learn to pay attention to positional relationships, linking adjectives strictly to their nouns. Head two might focus entirely on identifying named entities like cities or prominent people. Head three might track pronoun references across a long paragraph. Each head performs the standard self-attention calculation independently, producing its own unique output vector. In a practical implementation, if the input embedding is 512 dimensions, an eight-head model will split this into eight 64-dimensional calculations, running them all concurrently.
After all heads have completed their calculations, the transformer concatenates their outputs back together into a single vector. This combined vector is then multiplied by an additional weight matrix to project it back to the original required dimension. By running multiple attention mechanisms in parallel, the architecture builds a deeply nuanced mathematical representation of the text, capturing syntax, semantics, and context simultaneously without adding significant computational overhead.
Positional encoding to maintain word order
Because the transformer processes all words at exactly the same time, it gains immense computational speed but loses something critical: the inherent order of the sequence. To a raw self-attention mechanism, the sentence “Alice chased the dog” and “The dog chased Alice” look mathematically identical because the set of input words is exactly the same. To fix this structural flaw, the architecture requires a way to inject information about the absolute and relative positions of the words in the sequence.
The original transformer paper solves this problem using positional encodings. These are specific vectors of the same dimension as the word embeddings, added directly to the word embeddings before they enter the first attention layer. Instead of just numbering words sequentially, the authors used a clever mathematical combination of sine and cosine functions of different frequencies. This specific pattern allows the model to easily learn to attend by relative positions, because for any fixed offset, the positional encoding can be represented as a linear function of a previous position.
While some modern implementations use learned positional embeddings rather than fixed mathematical formulas, the original trigonometric approach has a distinct advantage. It allows the model to extrapolate to sequence lengths longer than those encountered during training. If your model was only trained on sentences of fifty words, but it suddenly encounters a sentence of sixty words during inference, the sine and cosine functions predictably generate the correct positional vectors for those new slots. This ensures the model always understands where a word sits in relation to its neighbours.
The encoder-decoder structure
The original transformer was designed specifically for machine translation, which requires taking a sequence in one language and generating a sequence in another. To do this, it employs a two-part encoder-decoder structure. The encoder reads the entire input sequence, using self-attention to build a rich, context-aware mathematical representation of the source text. Once the encoder has processed the input, it passes this final contextual representation to the decoder.
The decoder is responsible for generating the output sequence, predicting one word at a time. It uses two distinct types of attention to achieve this. First, it uses masked self-attention on the words it has already generated. The mathematical masking prevents the decoder from “looking ahead” at future words, which would ruin the learning process during training. Second, it uses cross-attention. In the cross-attention layer, the Queries come from the decoder’s previous layer, but the Keys and Values come directly from the output of the encoder. This allows every generated word to look back at the original input text for translation cues.
In a university exam context, you must also recognise how this original architecture fractured into two distinct branches over time. Encoder-only models, like BERT, dropped the decoder entirely. They are designed for tasks requiring deep understanding of a whole text, such as sentiment analysis or document classification. Decoder-only models, like the GPT family, dropped the encoder. They are designed for generative tasks, predicting the next word in a sequence based only on the preceding context. Understanding this architectural split is critical for applying the right model to a specific machine learning problem.
How to answer this in an exam
When an exam question asks you to explain the transformer architecture, the mark scheme heavily rewards precise technical terminology. You must explicitly mention “self-attention”, the calculation of “Queries, Keys, and Values”, and the necessity of “positional encoding”. Examiners want to see that you understand exactly why this architecture replaced RNNs. You will secure higher marks by directly contrasting the sequential computational bottleneck of an LSTM with the parallel processing capability of the transformer.
A common differentiator for top-grade answers is discussing computational complexity. You should note that while self-attention allows for extreme parallelisation, it has a time and memory complexity of O(n squared) with respect to the sequence length. This means if you double the length of the input text, the computational cost quadruples. Demonstrating this mathematical knowledge shows you understand the practical limitations of the model, not just the high-level theory.
If asked to describe the architecture visually, be prepared to draw a high-level block diagram. Show the input embeddings, the addition of positional encodings, the multi-head attention block, the feed-forward network, and the residual connections with layer normalisation. We cover exactly how to structure these technical answers, complete with full mark scheme breakdowns, in the Full Marks Press Deep Learning and Generative AI guide.
Frequently asked questions
What is a residual connection in a transformer? A residual connection is a pathway that bypasses a specific layer in the neural network, adding the original input of that layer directly to its output. In transformers, these connections help prevent the vanishing gradient problem during training, allowing for the successful construction of very deep networks.
How does the feed-forward layer work within the architecture? After the multi-head attention mechanism, the data passes through a position-wise feed-forward network. This consists of two linear transformations with a ReLU activation function in between, applied independently and identically to each position, adding vital non-linearity to the model.
Why is layer normalisation used instead of batch normalisation? Layer normalisation computes the mean and variance across the features of a single sequence, entirely independent of other examples in the batch. This is highly beneficial for natural language processing, where sequence lengths vary wildly and batch sizes are often constrained by memory limits.
What does “autoregressive” mean in the context of transformers? Autoregressive means the model generates outputs one step at a time, using its own previously generated outputs as inputs for the next step. Transformer decoders operate autoregressively when generating text, masking future tokens so they only predict based on the past.
You can download a free sample chapter of our revision guides to aid your exam preparation at fullmarkspress.com/free.