Tokenisation and lemmatisation: foundational NLP concepts to know
Tokenisation is the process of breaking raw text into smaller, manageable pieces called tokens, such as words or sub-words, so a machine learning model can process them. Lemmatisation is the subsequent step of reducing those tokens to their dictionary base form, or lemma, ensuring that different inflections of a word are treated as the same concept. Together, these techniques form the essential preprocessing pipeline for Natural Language Processing (NLP) systems.
When you build or evaluate an NLP model, you cannot simply feed it raw sentences. Computers require structured numerical input. Before we even get to vectorisation or word embeddings, we must standardise the text. Tokenisation slices the text up, and lemmatisation cleans those slices up by mapping variations like “running”, “ran”, and “runs” to the base word “run”. Understanding the mechanics of these two steps is essential for your university exams. Examiners expect you to know not just what these processes are, but exactly why they are necessary and how they behave when faced with unusual edge cases.
What tokenisation actually does
Fundamentally, tokenisation acts as the bridge between human-readable strings and machine-readable data. If you take the sentence “The cat sat on the mat”, a simple whitespace tokeniser splits this into six distinct word tokens. This sounds trivial, but real-world text is rarely this clean. You will encounter punctuation attached to words, contractions like “don’t”, and hyphenated phrases. A naive approach that just splits by spaces will treat “mat” and “mat.” as two completely different vocabulary terms. This inflates your vocabulary with redundant entries and makes the dataset unnecessarily sparse.
To solve this, practitioners use rule-based tokenisers or statistical models that understand the grammatical structure of the language. For example, the Penn Treebank tokenisation standard treats punctuation as separate tokens and splits clitics. Under this standard, “don’t” becomes “do” and “n’t”. This separation is vital because “n’t” carries the negation that flips the entire meaning of the sentence. If a model treats “don’t” as a single opaque block, it has to learn the concept of negation from scratch for every single contracted word in the English language.
Consider a short worked example you might see in a research paper. Take the string “Mr. Smith bought 2,000 shares.” A basic split yields “Mr.”, “Smith”, “bought”, “2,000”, “shares.”. A proper NLP tokeniser recognises “Mr.” as a single entity rather than an end-of-sentence marker, leaves “2,000” intact rather than splitting at the comma, and detaches the final full stop. The resulting tokens are “Mr.”, “Smith”, “bought”, “2,000”, “shares”, “.”. This structured output keeps the vocabulary size manageable and prevents the model from learning redundant representations.
Sub-word tokenisation: moving beyond basic words
While word-level tokenisation is intuitive, it creates major problems for modern neural networks. The most pressing issue is the out-of-vocabulary problem. If you train a model on ten million words, it will still inevitably encounter a word it has never seen before during inference. If it only knows whole words, it will map this unknown word to a generic unknown token, losing all semantic meaning in the process. Furthermore, highly agglutinative languages, which glue many morphemes together into single massive words, make word-level vocabularies impossibly large to store in memory.
To fix this, state-of-the-art models like BERT and GPT rely on sub-word tokenisation algorithms such as Byte-Pair Encoding (BPE) or WordPiece. These algorithms start with a vocabulary of individual characters and iteratively merge the most frequently co-occurring pairs into larger sub-words. Common words remain whole, while rare or completely novel words are broken down into familiar, meaningful chunks.
For instance, the word “unhappiness” might not be in the model’s vocabulary if it is relatively rare in the training data. A sub-word tokeniser splits it into “un”, “happi”, and “ness”. Because the model has seen “un” as a prefix and “ness” as a suffix in thousands of other contexts, it deduces the meaning of the unseen word. If an exam question asks how large language models handle unseen vocabulary, explaining sub-word tokenisation is exactly what the examiner expects.
Lemmatisation: finding the root meaning
Once text is tokenised, we often need to normalise it. Lemmatisation is the process of reducing a word to its base, dictionary form, which linguists call the lemma. In English, verbs change form depending on tense and nouns change depending on plurality. “Am”, “are”, and “is” all map to the lemma “be”. “Mice” maps to “mouse”. By applying lemmatisation, we group these variations together, telling the algorithm that they represent the exact same underlying concept.
This reduction is incredibly valuable for tasks like search engines, sentiment analysis, or topic modelling, where the core meaning matters far more than the grammatical tense. If a user searches for “computing”, they expect to see documents containing “compute”, “computes”, and “computed”. Without lemmatisation, your term frequency matrix treats these as entirely separate features, diluting the signal and requiring far more training data for the model to learn that they are related.
True lemmatisation requires a morphological analysis of the word. This means the algorithm needs to know the word’s part of speech before it can process it correctly. The word “saw” is a classic example of why this matters. If “saw” is used as a verb in a sentence like “I saw the dog”, its lemma is “see”. If it is used as a noun in a sentence like “I used a saw to cut wood”, its lemma remains “saw”. Modern lemmatisers use part-of-speech taggers under the hood to resolve these ambiguities. This makes them computationally heavier than simpler text normalisation methods, but highly accurate and dependable.
Stemming vs lemmatisation: a common exam trap
Examiners frequently test your understanding of the difference between stemming and lemmatisation. Students regularly mix them up because both techniques aim to reduce inflectional forms to a common base word. However, their mechanics and their outputs are entirely different. Stemming is a crude, heuristic process that simply chops off the ends of words based on a predefined list of common suffixes. It does not understand the grammatical context or the actual vocabulary of the language it is processing.
The Porter stemmer, a popular algorithm, looks at the word “caring”, strips the “ing”, and outputs “car”. It then looks at “cars”, strips the “s”, and also outputs “car”. The algorithm accidentally conflates empathy with an automobile. It also reduces “university” to “univers”, which is a non-word. Stemming is fast and computationally cheap, but it produces these non-words and makes frequent semantic errors.
Lemmatisation relies on a detailed dictionary, like WordNet, and morphological analysis. It correctly reduces “caring” to “care” and “cars” to “car”. In your exam, you should make this distinction perfectly clear. State that stemming uses suffix-stripping rules and often creates non-words, while lemmatisation uses dictionary lookups and part-of-speech tagging to return a valid base word. Knowing when to apply which technique is a core practitioner skill. Use stemming when processing speed is the absolute priority, and use lemmatisation when semantic accuracy is required.
How to answer this in an exam
When faced with a question about text preprocessing, mark schemes consistently reward students who link the technique back to the goal of reducing dimensionality and managing sparsity. Do not just define tokenisation and lemmatisation. Explain that a raw text dataset might contain 100,000 unique strings, but by tokenising punctuation away and
Frequently asked questions
What is the difference between tokenisation and lemmatisation? Tokenisation splits raw text into smaller pieces like words or sub-words so a machine can process them. Lemmatisation is the subsequent step of reducing those tokens to their dictionary base form to ensure variations of a word are treated as the same concept.
Why do practitioners separate contractions during tokenisation? Separating contractions isolates essential grammatical elements like negation. If a tokeniser splits “don’t” into “do” and “n’t”, the model does not have to learn the concept of negation from scratch for every single contracted word.
How do large language models handle words they have not seen in training? They use sub-word tokenisation algorithms to break completely novel words down into familiar, meaningful chunks. By splitting a rare word into a known prefix, root, and suffix, the model can deduce its underlying semantic meaning.
Why does true lemmatisation require a morphological analysis of the word? An algorithm must know a word’s part of speech because the correct base form depends on its context. For example, the word “saw” reduces to the lemma “see” when used as a verb, but remains “saw” if used as a noun.