Word embeddings explained: Word2Vec and GloVe for university students

Word embeddings are dense, low-dimensional vector representations of words where words with similar meanings are placed close together in a mathematical space. They capture the semantic and syntactic relationships of language by learning from vast amounts of text data, allowing machine learning models to process human language mathematically. Before these techniques emerged, natural language processing relied on sparse representations that treated every single word as a completely isolated entity.

By translating words into numbers based on the contexts in which they frequently appear, modern algorithms can understand synonyms, analogies, and grammatical structures without requiring manual programming or extensive rule-based systems. This shift from discrete symbols to distributed representations forms the very foundation of contemporary natural language processing. Understanding how these embeddings are generated, specifically through algorithms like Word2Vec and GloVe, is a core requirement for any university-level artificial intelligence exam.

Overcoming the limitations of sparse representations

To appreciate why word embeddings are revolutionary, you must first understand the problem they solve. Historically, text was converted into numbers using one-hot encoding. If you have a tiny vocabulary of just five words (apple, banana, car, dog, engine), you assign each word a unique position in a five-dimensional vector. “Apple” becomes [1, 0, 0, 0, 0], and “banana” becomes [0, 1, 0, 0, 0].

This creates two massive problems for machine learning models. The first is computational inefficiency. A real-world language model needs a vocabulary of at least 100,000 words. Using one-hot encoding, every single word is represented by a 100,000-dimensional vector containing a single one and 99,999 zeros. This creates sparse matrices that consume vast amounts of memory and are incredibly slow to process.

The second, more significant problem is mathematical isolation. If you calculate the dot product of the one-hot vectors for “apple” and “banana”, the result is zero. Mathematically, the vectors are orthogonal. As far as the computer is concerned, “apple” is just as unrelated to “banana” as it is to “engine”. There is no encoded concept of semantic similarity. Dense embeddings solve this by forcing the entire vocabulary into a much smaller, fixed number of dimensions, typically between 100 and 300. Every word is represented by continuous floating-point numbers across all dimensions, allowing the model to position related words near each other in the vector space.

Word2Vec: Learning meaning from local context

Developed by researchers at Google, Word2Vec was a major breakthrough that popularised dense embeddings. It operates purely on the distributional hypothesis. This linguistic theory states that words appearing in similar contexts tend to have similar meanings. Word2Vec trains a shallow neural network to predict words based on their neighbours. Crucially, we do not actually care about the final predictions of this network. We only care about the weights the network learns in its hidden layer, which become our final word embeddings.

Word2Vec comes in two distinct architectural flavours: Continuous Bag-of-Words (CBOW) and Skip-gram. CBOW takes a surrounding context of words and attempts to predict the target word in the middle. Skip-gram flips this entirely. It takes a single target word and attempts to predict the surrounding context words. Skip-gram tends to perform better with infrequent words and smaller datasets, while CBOW is computationally faster.

Let us look at a concrete worked example using the Skip-gram architecture. Imagine our training corpus contains the sentence: “The quick brown fox jumps over the lazy dog.” We set a context window size of two words. If our current target word is “fox”, our context words are the two words before it (“quick”, “brown”) and the two words after it (“jumps”, “over”). The Skip-gram neural network takes the input vector for “fox” and adjusts its internal weights to maximise the probability of outputting “quick”, “brown”, “jumps”, and “over”. As the algorithm slides this window across millions of sentences, the hidden layer weights are constantly adjusted. Eventually, words that frequently share context words end up with highly similar weight configurations.

GloVe: Capturing global statistical information

While Word2Vec is a predictive model that learns by sliding a local window across a text, GloVe takes an entirely different mathematical approach. GloVe stands for Global Vectors for Word Representation, developed by researchers at Stanford University. Instead of treating training as a local prediction task, GloVe is a count-based model that relies on matrix factorisation. It seeks to capture the global statistical information of the entire training corpus rather than just local contexts.

The GloVe algorithm begins by constructing a gigantic word-word co-occurrence matrix. Imagine a matrix where both the rows and columns represent your entire vocabulary. Every time word A appears near word B in your text corpus, you add a tally to that specific cell in the matrix. Over a corpus of billions of words, this matrix records exactly how many times every single word appeared in the vicinity of every other word.

GloVe then uses matrix factorisation to compress this massive, sparse matrix into a lower-dimensional, dense vector space. The objective function of GloVe is designed so that the dot product of two word vectors equals the logarithm of their probability of co-occurrence. For example, the word “ice” will co-occur frequently with “solid” and “water”, but rarely with “steam” or “gas”. GloVe explicitly encodes these global frequency ratios into the vectors. The result is a set of embeddings that captures semantic relationships just as well as Word2Vec, but relies entirely on global counting statistics rather than local neural network predictions.

Measuring semantic relationships with vector mathematics

Once you have trained your word embeddings, you need a way to measure the similarity between them. In standard geometry, we often use Euclidean distance to measure the straight-line distance between two points. However, in natural language processing, Euclidean distance is highly vulnerable to differences in word frequency. A common word and a rare word might have similar meanings but wildly different vector magnitudes.

Instead, practitioners use cosine similarity. This metric measures the angle between two vectors, completely ignoring their magnitude. If two word vectors point in exactly the same direction, their cosine similarity is 1. If they are completely orthogonal, the similarity is 0. If they point in opposite directions, the similarity is -1. This allows the model to correctly identify that “huge” and “gigantic” are highly similar concepts, regardless of how often they appeared in the training data.

This vector space also allows for remarkable algebraic operations that demonstrate the model has learned semantic relationships. The most famous example is vector analogy. If you take the vector for “King”, subtract the vector for “Man”, and add the vector for “Woman”, the resulting mathematical point in space is closest to the vector for “Queen”. The operation isolates the concept of royalty and applies it to a female concept. Showing that you understand this algebraic property is a superb way to demonstrate deep comprehension of the topic.

How to answer this in an exam

When university examiners set questions on word embeddings, they are primarily testing your ability to contrast different approaches and explain the underlying mathematical intuition. Marks are almost always awarded for clearly defining the difference between sparse representations (like one-hot encoding) and dense representations. You should explicitly mention that dense vectors solve the curse of dimensionality and capture semantic similarity.

If asked to compare Word2Vec and GloVe, the specific phrasing examiners look for is that Word2Vec is a “predictive model” based on “local context windows”, whereas GloVe is a “count-based model” relying on “global matrix factorisation”. Make sure you can name both the CBOW and Skip-gram architectures, and briefly explain how their input and output targets differ. Naming the distributional hypothesis will frequently secure a bonus mark.

Finally, ensure you understand how to evaluate these models. Mark schemes reward students who mention cosine similarity as the preferred distance metric for text, and those who can provide a concrete example of vector mathematics capturing semantic relationships. The Full Marks Press Natural Language Processing guide covers this topic in extensive detail with several worked exam questions and annotated mark schemes to help you structure your exact answers.

Frequently asked questions

What is the distributional hypothesis? It is a theory from linguistics stating that words occurring in similar contexts tend to have similar meanings. This hypothesis forms the fundamental theoretical basis for both Word2Vec and GloVe, as they rely on surrounding context to calculate vector values.

Why is cosine similarity used instead of Euclidean distance? Cosine similarity measures the angle between two vectors, ignoring their magnitude (length). In text data, magnitude is often heavily influenced by word frequency, so measuring the angle provides a much more accurate reflection of pure semantic similarity.

What is the difference between CBOW and Skip-gram? They are the two architectures of Word2Vec. Continuous Bag-of-Words takes the surrounding context words as input to predict a single target word. Skip-gram takes a single target word as input and attempts to predict the surrounding context words.

Can Word2Vec and GloVe handle words with multiple meanings? No, they cannot. Both algorithms assign a single, static vector to every word in the vocabulary. The word “bank” will have the same vector whether the text discusses a financial institution or the side of a river. This limitation eventually led to the development of dynamic, contextual embeddings like BERT.

You can claim a free digital copy of our practitioner-written exam revision guides by visiting fullmarkspress.com/free.