Matrix multiplication explained simply for AI students
Matrix multiplication is the mathematical operation of taking the rows of a first matrix and calculating the dot product with the columns of a second matrix to produce a new set of values. In artificial intelligence, it acts as the primary engine for processing data, allowing neural networks to apply thousands of weights to input features simultaneously rather than calculating them one by one. This operation transforms input data through sequential layers to produce predictions efficiently.
When you revise for your machine learning exams, you will notice that almost every algorithm eventually boils down to linear algebra. Neural networks, support vector machines and principal component analysis all depend on matrix operations. Students often memorise the formulas without understanding what the operation actually does to the underlying data. If you understand the mechanics, you can debug your code faster and pick up easy marks on theory papers.
The operation is essentially a massive system of simultaneous equations written in a highly compressed format. Instead of writing out hundreds of separate algebraic formulas, you pack all your variables into a single grid. When a neural network learns, it is simply updating the numbers inside these grids.
The rules of dimension matching
Before you can multiply two matrices, you must check their dimensions. We describe a matrix by its number of rows followed by its number of columns. A matrix with three rows and two columns is a 3 by 2 matrix. The fundamental rule of matrix multiplication is that the inner dimensions must match exactly. If you want to multiply Matrix A by Matrix B, the number of columns in A must equal the number of rows in B. If A is a 3 by 2 matrix, B must be a 2 by something matrix. If these numbers do not align, the operation is mathematically impossible.
When the inner dimensions match, the outer dimensions tell you the exact size of your final answer. Multiplying a 3 by 2 matrix with a 2 by 4 matrix will result in a brand new 3 by 4 matrix. Examiners frequently test this concept with multiple-choice questions or short-answer questions asking you to identify the dimensions of a hidden weight matrix. If you know your input size and your desired output size, you can instantly deduce the exact dimensions the weight matrix must have.
The actual calculation relies on the dot product. You take the first row of your first matrix and the first column of your second matrix. You multiply the corresponding elements together and sum the results. This single number becomes the very first element in the top-left corner of your new output matrix. You then slide across to the second column of the second matrix, repeat the multiplication and addition, and place the result in the next position. You continue this row-by-column sweeping pattern until every row from the first matrix has been paired with every column from the second matrix.
In the context of machine learning, this row-by-column action has a specific physical meaning. The row often represents a single data point, such as the pixels of an image or the financial features of a house. The column represents the weights connecting those features to a specific neuron in the next layer. The dot product condenses all that input information and its relative importance into a single activation signal for that specific neuron.
A concrete worked example
Let us walk through a small mathematical example to solidify the mechanics. Imagine you have a matrix A which is 2 by 2, and a matrix B which is also 2 by 2. Matrix A represents a batch of two data points with two features each. Let the top row of A be the numbers 1 and 2, and the bottom row be 3 and 4. Matrix B represents your network weights. Let the top row of B be 5 and 6, and the bottom row be 7 and 8. Because both are 2 by 2, the inner dimensions match, and the output will also be a 2 by 2 matrix.
To find the top-left number of our new matrix, we isolate the top row of A (1 and 2) and the left column of B (5 and 7). We multiply the first items together to get 5, and the second items together to get 14. Adding these gives 19. This number 19 is the final computed value for the top-left position of the output matrix. It represents the combined signal of the first data point passing through the first set of weights.
Next, we calculate the top-right position. We keep the top row of A (1 and 2) but move to the right column of B (6 and 8). Multiplying the corresponding parts gives 6 and 16. Summing them yields 22. Moving to the bottom row of our output, we take the bottom row of A (3 and 4) and the left column of B (5 and 7). This calculation is 15 plus 28, which equals 43. Finally, the bottom-right position pairs the bottom row of A (3 and 4) with the right column of B (6 and 8), giving 18 plus 32, which totals 50.
Your final output matrix reads 19 and 22 on the top row, and 43 and 50 on the bottom row. While this looks like simple arithmetic, reproducing it flawlessly under exam conditions requires practice. A single arithmetic slip early in the calculation ruins the final matrix. When you sit your exam, write down the pairings explicitly before you calculate the sums. This habit prevents careless errors and guarantees that you pick up method marks even if your final addition is slightly off.
Why artificial intelligence relies entirely on matrix multiplication
When you build a standard feed-forward neural network, you pass input data through multiple hidden layers. If you have an input layer with 784 neurons connected to a hidden layer with 128 neurons, you are dealing with over 100,000 individual weight connections. Writing a standard software loop to calculate each connection sequentially would take an impractical amount of time. Matrix multiplication solves this by structuring the entire layer’s calculations into a single mathematical step.
Hardware accelerators like Graphics Processing Units are designed specifically to exploit this mathematical structure. A standard computer processor handles tasks sequentially, calculating one dot product at a time. A graphics card, however, contains thousands of smaller cores that can perform simple arithmetic simultaneously. Because the calculation for the top-left element of an output matrix is completely independent of the calculation for the bottom-right element, a graphics card can compute all the elements at the exact same time. This parallelisation is the primary reason deep learning became practical over the last decade.
Furthermore, matrix multiplication allows networks to process data in batches rather than one example at a time. In our worked example, the input matrix had two rows, meaning it processed two data points simultaneously against the same weight matrix. In real-world training, you might pass batches of 32, 64 or 128 examples through the network at once. The weight matrix stays exactly the same, but the input matrix simply gains more rows. The output matrix grows proportionally, providing predictions for the entire batch in a fraction of a second. This batch processing stabilises the learning algorithm and significantly speeds up training times.
How to answer this in an exam
University examiners testing matrix multiplication in an AI context are usually looking for three specific things. Firstly, they want to see that you understand the dimension rules. You will frequently face questions that provide the input shape and the output shape, asking you to define the shape of the weight matrix. State the rule clearly on your paper: inner dimensions must match, outer dimensions dictate the result. Writing this out shows the examiner you are not just guessing numbers. The Full Marks Press Mathematics for AI guide covers this with worked exam questions, showing you exactly how marks are awarded for these deduction steps.
Secondly, examiners will test your understanding of non-commutativity. In standard arithmetic, multiplying three by four is the same as multiplying four by three. Matrix multiplication does not work this way. Multiplying Matrix A by Matrix B will almost always produce a completely different result to multiplying Matrix B by Matrix A, assuming the dimensions even allow the reverse operation. If an exam question asks you to write out the forward propagation equation, the order of the matrices is critical. Placing the weight matrix and the input vector in the wrong order will cost you the entire mark for that specific question.
Finally, always show your working for manual calculations. If a question asks you to perform a small matrix multiplication by hand, the final correct answer is usually only worth one mark. The majority of the marks are allocated to the method. Write down the explicit dot product calculations for at least the first two elements. If the pressure of the exam hall causes you to state that five times seven is thirty-six, the examiner can still award you the method marks because they can see you knew exactly how the rows and columns were supposed to interact.
Frequently asked questions
Does the order of matrix multiplication matter? Yes, the order is absolutely critical. Matrix multiplication is not commutative. Multiplying Matrix A by Matrix B yields a completely different result to multiplying Matrix B by Matrix A, and reversing the order often results in a dimension mismatch that makes the calculation mathematically impossible.
What is the difference between element-wise multiplication and matrix multiplication? Element-wise multiplication takes two matrices of the exact same size and multiplies the numbers in the matching positions together. Matrix multiplication is a completely different operation involving the dot products of rows and columns, and it usually involves matrices of different shapes.
How do biases fit into matrix multiplication for neural networks? After you multiply the input matrix by the weight matrix, you add a bias vector to the result. This addition is performed element-wise. The matrix multiplication scales the inputs, while the bias shifts the resulting values up or down before they pass into the activation function.
Why do we use matrices instead of for-loops? Matrices allow us to express thousands of calculations as a single operation. This mathematical structure allows parallel processing hardware like graphics cards to compute the entire layer simultaneously, reducing training time from weeks down to hours.
You can download a free copy of our exam-revision guides at fullmarkspress.com/free.