Linear Algebra for ML: Vectors, Matrices & the Dot Product

Linear Algebra for Machine Learning

Every machine-learning model — from linear regression to a 100-billion-parameter neural network — is, underneath, linear algebra: numbers arranged in vectors and matrices, multiplied together. Once you can read and manipulate vectors and matrices, ML stops looking like magic and starts looking like arithmetic you can follow. This lesson gives you that fluency.

Data is vectors

A vector is an ordered list of numbers. One house, described by size = 1400, bedrooms = 3, age = 12, is a 3-dimensional vector:

$$\mathbf{x} = \begin{bmatrix} 1400 \\ 3 \\ 12 \end{bmatrix}$$

We write vectors in bold lowercase ($\mathbf{x}$) and their entries as $x_1, x_2, x_3$. In ML, one vector = one example (one house, one customer, one image flattened into pixels), and each entry is a feature.

Two operations you will use constantly:

• Scalar multiply — stretch every entry: $2\mathbf{x} = [2800, 6, 24]$.
• Addition — add entry-by-entry, and the two vectors must be the same length: $[1,2] + [3,4] = [4,6]$.

Geometrically, a 2-D vector is an arrow from the origin to a point. Adding vectors places them tip-to-tail; scaling changes their length.

The dot product: the single most important operation in ML

The dot product of two same-length vectors multiplies them entry-by-entry and sums the result:

$$\mathbf{a} \cdot \mathbf{b} = \sum_{i} a_i b_i$$

Worked example. With $\mathbf{a} = [2, 3, 5]$ and $\mathbf{b} = [1, 0, 4]$:

$$\mathbf{a}\cdot\mathbf{b} = (2)(1) + (3)(0) + (5)(4) = 2 + 0 + 20 = 22$$

Why it matters: a prediction from a linear model is a dot product. If your learned weights are $\mathbf{w} = [w_1, w_2, w_3]$ and an example is $\mathbf{x}$, the model''s raw prediction is

$$\hat{y} = \mathbf{w}\cdot\mathbf{x} = w_1x_1 + w_2x_2 + w_3x_3$$

— a weighted sum of the features. Every neuron in a neural network computes exactly this dot product before applying its activation. Learn the dot product and you understand the beating heart of ML.

The dot product also measures alignment: it is large and positive when two vectors point the same way, near zero when they are perpendicular, and negative when they oppose. That is why it shows up in similarity search, attention, and embeddings.

How big is a vector? Norms and distance

The L2 norm (Euclidean length) of a vector is the square root of its dot product with itself:

$$\|\mathbf{x}\|_2 = \sqrt{\mathbf{x}\cdot\mathbf{x}} = \sqrt{\sum_i x_i^2}$$

For $\mathbf{x} = [3, 4]$: $\|\mathbf{x}\|_2 = \sqrt{9 + 16} = \sqrt{25} = 5$. The distance between two points is the norm of their difference, $\|\mathbf{a}-\mathbf{b}\|_2$ — the engine behind k-nearest-neighbours and clustering.

A dataset is a matrix

Stack many example-vectors as rows and you get a matrix: a grid of numbers with a shape written (rows, columns). A dataset of 3 houses with 3 features each is a $3\times3$ matrix $X$ (written bold uppercase), where rows are examples and columns are features:

$$X = \begin{bmatrix} 1400 & 3 & 12 \\ 1600 & 4 & 5 \\ 900 & 2 & 30 \end{bmatrix}$$

Shape is everything in ML. When code breaks, it is almost always a shape mismatch — so always know the shape of what you are holding.

Matrix times vector = predictions for the whole dataset at once

Here is the payoff. To score all examples with one weight vector, multiply the data matrix by the weights. Matrix-vector multiplication takes the dot product of each row with the vector:

$$X\mathbf{w} = \begin{bmatrix} \text{row}_1 \cdot \mathbf{w} \\ \text{row}_2 \cdot \mathbf{w} \\ \text{row}_3 \cdot \mathbf{w} \end{bmatrix}$$

Worked example. With $\mathbf{w} = [0.1, 20, -2]$ and the $X$ above, the first prediction is $(1400)(0.1) + (3)(20) + (12)(-2) = 140 + 60 - 24 = 176$. The whole vector of predictions falls out of one operation. This is why ML is fast: libraries turn "loop over a million examples" into a single matrix multiply that your CPU or GPU runs in parallel — vectorization.

The dimension rule: an $(m \times n)$ matrix can only multiply an $(n \times k)$ matrix — the inner dimensions must match, and the result is $(m \times k)$. So $X$ of shape $(3\times3)$ times $\mathbf{w}$ of shape $(3\times1)$ gives predictions of shape $(3\times1)$. If the inner numbers disagree, the multiply is undefined — and that is the error you will meet most often.

Common pitfalls

• Shape mismatch. Multiplying $(3\times3)$ by $(2\times1)$ fails: inner dims $3 \ne 2$. Before any multiply, say the shapes out loud.
• Order matters. $AB \ne BA$ for matrices. Swapping operands usually changes the answer or breaks the dimensions.
• Elementwise is not the dot product. In NumPy, a * b multiplies entry-by-entry; a @ b (or np.dot) is the dot / matrix product. Confusing them is the classic beginner bug.
• Row vs column vectors. Shapes (3,), (3,1), and (1,3) behave differently in multiplication. Keep track of which you hold.

Why this is your on-ramp

You will not derive theorems here — you will recognise these shapes everywhere. A linear regression is $X\mathbf{w}$; a neural layer is $XW + \mathbf{b}$; an embedding lookup is a matrix slice; cosine similarity is a normalised dot product. Get comfortable now, and every later lesson reads like a sentence in a language you already speak.