Why embeddings find meaning: vectors from first principles

A customer writes "my parcel never showed up". Your help-centre article is titled "Missing delivery". The two sentences share no words at all, so a SQL LIKE '%parcel%' or a keyword search finds nothing. Yet a RAG (retrieval-augmented generation: look up relevant text, then give it to the LLM) system has to find that article, or the LLM will answer from guesswork.

Embeddings solve this. An embedding model turns a piece of text into a list of a few hundred numbers, and texts with similar meaning get similar lists. This article builds that idea from nothing: first by hand with a tiny matrix, then with a real model, so that "vectors capture meaning" stops being a slogan and becomes something you can predict.

Part I · Meaning as position

Chapter 1: A vector is a point in space

The problem

A vector here is just a fixed-length list of numbers, like a Java double[]. Its length is called its dimension. Why would a list of numbers say anything about meaning?

Details, built on the model

With 2 numbers we can see it directly. These coordinates are made up by hand to show the idea:

import numpy as np

pins = {
    "parcel missing":   np.array([0.9, 0.1]),
    "delivery late":    np.array([0.8, 0.3]),
    "reset password":   np.array([0.1, 0.9]),
}
query = np.array([0.85, 0.2])     # "where is my package?"

for text, v in pins.items():
    dist = np.linalg.norm(query - v)  # straight-line distance
    print(f"{text:16} distance {dist:.2f}")

parcel missing   distance 0.11
delivery late    distance 0.11
reset password   distance 1.03

Everything hard lives in one question: who decides the coordinates? The next chapter makes a computer decide them from text alone.

Chapter 2: Coordinates from the company a word keeps

The problem

Nobody labels millions of words with coordinates by hand. The trick is an old linguistics idea, the distributional hypothesis: words that appear in similar contexts have similar meanings. "Refund" and "reimburse" both show up near "money", "order", "back".

Details, built on the model

A co-occurrence matrix counts, for each pair of words, how often they appear in the same sentence. Here it is on a tiny support corpus:

import numpy as np
from itertools import combinations

docs = [
    "refund money back for order",
    "reimburse money back for order",
    "refund the payment to card",
    "reimburse the payment to card",
    "reset password for account",
    "change password for account login",
]
vocab = sorted({w for d in docs for w in d.split()})
idx = {w: i for i, w in enumerate(vocab)}
M = np.zeros((len(vocab), len(vocab)))
for d in docs:
    for a, b in combinations(set(d.split()), 2):
        M[idx[a], idx[b]] += 1
        M[idx[b], idx[a]] += 1

n = len(vocab)
print(f"{n} words, so each word is a {n}-dimensional vector")
print("refund   ", M[idx["refund"]].astype(int))
print("reimburse", M[idx["reimburse"]].astype(int))
print("password ", M[idx["password"]].astype(int))

15 words, so each word is a 15-dimensional vector
refund    [0 1 1 0 1 0 1 1 0 1 0 0 0 1 1]
reimburse [0 1 1 0 1 0 1 1 0 1 0 0 0 1 1]
password  [2 0 0 1 2 1 0 0 0 0 0 0 1 0 0]

"refund" and "reimburse" never appear in the same sentence, yet their rows are identical, because their neighbours are. "password" has a completely different row. That's meaning from context, with nothing but counting.

Two problems with raw rows follow from the model: they're as long as the vocabulary (real vocabularies have 50,000+ words), and mostly zeros. Real embedding models learn short, dense vectors (384 to 3,072 numbers) with a neural network trained on billions of sentences, but the principle is the same: you're placed by the company you keep.

Part II · Measuring nearness

Chapter 3: Cosine similarity is the angle between arrows

The problem

We need a number for "how near". Straight-line distance has a flaw: a long document mentioning "refund" 10 times gets a row 10 times bigger than a short one, so it looks far away despite meaning the same.

Details, built on the model

The formula is the dot product (multiply matching positions and add up) divided by both lengths:

cos(a, b) = (a · b) / (|a| × |b|)

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

r, rb = M[idx["refund"]], M[idx["reimburse"]]
p = M[idx["password"]]
print("refund vs reimburse:", round(cosine(r, rb), 2))
print("refund vs password :", round(cosine(r, p), 2))
print("refund vs 10*refund:", round(cosine(r, 10 * r), 2))

refund vs reimburse: 1.0
refund vs password : 0.21
refund vs 10*refund: 1.0

The last line is the model's promise: scaling an arrow doesn't change its direction, so similarity stays 1.0.

A useful consequence: if you normalise every vector to length 1 first, cosine similarity is just the dot product. Vector databases rely on this, because a dot product is one fast multiply-and-add.

def unit(v):
    return v / np.linalg.norm(v)

print(round(unit(r) @ unit(rb), 2), round(unit(r) @ unit(p), 2))

1.0 0.21

Chapter 4: A real embedding model

The problem

Counting words can't handle sentences ("not good" vs "good"), typos or words it never saw. Modern embedding models read the whole sentence with a transformer (the same architecture as LLMs) and output one vector for it. We'll use all-MiniLM-L6-v2, a small open model that outputs 384 numbers.

Details, built on the model

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2")
articles = [
    "Missing delivery: what to do if your order has not arrived",
    "How to request a refund for a damaged item",
    "Change your account password",
    "Delivery charges and free shipping thresholds",
]
A = model.encode(articles, normalize_embeddings=True)
print(A.shape)

def search(q):
    qv = model.encode([q], normalize_embeddings=True)[0]
    scores = A @ qv                    # cosine, via unit vectors
    for i in np.argsort(-scores)[:2]:
        print(f"  {scores[i]:.2f}  {articles[i]}")

for q in ["my parcel never showed up",
          "I want my money back, the box was crushed"]:
    print(q)
    search(q)

(4, 384)
my parcel never showed up
  0.57  Missing delivery: what to do if your order has not arrived
  0.30  How to request a refund for a damaged item
I want my money back, the box was crushed
  0.45  How to request a refund for a damaged item
  0.26  Missing delivery: what to do if your order has not arrived

"parcel never showed up" found "Missing delivery" with zero shared words. That is the whole point of embeddings. Notice the scores themselves are modest (0.57): what matters is the ranking, not the absolute number.

Chapter 5: What the map gets wrong

The problem

Embeddings fail in predictable ways, and the model tells you which.

Details, built on the model

pairs = [
    ("The refund was approved", "The refund was not approved"),
    ("Order ORD-48213 is late", "Order ORD-99107 is late"),
    ("Refund of 500 rupees", "Refund of 50,000 rupees"),
]
for a, b in pairs:
    va, vb = model.encode([a, b], normalize_embeddings=True)
    print(f"{va @ vb:.2f}  {a!r} vs {b!r}")

0.95  'The refund was approved' vs 'The refund was not approved'
0.92  'Order ORD-48213 is late' vs 'Order ORD-99107 is late'
0.83  'Refund of 500 rupees' vs 'Refund of 50,000 rupees'

All three pairs score as near-duplicates, though each pair differs on exactly the detail a support agent cares about. Two practical rules follow:

• Retrieval by embeddings finds candidates on the right topic. Exact matching (IDs, SKUs, error codes) needs keyword search alongside it. That's the subject of the next article.
• A similarity score isn't a probability or a "confidence". 0.8 doesn't mean "80% relevant", and scores aren't comparable across different models.

All the mental models, in one place