Embeddings and Vector Space

Lesson 5 turned text into a sequence of integer token IDs. Those integers still carry no meaning on their own. This lesson covers the next step: mapping each token into a point in a continuous, high-dimensional space, where the actual geometry, the distances and angles between points, encodes something about meaning.

Two pieces of text that mean the same thing, even if they share no words at all, land near each other in embedding space. Two pieces of text about entirely different topics land at roughly right angles to each other. That is not a metaphor. It is a measurable geometric fact about how these vectors get trained.

This lesson explains why dense vectors replaced sparse one-hot encodings, verifies the real research behind word2vec’s famous analogy example, works through cosine similarity and Euclidean distance with actual numbers, covers the dimensionality tradeoffs behind real embedding models, and shows why a raw transformer’s internal hidden states are not the same thing as a retrieval embedding.

Section 01From one-hot vectors to dense embeddings

Before continuous representations, natural language processing relied on one-hot encoding. Given a vocabulary V, a specific token w gets mapped to a sparse vector where exactly one coordinate, the one corresponding to that word’s position in the vocabulary, equals 1, and every other coordinate equals 0.

This scheme has two structural problems. First, the curse of dimensionality: the vector’s length has to equal the entire vocabulary size, so memory and computation scale linearly with how many distinct words exist, producing enormous, mostly-empty vectors. Second, and more damaging: the dot product of any two distinct one-hot vectors is always exactly zero. Every word is geometrically orthogonal to every other word. “Cat” and “feline” are exactly as unrelated to each other, as far as the vector math is concerned, as “cat” and “microprocessor.” One-hot encoding cannot represent similarity at all, only identity.

Dense embeddings fix this by mapping tokens into a much lower-dimensional continuous space, typically somewhere between 100 and a few thousand dimensions, instead of a space as large as the vocabulary. Instead of reserving one dimension per word, dense embeddings use distributed representations: every concept is spread across all available dimensions at once, and every dimension contributes to representing multiple different semantic properties simultaneously. Geometrically, this turns the embedding space into a continuous metric space, where semantic properties become geometric properties. Words that share meaning or context land close together. Specific, consistent semantic shifts, such as changing verb tense or grammatical gender, tend to show up as a similar, repeatable vector offset no matter which specific words are involved.

Section 02Word2vec and the analogy vectors: what the research actually showed

The shift to dense representations accelerated with Mikolov et al.’s 2013 paper, “Efficient Estimation of Word Representations in Vector Space,” introducing the word2vec family of models. Word2vec deliberately stripped out the expensive non-linear hidden layers used in earlier neural language models, replacing them with two simpler, log-linear architectures built for fast training on huge amounts of unlabeled text.

• Continuous Bag-of-Words (CBOW): predicts a target center word, given the words surrounding it in a sliding window.
• Continuous Skip-gram: predicts the surrounding context words, given a single target center word.

By removing the non-linearity, these architectures cut the computational cost per training token substantially, which is exactly what made training on web-scale text corpora practical in the first place.

The analogy identity, verified against its actual source

A frequently repeated claim about word2vec is the identity v_king − v_man + v_woman ≈ v_queen. This exact example is widely cited in secondary sources, but the precise mathematical formulation and rigorous evaluation methodology were formally established in Mikolov, Yih, and Zweig (2013), “Linguistic Regularities in Continuous Space Word Representations,” and expanded in Mikolov et al. (2013), “Distributed Representations of Words and Phrases and their Compositionality.”

Mechanically, to answer an analogy of the form “a is to b as c is to ?”, the method computes an algebraic vector offset:

The resulting coordinate y usually does not land exactly on any real word’s vector. The method searches the vocabulary for whichever word w* maximizes cosine similarity to y, using the ratio of the dot product to the product of the vector norms, explicitly excluding a, b, and c themselves from the search so the model cannot trivially answer with one of the input words:

Levy and Goldberg (2014), in “Neural Word Embedding as Implicit Matrix Factorization,” showed why this arithmetic works at all: Skip-gram with Negative Sampling implicitly factorizes a shifted Pointwise Mutual Information (PMI) matrix built from word-context co-occurrence statistics. Because pointwise mutual information preserves the log-odds of co-occurrence, subtracting and adding vectors this way isolates a directional axis in the space that corresponds to a real underlying semantic attribute, such as “royalty” or “gender,” rather than being an arithmetic coincidence.

Where static embeddings fall apart

A single word2vec vector is fixed and invariant: every occurrence of a given string in the training corpus contributes to exactly one vector. This means a genuinely ambiguous word, like “bank” in a financial sense versus a riverbank, gets collapsed into a single vector representing an unweighted average of every context it ever appeared in during training. Modern architectures solve this with contextualized representations, covered in Section 5, which produce a different vector for the same word depending on the sentence it appears in.

Section 03Measuring closeness: cosine similarity vs. Euclidean distance

Given two vectors, there are two standard ways to measure how close they are.

Dot product (inner product): u · v = Σᵢ uᵢvᵢ

Euclidean distance (the L2 norm of the vector difference): d(u, v) = ‖u − v‖₂ = √(Σᵢ(uᵢ − vᵢ)²)

Cosine similarity (the cosine of the angle θ between the two vectors): cos(θ) = (u · v) / (‖u‖₂‖v‖₂)

Cosine similarity is bounded between −1 and 1, and measures pure directional alignment. It stays completely unaffected by how long either vector is, only by which direction each one points.

A worked example, including why magnitude matters

Take u = [3, 4] and v = [4, 3].

Dot product: u · v = (3 × 4) + (4 × 3) = 12 + 12 = 24

Norms: ‖u‖₂ = √(3² + 4²) = √(9 + 16) = √25 = 5, and ‖v‖₂ = √(4² + 3²) = √(16 + 9) = √25 = 5

Cosine similarity: cos(θ) = 24 / (5 × 5) = 24/25 = 0.96

Euclidean distance: d(u, v) = √((3−4)² + (4−3)²) = √((−1)² + 1²) = √2 ≈ 1.4142

Now take v’ = [8, 6], which points in exactly the same direction as v, just twice as long (‖v’‖₂ = 10). Recomputing both metrics between u and v’:

Dot product: u · v’ = (3 × 8) + (4 × 6) = 24 + 24 = 48

Cosine similarity: cos(θ’) = 48 / (5 × 10) = 48/50 = 0.96, exactly unchanged.

Euclidean distance: d(u, v’) = √((3−8)² + (4−6)²) = √((−5)² + (−2)²) = √29 ≈ 5.3852, more than triple what it was against v.

Even though v’ points in the identical direction as v, its Euclidean distance from u jumped from 1.4142 to 5.3852, purely because v’ is longer. Cosine similarity, by contrast, did not move at all, because it only cares about direction.

Why search systems use cosine similarity

In text embedding systems, a vector’s magnitude often reflects something superficial, like document length or how many times a common word appears, rather than which topic the text is actually about. Unnormalized Euclidean distance penalizes a long document even when it is conceptually right on target for a search query.

When vectors are normalized to unit length before storage, so ‖u‖₂ = ‖v‖₂ = 1, an exact algebraic relationship connects the two metrics:

Under this unit-norm constraint, maximizing cosine similarity is mathematically identical to minimizing squared Euclidean distance. This identity is why vector search engines, such as Milvus, Qdrant, and FAISS, can run fast inner-product operations directly on hardware accelerators while still guaranteeing a ranking that is invariant to vector magnitude.

Section 04Dimensionality: real models and the storage tradeoff

Real embedding models span a wide range of dimensions, chosen as a deliberate tradeoff between how much a vector can express and how much it costs to store and search.

(Model names and dimensions are a dated snapshot as of this document’s sourcing; specific commercial offerings change over time.)

Higher dimensionality gives the space more room to represent fine-grained distinctions with lower risk of unrelated concepts colliding. But it also scales storage and compute linearly: Memory overhead = N × d × S_bytes, where N is the number of stored vectors, d is the dimension, and S_bytes is the byte size per number, commonly 4 bytes for single-precision floats. A million vectors at d = 3072, stored as raw single-precision floats, requires 10⁶ × 3072 × 4 bytes ≈ 12.288 GB of RAM, before even accounting for the extra memory overhead that graph-based approximate nearest-neighbor indexes, such as HNSW, add on top.

Matryoshka Representation Learning: one model, several usable sizes

To avoid forcing a hard choice between expressive power and index size, modern encoders increasingly use Matryoshka Representation Learning (MRL), introduced by Kusupati et al. (2022). Standard training spreads information roughly evenly across every dimension of the output vector. MRL instead structures the representation hierarchically on purpose: it trains a single encoder backbone with several prediction heads attached to nested prefixes of the output vector, so that the first 64, 256, or 512 dimensions on their own already capture the core semantic signal, and later dimensions add progressively finer detail. The training loss sums the individual loss at each of these nested prefix lengths:

where f(x; Θ)_{1:m} denotes just the first m dimensions of the model’s output vector, and c_m weights how much each prefix length contributes to the total loss. Models trained this way, including OpenAI’s text-embedding-3-large, let an engineer truncate a stored vector down to a shorter prefix with simple array slicing and still get a meaningful, usable embedding. This supports a practical two-stage retrieval pattern: use short, cheap vector prefixes to quickly filter down a huge candidate set, then re-rank that smaller shortlist using the full, high-dimensional vectors for precision.

Section 05Why a transformer’s internal hidden states are not a retrieval embedding

There is an important structural difference between the token-by-token hidden states a transformer produces internally, covered in Lesson 4, and a single vector meant to represent an entire sentence or document for search.

When a sequence of tokens passes through an L-layer transformer encoder, every layer produces its own hidden state, one vector per token, not one vector for the whole input. These raw intermediate representations have two properties that make them poor retrieval embeddings on their own.

Granularity. They are token-level: a matrix with one row per token, not a single vector summarizing the whole sequence.

Anisotropy. Unfine-tuned intermediate hidden states tend to cluster tightly inside a narrow cone in the high-dimensional space, rather than spreading out isotropically in every direction. This directional clustering artificially inflates the baseline cosine similarity between essentially unrelated sentences, since almost everything ends up pointing in a similar rough direction regardless of actual meaning.

Turning many token vectors into one sequence vector: pooling

Converting a token-level matrix into a single sequence-level vector requires a pooling operation.

• Mean pooling: average every non-padded token’s vector together, z_mean = (1 / Σᵢmᵢ) Σᵢ mᵢhᵢ, where m is a binary mask marking which positions are real tokens versus padding. This is the pooling strategy used by Sentence-BERT and the E5 model series.
• [CLS] token pooling: simply use the hidden state of a special classification token inserted at the start of the sequence, z_CLS = h_[CLS].
• Last-token pooling: use the hidden state of the final token in the sequence. This is the standard approach when adapting an autoregressive decoder model, such as Llama or Mistral, to produce embeddings.

Contrastive fine-tuning: what actually fixes the anisotropy problem

Pooling alone does not solve the anisotropy problem. Reimers and Gurevych (2019), in “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” showed that applying standard pooling directly to an untuned BERT model’s outputs produced weak semantic search results, often worse than the much older, purely static GloVe word vectors. Their fix was to fine-tune a Siamese or Triplet network architecture using a contrastive objective. A common version of this is the InfoNCE loss:

q is the query embedding, d⁺ is a genuinely relevant document vector, the dⱼ⁻ terms are irrelevant negative document vectors, sim denotes cosine similarity, and τ is a temperature hyperparameter controlling how sharply the loss penalizes near-misses. This objective explicitly pulls matched query-document pairs closer together in the space and pushes unmatched pairs apart. That direct pressure is what reshapes the representation space from the tight, anisotropic cone of a raw transformer into a properly spread-out, isotropic space that cosine similarity search can actually rely on.

Section 06A worked sentence-similarity example

Take three sentences:

• Sentence A: “The quick brown fox jumps over the lazy dog.”
• Sentence B: “A fast auburn canine leaps above the sleepy hound.”
• Sentence C: “The central bank raised benchmark interest rates.”

(The following vectors are illustrative, built in a simplified 5-dimensional space to demonstrate the metric calculations cleanly, not real model output.)

Assume five conceptual axes: d₁ = fauna/canine concepts, d₂ = speed/agility, d₃ = leaping/vertical motion, d₄ = financial institutions, d₅ = monetary policy. The unit-normalized illustrative vectors:

All three are already unit-normalized (each vector’s own norm rounds to approximately 1), so cosine similarity reduces directly to a dot product.

Similarity between A and B (paraphrased meaning, zero shared tokens):

cos(θ_A,B) = v_A · v_B = (0.58 × 0.56) + (0.55 × 0.58) + (0.60 × 0.59) + (0 × 0) + (0 × 0) = 0.3248 + 0.3190 + 0.3540 = 0.9978

Despite sharing no exact words at all (“fox” vs. “canine,” “quick” vs. “fast,” “jumps” vs. “leaps,” “dog” vs. “hound”), the two sentences land at a cosine similarity of nearly 1, because a properly contrastively-trained encoder maps paraphrased meaning to aligned directions in the space, not matching surface tokens.

Similarity between A and C (unrelated domains):

cos(θ_A,C) = v_A · v_C = (0.58 × 0) + (0.55 × 0) + (0.60 × 0) + (0 × 0.70) + (0 × 0.714) = 0.00

Sentence C’s active coordinates sit entirely on the financial-institution and monetary-policy axes, which are exactly zero in Sentence A’s vector, and vice versa. The result is a dot product of exactly zero: the two sentences are geometrically orthogonal, matching their complete lack of shared subject matter.

Conclusion

Every property covered in this lesson traces back to the same underlying claim: meaning becomes distance and direction once text is embedded. One-hot vectors could only represent identity, never similarity, because every word sat at a fixed right angle to every other word. Dense embeddings fix that by distributing meaning across every dimension, and word2vec’s analogy arithmetic works because Skip-gram’s training objective implicitly factorizes real co-occurrence statistics, not because vector subtraction is inherently meaningful. Cosine similarity, not raw Euclidean distance, is the standard metric for search precisely because it ignores magnitude and measures only direction, and that same property is what makes fast, magnitude-invariant vector search possible at scale. Matryoshka Representation Learning lets a single model serve multiple size-cost tradeoffs at once, and contrastive fine-tuning is the specific step that turns a transformer’s raw, anisotropic internal hidden states into the well-calibrated, isotropic vectors that retrieval systems actually depend on.

The next lesson turns to something this lesson’s own worked examples already relied on informally: what happens when a model produces confident, fluent output that turns out to be false, and why the same next-token training objective from Lesson 6 does not, by itself, optimize for truthfulness.