Understanding Random Embeddings

Looking for TL;DR? Skip to key takeaways. And feel free to DM me on X or LinkedIn if I got something wrong :)

I've always noticed that random vectors degrade vector search, but almost nobody (including me) truly understood why. So I decided to investigate with 1024-dimensional 10,000 random vs real embeddings (jina-embeddings-v3). Why this model? Because it has MRL support which forms reals embeddings at slices of the full embedding too. It was really eye opening and I've tried my best to make it intuitive!

The closest of all 10,000 sits at cosine 0.12. A random pair sits at 0.00. So the "nearest neighbor" is basically the least-far of 10,000 roughly-equidistant points — there's nothing special about it.

Now the same test with real jina embeddings over 10,000 sentences, same count, same 1024 dims: the nearest neighbor sits at 0.55, the median pair at just 0.10. Look at that for a second. A real embedding's nearest neighbor (0.55) towers over both its own typical pair (0.10) and random's best-of-10,000 (0.12). The whole distribution has a tail random barely has. That 0.12-vs-0.55 gap is what this whole post is about: why random vectors don't have it, what "structure" is as the cure, and what that structure actually buys you (mostly compute, it turns out, not recall).

What is an embedding and cosine similarity anyways?

Any set of extracted features can act like embedding dimensions: (height, weight, age) is a 3-dim embedding. Some features carry their own information; others are partly redundant, predictable from the rest (weight ≈ f(height)); some are basically independent. A good learned embedding leans hard on the redundant kind — it packs a lot of correlated structure into its numbers.

Now watch what cosine similarity does to this. To compare two embeddings you divide each by its length, pinning every vector onto the unit sphere x² + y² + z² + … = 1. That quietly does two things:

1. It throws away magnitude — only direction survives.
2. It gives every axis equal weight, treating them as independent, orthogonal directions — even the ones that were redundant. So two correlated coordinates (y ≈ f(x)) get double-counted: their shared signal sits in both, and the dot product adds it twice. Packing the real information into as few independent directions as possible (so that double-counting can't happen) turns out to be exactly what "structure" means later.

And the sphere is a constraint, not a free space: x² + y² + z² = 1 is a fixed budget — push one coordinate up and the rest must shrink; the axes can't all grow at once. Across a dataset it's exact: the total variance over all coordinates is pinned to 1.

The punchline: once you center each vector, cosine similarity is just the Pearson correlation between the two coordinate lists. "Are these embeddings similar?" is literally "are their coordinate patterns correlated?" — so the rest of this post is really a story about correlation: when it vanishes (random), when it survives (real neighbours), and what it costs to search for.

🎲 In random 1024-d, there is no neighbour

Why is a random vector's nearest neighbour so far away? Think of every other vector as casting a vote on yours — it can point with you, against you, or land in between, and the cosine is how strong that vote is. A vector is just a list of 1024 coordinates with random signs, and each coordinate casts its own little vote — with you or against you, about a coin flip either way. Across 1024 independent coordinates the withs and againsts almost perfectly cancel: the vote is neutralized, and every extra dimension only adds another chance to disagree, pinning the total (the cosine) ever closer to zero. The points aren't bunched together and aren't spread apart — it's just that nothing votes strongly for anything else. Geometrically, a neutralized vote is perpendicular (90°), and that's what breaks nearest-neighbour search — known since Beyer et al. (1999) asked when is "nearest neighbor" meaningful?. It's easiest to see from a single vector's point of view: put your point where the x-axis meets the circle (0°), then place every other vector at its true angle to it, so a dot's horizontal position is literally its cosine. Bump the dimension buttons from 2 up to 1024 and watch:

As d climbs, the yellow (random) cloud collapses onto the vertical 90° line — every other vector's vote gets neutralized. That's the cancellation counted up: a unit vector spreads its length across d axes at about 1/√d each, so the cosine of two of them sums d coin-flip votes that mostly cancel, landing near 0 with spread 1/√d. More dimensions, tighter to zero — the share of random pairs within ±0.1 of neutral climbs from 6.5% in 2D to 99.9% by 1024-d.

You can read that vote exactly: the share of coordinates voting with you is ½ + arcsin(cos)/π — 100% at 0° (fully with you), a 50/50 coin toss at neutral (90°), 0% at 180° (fully against). So "neutralized" is literal — at 90° a vector's coordinates split evenly for and against, and cancel to nothing. (That net vote, with minus against, is exactly Kendall's τ = (2/π)·arcsin(cos): the rank-correlation twin of the Pearson correlation from the intro. Cosine and the vote are two faces of the same correlation.)

The green real jina overlay is the twist I didn't expect. Its bulk sits near that same 90° line — most real sentence pairs are also near-neutral (that +0.11 nudge), and two sentences about different things (a company and a plant) genuinely should be. But it trails a warm tail of true nearest neighbours voting with you toward 0°: not one lucky hit but a whole neighbourhood (nearest ~0.55, even the 10th-nearest ~0.46, against a ~0.10 background), plus clusters (next section). Random has neither — its nearest (0.12) is barely above its median (0.00). There is no neighbourhood to find. (The real tail barely fades even truncated to a 32-dim slice.)

So searchability can't come from raw distance — in high dimensions there's barely any distance contrast left. It comes from the tail and the clusters you just saw leaning off the vertical. What that structure is takes up the rest of the post.

🌐 Haze vs continents, on a globe

So far this has been pairs of vectors and cosines. Here's the same conclusion as a picture you can spin. I run the exact same pipeline on both random and real vectors, so any difference is the data, not the method: cluster them into 14 groups with k-means, project to a 3D globe with UMAP (the same tool people use to map real embeddings), and colour by cluster.

k-means always hands you 14 groups, so the fair question is whether they mean anything. On random vectors they don't: the colours come out as confetti, scattered evenly across the globe with no coherent regions, at every dimension. There's nothing to pull together because every point is ~equidistant from every other.

Flip the toggle to real embeddings and the identical clustering does the opposite: it finds distinct continents. And they aren't arbitrary — the 10,000 sentences come from 14 topics (companies, athletes, animals, films…), and k-means recovers those topics almost exactly (cluster↔topic agreement 0.87). The continents even hold together truncated to a 32-dim Matryoshka slice (still 0.64). Same method, opposite result — the difference is entirely in the data.

The clusters differ in one more way: their sizes. Sort the 64 clusters biggest-to-smallest and the shapes couldn't be more different:

Random's line is almost flat: every cluster holds ~110–150 points, because k-means slices a uniform cloud into near-equal cells (Gini 0.04). Real sentence embeddings fall off a cliff: 296 down to 40, a ~7× spread (Gini 0.23), with some topics dense and others sparse. And real input tokens are lumpier still, 383 down to 35, an ~11× spread (Gini 0.29): the vocabulary's frequency structure, a few big groups and a long tail of small ones. Real embeddings, tokens or sentences, carve the sphere unevenly; random can't (Gini 0.04) because there's no skew to carry. (This is where the token vs sentence contrast lives now: on the pairwise-cosine histogram above, jina's tokens and sentences both sit near +0.11 and overlap, so I left tokens off that chart.)

That structure (real neighbourhoods versus structureless confetti) is the whole thing search runs on, and it's the geometry the rest of the post is about.

🧮 Anatomy of a cosine: clusters, centering, and hubs

A cosine between two unit vectors is just a sum: cos(a,b) = a₁b₁ + a₂b₂ + … + a₁₀₂₄b₁₀₂₄ — 1024 little products, added up. So everything this post has shown fits in one question: what do those 1024 products do differently for a random pair (cos 0.00), a different-topic pair (+0.10), a same-topic pair (+0.25), and a vector with its single closest neighbour out of all 10,000 (top-1, +0.55)? Watch them add up — these are real sentence pairs from the corpus, products summed left to right (↻ draws new ones):

Three things to see, all measured:

Every pair starts with the same raw material. Multiply any two of these vectors component by component and the products total ~0.65–0.75 in absolute size — random pairs included. The entire difference is cancellation. A random pair's products split 50/50 between positive and negative, so the walk wiggles inside the shaded band and ends at ~0 — only ~4% of the gross survives. A nearest-neighbour pair's products are ~67% positive: the walk climbs, and ~71% survives. A cosine doesn't measure how big two vectors are together — it measures how consistently their products point the same way.

A cluster is a set of sentences whose products keep pointing up. That's what training does, mechanically: it adjusts same-topic sentences to be large on the same components with the same signs, so their pairwise products come out positive and stack instead of cancelling. Nothing more is meant by "cluster" — and the clusters this recipe builds are the topics themselves: k-means on these vectors recovers the 14 DBpedia categories almost exactly (agreement 0.87). The ladder in the chart is cluster structure: the more components a pair shares with matching signs, the higher its walk climbs.

Part of every real walk is the same escalator. Click center vectors. All three real traces sag together (the different-topic pair collapses from +0.10 to ~+0.04) while random doesn't move. What got removed is the shared component: every real vector contains a copy of the corpus-average vector μ, and that copy adds ‖μ‖² ≈ +0.11 to every pair's sum, whatever the topics (that's the E[cos] = ‖μ‖² identity from the intro). A term that appears in every cosine distinguishes nothing — so deleting it (centering = subtracting the average vector, nothing more) barely changes search results (precision@10: 0.878 → 0.880). The component-level receipt: real pairs' products are 53.8% positive before centering and exactly 50.0% after. The +0.11 background was the shared component; the climb above it is the clusters.

The dashed verticals are jina's Matryoshka cut-points. A bonus of reading the cosine as a running sum: the walk's height at component k is exactly the (un-normalised) similarity of the two k-dim MRL slices. The first 32 components (3% of the vector) already hold 14–31% of a real pair's final cosine, and the first half holds ~70–80%: MRL training front-loads the similarity, which is why the truncated slices you saw on the globe still work. And since a real k-dim slice gets re-normalised at k dims, flip renormalize each slice to see the true slice-level cosine a k-dim index would compute: the nearest-neighbour trace sits near its final value from dim 32 onward, while a random pair swings inside a ±2/√k funnel that only narrows to ~0 at full width.

One paragraph of jargon-decoding, because every embedding paper will throw this word at you: anisotropy ("not the same in all directions") gets used for two different measurements of what you just watched. Sometimes it means the average pairwise cosine — which, as the centering click showed, measures only the shared component (≈ 0.11 here) and vanishes with it. Sometimes it means variance concentrated into few directions — which is the cluster structure itself (real vectors effectively use ~25 of their 1024 dimensions, random ~230), survives centering, and is what search runs on. Same word, two referents: one a removable constant, the other the signal. When a paper says "embeddings are anisotropic and that's bad," check which one it measured.

Hubs and orphans: clusters' odd side-effect

The stacking has one strange consequence. Ask every vector who's in my 10 nearest? and count how often each point gets named. Fair share is ~10 mentions each:

• Random: the most-named point is named just 2.6× its fair share, and nobody goes unnamed — but only because there are no real neighbours to hand out.
• Real jina: the biggest hub is named 11× its fair share, and ~1% of points are orphans — named by nobody.

Random is a tight bump that stops dead by ~26; real has a spike at 0 (the orphans) and a long right tail — its worst hubs run out to 107, far off the right edge here. Notice which cloud is lopsided: the structureless one is even, and it's the real, clustered embeddings that grow hubs. This is hubness, and it follows from the walk above:

• A point near the middle of its cluster shares many same-sign components with every cluster-mate, so its product-sums run above average with all of them at once — it lands in everyone's top-10. Measured: a point's named-count tracks the density of its neighbourhood (rank correlation 0.57).
• A point aligned with the shared component collects that +0.11 escalator with everyone on top (rank correlation 0.33).
• A fringe point whose components match nobody's strongly becomes an orphan — it never surfaces in anyone's results.

Centering removes the second cause but not the first: the worst hub drops 11× → 6× (orphans 1.2% → 0.9%), and the remaining 6× still dwarfs random's 2.6× — the clusters themselves, not the shared component, do most of the hub-making. That residue can't be removed, and you wouldn't want to: it is the structure.

The through-line: with no structure the products cancel and nothing is anyone's neighbour; with structure they stack — and a few well-placed points collect more than their share. What both facts cost a search index is next.

🔍 What that costs a search index

So what does this cost an actual index? HNSW finds nearest neighbors by greedily walking a graph: from some entry node, hop to whichever neighbor is closer to the query, repeat. That only works if there's a distance gradient to follow, i.e. each step has to get you meaningfully closer. Random vectors have no gradient (every direction is ~equidistant), so the walk wanders and the index has to explore far more candidates to compensate.

I built HNSW indexes (M=16, efConstruction=200) over both datasets and swept the search-effort knob efSearch, measuring recall@10 against exact brute force plus the distance comparisons per query — the actual work HNSW does, and a machine-independent stand-in for latency (wall-clock timings are too noisy to trust):

Real embeddings hit 0.90 recall by efSearch=32 (about 670 distance comparisons per query) and stay pinned there. Random never gets there: even at efSearch=256 (8× the effort, ~5,500 comparisons, 2× real's work) it tops out at 0.867. Note that this is the same index, same 1024 dims, same 10,000 vectors. The only thing that changed is whether the vectors have structure, and you can't efSearch your way out of geometry.

🧱 So what is structure, then?

"Structure" is a vague word, so let me pin it to something I can measure. I started from pure random vectors and injected structure three ways, then watched what each did to search. Each is a knob from "random" toward "real":

1. Low intrinsic dimension — generate the vectors inside an r-dimensional subspace, then embed that into 1024-d. Sweep r from 2 to 1024.
2. Spectral decay — keep all 1024 dimensions but scale them by a decaying spectrum (i^(-α/2)), so a few directions dominate. Sweep α. This is the anisotropy real embeddings are famous for.
3. Clusters — draw the vectors from a mixture of 64 centers and sweep how tight the clusters are.

To put all three knobs on a common axis I use intrinsic dimension: roughly, how many numbers you'd actually need to pin down a point, regardless of how many it's stored with. A sheet of paper crumpled inside a 3D room is still intrinsically 2D; random vectors are the opposite, using up nearly every dimension they're given. I estimate it with TwoNN, a standard intrinsic-dimension estimator, and the contrast is the key number here: random comes out ~230, real jina ~25. Plotting query latency against intrinsic dimension (each point is one knob setting; hover to see which; lower-left is cheaper) separates two distinct mechanisms:

Two mechanisms, and they're not the same:

• Lowering effective dimensionality (subspace rank or spectral decay) helps, and the gain tracks intrinsic dimension — both curves slide down-and-left together. This directly restores distance contrast: on a low-dimensional surface there are fewer independent coordinates to disagree on, so neighbors become meaningful again.
• Clustering helps through a different door. As clusters tighten, latency drops 4–5× (0.056 → 0.013ms) while the intrinsic dimension stays high (~145). The cluster points fall almost straight down, off the dimensionality trend. Think of the clusters as islands: the greedy walk ferries to the right island cheaply, then does the fine search locally. That's a different win from low dimensionality, and the intrinsic-dimension estimate can't even see it.

Real jina (the red star) has both: a low intrinsic dimension (~25, vs random's ~230) and cluster structure — which is why it sits where it does, far cheaper than random.

There's a second, quieter result hiding in this sweep. Structure mostly buys compute, not recall. Once any of these knobs is turned even slightly past "random," recall snaps to the ~0.90 HNSW ceiling and stays there; the dramatic 4–5× swings are all in latency. The recall gap is really just the one-time jump from random's 0.81 floor to the structured 0.90 ceiling — after that, structure is a compute story, not an accuracy one.

For search, isotropy is the enemy

Here's the part that cuts against standard advice. Random vectors are the most isotropic data you can have (they fill every dimension equally) and they're the worst to search. Every structured dataset above is less isotropic than random. So the usual NLP framing of "isotropy good, anisotropy bad" is about a different problem (cosine saturation in similarity scores); for nearest-neighbor search you actually want anisotropy, because variance piled into fewer directions is what gives the greedy walk a gradient to follow. (Which single number best measures "structure," and why the average-cosine anisotropy the field leaned on for years barely detects it, is a rabbit hole I save for the follow-up post.)

🚫 You can't bolt structure on at index time

If structure is what makes search cheap, can you just bolt it on after the embeddings are fixed, at index-build time, without touching the vectors (so recall stays put)? I tried the three obvious levers. All three came up empty, and the empty results are the interesting part.

Insertion order. HNSW is built incrementally, so the order you insert vectors changes the graph. I tried inserting cluster-by-cluster, cluster-medoids-first, and densest-points-first against a random baseline. The graph changes; the cost doesn't.

Smart entry points. HNSW spends hops descending its upper layers to find a good place to start the base-layer search. So I tried skipping that: route the query to its cluster first (compare to 64 centroids) and start the search right inside that cluster. It didn't help — and the reason is the useful part. A single fixed entry point with no hierarchy at all costs the same as the full hierarchy:

At a practical efSearch, the cost is dominated by the base-layer beam (the work of examining the ~efSearch candidates around the query to polish the answer), not by getting to the right neighborhood. A better starting point only saves the cheap descent. (This flips at very large N, where the descent grows ~log N and routing starts to matter — worth revisiting at 100M+ scale, but at 10k it's noise.)

The third lever, M and efConstruction, does move the numbers — but only by trading compute for recall along the usual curve, not by manufacturing a free speedup.

The throughline: none of the index-side tricks make search cheaper for free. The ~3× spread we keep seeing (random at ~1,880 distance comps vs real at ~1,080 vs tight clusters at ~570) comes from the data's own geometry, which is baked in when the embeddings are produced. An index can trade compute for recall; it can't put structure into vectors that don't have it.

📌 Key takeaways

• Random vectors have no neighbors. The nearest of 10,000 random 1024-d vectors sits at cosine 0.12 vs 0.00 for a random pair — barely a gap. Real embeddings have a fat tail: nearest ~0.55 against a ~0.10 typical pair. That tail is what nearest-neighbor search runs on.
• In 1024 dimensions, every vote is neutralized. A cosine is the net of d coin-flip per-coordinate votes (with/against) that cancel to ~0 (spread 1/√d) — the share voting "with" is exactly ½ + arcsin(cos)/π, i.e. a 50/50 toss at 90°. So two random vectors almost always land at neutral (perpendicular); by 1024-d, 99.9% of pairs are within ±0.1 of orthogonal, and "nearest" barely means anything.
• Random embeddings are expensive to search and cap out lower. Same index, same dims: real hits 0.90 recall at efSearch=32 (~670 distance comparisons); random never beats 0.867 even at 8× the effort (~5,500).
• "Structure" is two separate things. Low effective dimensionality (restores distance contrast; the gain tracks intrinsic dim) and clustering (separable basins; 4–5× cheaper search, independent of dimensionality). Real embeddings have both: intrinsic dim ~25 plus clusters.
• Structure mostly buys compute, not recall. Past the random→structured jump, recall pins at the HNSW ceiling; the big wins are all in latency.
• For search, isotropy is the enemy. Random is the most isotropic data and the worst to search. You want variance piled into fewer directions, not spread evenly.
• You can't add structure in the index. Insertion order and smart entry points give nothing at this scale (the base-layer beam dominates, not the descent); M/efConstruction only trade compute for recall. The geometry is set when the embeddings are made; the index can't fake it.

🔭 Closing thoughts

The thing I keep coming back to: structure isn't something the index provides, it's the compute the encoder already did, frozen into the geometry. A trained embedding model spends a forward pass folding a sentence onto a low-dimensional, clustered surface, and the index just cashes that check at query time. Random vectors never had a forward pass, so there's nothing to cash, and no amount of efSearch or clever insertion order can invent it after the fact.

Which raises the obvious next question: if structure is just cached compute, can we manufacture it directly, without training a model? Can you write down a generator that produces vectors as searchable as real ones? I tried, and the short version is: you can match every aggregate statistic of real embeddings and still fail a shape test, until you copy the geometry locally. That's the follow-up post — plus a look at what a model's own 24 layers are each doing to the geometry, from the raw word-embedding table to the final sentence vector.

Thanks for reading! Feel free to DM me on X or LinkedIn if you spot a mistake or want to nerd out about this.