Embeddings: How Machines Represent Meaning

Computers are good with numbers and bad with meaning. A spreadsheet can sort prices, but it has no idea that "sofa" and "couch" refer to the same thing. Embeddings close that gap by turning words, sentences, images, products, or users into lists of numbers arranged so that similar things end up close together.

Once you understand embeddings, a lot of modern software makes more sense: semantic search, "customers also viewed," duplicate detection, and many classification systems all rely on the same idea.

What an embedding is

An embedding is a vector, meaning a fixed-length list of numbers, that represents an item. A sentence might become a list of several hundred numbers. A product might become a list of a few dozen. The length is called the dimension of the embedding.

The individual numbers usually have no human-readable meaning. You cannot point at position 47 and say "this measures formality." What matters is the geometry: the position of each vector relative to others.

The defining property is simple:

Items that are similar in the ways that matter for the task get vectors that are close together. Items that are different get vectors that are far apart.

From one-hot vectors to learned vectors

To see why embeddings help, consider the naive alternative.

The one-hot problem

Suppose your vocabulary has 50,000 words. A one-hot representation gives each word a vector of 50,000 zeros with a single 1 in its own position. "Sofa" has a 1 at position 12,003. "Couch" has a 1 at position 9,871.

This works for counting, but every pair of words is equally different. "Sofa" is as far from "couch" as it is from "volcano." The representation carries no notion of similarity, and it is huge and sparse.

Dense, learned representations

An embedding replaces that with a short, dense vector learned from data. The learning process pushes "sofa" and "couch" toward similar vectors because they appear in similar contexts: near "cushion," "living room," "sit." The idea behind this is old and intuitive: you can tell a lot about a word by the company it keeps.

The same logic extends beyond words:

• Sentences and documents that express similar ideas get nearby vectors even when they share few words.
• Images with similar visual content cluster together.
• Products bought or viewed by the same people end up near each other.
• Users with similar behavior share a neighborhood.

How embeddings are learned

An embedding is always a byproduct of some training objective. The objective decides which kind of similarity the space captures.

Prediction from context

Early word embedding methods trained a small model to predict a word from its neighbors, or neighbors from a word. The weights the model learned for each word became its embedding.

Contrastive training

Many modern embedding models are trained with pairs. The model sees pairs that should match, such as a question and its answer, a photo and its caption, or two paraphrases, and pairs that should not. Training pulls matching pairs together and pushes non-matching pairs apart. This produces spaces that work well for retrieval.

Byproducts of larger models

Neural networks trained for other tasks build internal representations along the way. Take the activations from a late layer of an image classifier or a language model, and you have an embedding. These are often useful out of the box, though models trained specifically for embedding tend to do better at similarity tasks.

Collaborative signals

In recommendation systems, embeddings for users and items can be learned purely from interaction data. If a user's vector and an item's vector point in similar directions, the model predicts the user will like the item. No text or images are required.

Why the objective matters

Imagine two embedding models for customer reviews. One was trained to group reviews by topic. The other was trained to group them by sentiment. "The battery died after a week" and "The battery lasts forever" would be close in the first space and far apart in the second. Neither is wrong. They answer different questions, so choose a model whose training resembles your use case.

Measuring similarity

With items as vectors, similarity becomes arithmetic.

Cosine similarity

The most common measure is cosine similarity, which looks at the angle between two vectors and ignores their lengths. It ranges from -1 (opposite directions) to 1 (same direction). Scores near 1 mean the items are very similar.

Dot product

The dot product multiplies matching positions and sums them. It reflects both direction and length. Some models are trained so that vector length carries information, like popularity or confidence, and the dot product uses it. If vectors are normalized to length 1, the dot product and cosine similarity give identical rankings.

Euclidean distance

Euclidean distance is the straight-line distance between two points. Smaller means more similar. For normalized vectors, it ranks results the same way as cosine similarity.

The practical rule: use the measure the embedding model was trained with. The model's documentation usually says which one.

import numpy as np

def cosine(a, b):
    # Cosine similarity is undefined for a zero vector: the norms below would
    # make the denominator zero and NumPy would return NaN. Real embeddings are
    # never all zeros, but padding rows and empty inputs sometimes are.
    norms = np.linalg.norm(a) * np.linalg.norm(b)
    if norms == 0:
        raise ValueError("cosine similarity needs two non-zero vectors")
    return np.dot(a, b) / norms

Scores are relative, not absolute

A cosine similarity of 0.8 does not mean "80 percent similar," and a threshold that works for one model can be meaningless for another. Different models spread their vectors differently. Calibrate thresholds on examples from your own data.

Where embeddings are used

Semantic search

Keyword search matches words. Semantic search matches meaning. You embed every document once and store the vectors. At query time, you embed the query and return documents whose vectors are nearest.

Imagine a help center where a user types "how do I get my money back." A keyword engine might miss an article titled "Requesting a refund." A semantic engine finds it, because the two phrases land near each other in the embedding space.

In practice, many teams combine both approaches. Keyword matching handles exact terms like product codes, error numbers, and names, which embeddings sometimes blur. Semantic matching handles paraphrases. Merging the two rankings is a common and effective pattern.

Recommendations

Embeddings let a system answer "what is similar to this?" quickly:

• Item-to-item: show products whose vectors are near the one being viewed.
• User-to-item: represent a user as a vector, perhaps an average of items they engaged with, and find the nearest items.
• Cold start: a new product with no interaction history can still get a content-based embedding from its description and images, so it can be recommended from day one.

Clustering and topic discovery

Run a clustering algorithm on embeddings of support tickets, survey answers, or reviews, and you get groups of semantically related text. A product manager can read a handful of examples from each cluster to see what customers are talking about without reading everything. This is far more effective on embeddings than on raw word counts, because paraphrases fall into the same cluster.

Deduplication and near-duplicate detection

Items whose vectors are extremely close are likely duplicates or near-duplicates: the same product listed twice with different titles, the same question asked in slightly different words, or re-uploaded images with minor edits.

Classification with few labels

Embeddings make strong features. Train a simple classifier, such as logistic regression, on top of text embeddings, and you often get a solid model from a relatively small labeled set. The embedding model already did the heavy lifting of understanding language.

Anomaly detection

Items whose vectors are far from all their neighbors are unusual. That can flag odd transactions, off-topic content, or data entry errors.

Storing and searching vectors at scale

Comparing a query against every stored vector is fine for thousands of items. For millions, it becomes slow.

Approximate nearest neighbor (ANN) indexes solve this by organizing vectors so that a search only examines a small, promising portion of the collection. They trade a small amount of accuracy, occasionally missing a true nearest neighbor, for a large speedup. Common approaches include graph-based indexes, which hop between neighboring vectors toward the query, and partition-based indexes, which group vectors into regions and only search the nearest regions.

You can use ANN indexes through dedicated vector databases, through vector extensions in general-purpose databases, or through standalone libraries. For many teams, adding vector search to a database they already run is the simplest path.

Practical decisions and pitfalls

Choosing a model

• Match the domain. A general-purpose text model may handle legal contracts or medical notes less well than one trained or fine-tuned on similar text. Test with your own examples.
• Match the language. If your content spans several languages, use a multilingual model so that equivalent sentences in different languages land near each other.
• Balance dimension and cost. Higher-dimensional vectors can capture more nuance but take more storage and slow down search. Some models support truncating vectors to shorter lengths with modest quality loss.

Keep everything in one space

Vectors from different models, or even different versions of the same model, are not comparable. If you upgrade the embedding model, re-embed your whole collection. Mixing old and new vectors silently degrades results.

Mind the input length

Most text embedding models have a maximum input length. Long documents are usually split into chunks, and each chunk gets its own vector. How you chunk affects what can be found: chunks that are too long blur several topics into one vector, and chunks that are too short lose context.

Evaluate on real queries

Build a small test set: realistic queries paired with the items that should come back. Measure how often the right items appear near the top. This catches problems that general benchmarks will not, such as a model that ignores your internal product names.

Remember what embeddings encode

Embeddings reflect patterns in their training data, including stereotypes and skewed associations. For systems that affect people, such as matching candidates to jobs, check whether similarity scores differ in unwanted ways across groups.

Key takeaways

• An embedding is a dense vector that places items so that similar ones sit close together; the geometry carries meaning, not individual numbers.
• The training objective determines what "similar" means, so choose a model whose training matches your task.
• Cosine similarity, dot product, and Euclidean distance are the standard similarity measures; use the one the model was built for and calibrate thresholds on your own data.
• Embeddings power semantic search, recommendations, clustering, deduplication, anomaly detection, and label-efficient classification.
• At scale, approximate nearest neighbor indexes make vector search fast at a small cost in accuracy.
• Never mix vectors from different models, and re-embed everything when you switch.