Infrastructure
Embedding
Also known as: vector representation
A list of numbers that captures the meaning of a piece of text (or an image), so that similar meanings end up close together.
An embedding turns a word, sentence or document into a list of numbers that represents its meaning. The clever part: things that mean similar things get similar numbers, so "dog" and "puppy" land close together, while "dog" and "invoice" are far apart. This is what lets AI search by meaning instead of exact words, and it is the backbone of AI-powered search and recommendations.
An embedding is a numeric representation, a vector, of a piece of content's meaning. A model converts text (or an image, or audio) into a fixed-length list of numbers, positioned in a "meaning space" so that semantically similar items are near each other and unrelated ones are far apart.
Why they are useful
- Semantic search: find results by meaning, not just matching keywords.
- RAG: documents and questions are embedded so the system can fetch the most relevant passages.
- Clustering and recommendation: group similar items or suggest related ones.
Closeness is usually measured with cosine similarity, essentially the angle between two vectors.
An embedding is a mapping of discrete or high-dimensional input into a dense, continuous vector space in which geometric proximity encodes semantic or functional similarity. The idea traces to distributional semantics and was popularised for words by word2vec (Mikolov et al. 2013); modern text embeddings are produced by transformer encoders that yield contextual, sentence- or document-level vectors, typically compared via cosine similarity or dot product.
Inside an LLM, each token id is first mapped to a learned embedding before entering the network, so embeddings are both an internal representation and a standalone product. Operationally they power vector search: embeddings are indexed in a vector database and queried with approximate nearest-neighbour algorithms, the retrieval substrate of RAG. Key considerations are the embedding model's dimensionality and domain fit, normalisation, and the distinction between symmetric (similarity) and asymmetric (query-document) use cases.