Turning text into vectors · Building with Language Models

If language models are the first big idea of modern AI, embeddings are the second — and they solve an entirely different problem. Where LLMs generate text, embeddings represent it. An embedding turns a piece of text into a list of numbers such that similar meanings end up as similar numbers. This unlocks searching, clustering, and recommendation over text at scales that keyword search cannot approach. This section explains what an embedding is, why it matters, and how to compute one.

The idea, in one paragraph

An embedding is a numeric representation of meaning. Feed the text "नमस्ते" (Nepali) into an embedding model, and out comes a fixed-size list of numbers — typically 768, 1024, or 3072 dimensions. Feed in "Hello" (English), and you get a different list of numbers, but numerically close to the first one — because the model has been trained to know these two greetings mean similar things.

The critical property: similar meanings produce similar vectors. Different meanings produce different vectors. Numerical “closeness” (measured by cosine similarity) equals semantic closeness.

That’s it. Everything downstream — semantic search, recommendation, clustering, RAG — is built on that one property.

Why this changes what’s possible

Traditional keyword search fails in obvious ways:

Searching for "heart attack" misses documents about "cardiac arrest" even though they mean the same thing.
Searching for "नेपाली भाषा" misses "the Nepali language" — different scripts, same meaning.
Searching for "cheap smartphone" doesn’t find "budget mobile phone".

Embeddings solve all three at once. Because the embeddings of "heart attack" and "cardiac arrest" are numerically close, a semantic search over documents will surface both. Same for Nepali/English versions of the same idea. Same for "cheap" and "budget".

This is not incremental. It’s the difference between a search engine that finds what you literally typed and one that finds what you meant.

Computing an embedding

Several providers offer embedding APIs. We’ll use OpenAI’s text-embedding-3-small because it’s cheap, high-quality, and multilingual (including Nepali). Anthropic doesn’t currently ship its own; the industry mostly uses OpenAI’s for the embedding step even in Claude-based applications.

Setup:

pip install openai numpy

.env file:

OPENAI_API_KEY=sk-...

The call:

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()


def embed(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return response.data[0].embedding


v = embed("नमस्ते! यो कस्तो छ?")
print(f"Dimensions: {len(v)}")
print(f"First 5:    {v[:5]}")

Output:

Dimensions: 1536
First 5:    [-0.023, 0.014, -0.007, 0.031, -0.019]

1536 numbers per piece of text. Every text gets its own vector.

Similarity, measured

The standard way to measure how similar two vectors are is cosine similarity — a number between -1 and 1 (in practice, embeddings are usually positive-valued so you see 0 to 1):

1.0 — identical meaning.
~0.9+ — very similar meaning.
~0.7 — related but distinct.
~0.5 — loosely related.
<0.3 — probably unrelated.

The formula: dot product of the two vectors divided by their magnitudes.

import numpy as np


def cosine_similarity(v1: list[float], v2: list[float]) -> float:
    a = np.array(v1)
    b = np.array(v2)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


# A few examples
pairs = [
    ("Hello",                    "नमस्ते"),
    ("Hello",                    "Goodbye"),
    ("Kathmandu is the capital", "काठमाडौं नेपालको राजधानी हो"),
    ("Kathmandu is the capital", "cars are fast"),
]

for a, b in pairs:
    sim = cosine_similarity(embed(a), embed(b))
    print(f"{a[:40]:40s} <-> {b[:40]:40s} {sim:.3f}")

You’ll see something like:

Hello                                    <-> नमस्ते                                   0.792
Hello                                    <-> Goodbye                                  0.680
Kathmandu is the capital                 <-> काठमाडौं नेपालको राजधानी हो             0.891
Kathmandu is the capital                 <-> cars are fast                            0.148

Read this. "Hello" and "नमस्ते" are ~0.79 similar — the model correctly recognises they mean roughly the same thing across languages. "Kathmandu is the capital" in English and Nepali are 0.89 — very similar. "Kathmandu is the capital" and "cars are fast" are 0.15 — clearly different.

The numbers encode meaning.

What “similar” really means

Cosine similarity captures a specific kind of similarity — overall meaning. It does not capture:

Truth. "Sun rises in the east" and "Sun rises in the west" will have very high similarity even though one is true and one isn’t. Embeddings care about what a sentence is about, not whether it’s correct.
Sentiment reversal. "Great product, love it" and "Terrible product, hate it" are semantically very similar (both about product opinion), just with opposite valence. You often need separate sentiment models for this.
Precise numeric differences. "5 kg" and "50 kg" are close in embedding space — both about weight — but numerically very different. Don’t use embeddings for tasks that hinge on exact numbers.

For most practical applications — retrieval, clustering, general similarity — cosine similarity of embeddings is exactly right. Just know its blind spots.

Cost and speed

Embeddings are dramatically cheaper than LLM calls:

text-embedding-3-small: $0.02 per 1M tokens.
text-embedding-3-large: $0.13 per 1M tokens (better quality, 3072 dims).

For comparison, Claude Haiku input is $0.25 per 1M tokens — over 10x more expensive than the small embedding model. Embedding 100,000 documents (say, 500 tokens each = 50M tokens) costs about $1.

Latency is also better — an embedding call typically returns in under 500ms. And crucially, once you embed a document, you don’t have to embed it again. Store the vectors and reuse them forever.

Batching for efficiency

Embedding one text at a time is slow and expensive per-request. Batch instead:

def embed_batch(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return [d.embedding for d in response.data]


vectors = embed_batch([
    "नेपाल एक सुन्दर देश हो",
    "काठमाडौं राजधानी हो",
    "Cars are fast",
    "The mountains are beautiful",
])

OpenAI accepts up to ~2048 inputs per call. For processing large document collections (thousands of chunks), batching is essential.

Nepali specifically

text-embedding-3-small handles Nepali well. Some observations:

Devanagari and Romanised Nepali are recognised as similar enough — a query in one form finds documents in the other, though not as sharply as Devanagari-Devanagari matches.
Nepali-English cross-lingual similarity is decent. "loan" and "ऋण" have similarity ~0.7, enough that a query in one language surfaces relevant content in the other.
Small Nepali variants (informal छ, formal हुनुहुन्छ) are recognised as related. Regional variants are less well-captured — the model has seen far more Kathmandu-standard Nepali than dialectal variants.

For most Nepali applications this is more than good enough. When you need higher-quality Nepali specifically, dedicated multilingual embedding models like intfloat/multilingual-e5-large (open-source, requires GPU) or Cohere embed-multilingual-v3 (API) can outperform OpenAI’s on Nepali-heavy workloads.

A short taxonomy of embedding models

For reference:

text-embedding-3-small (OpenAI, API, $0.02/M) — cheap workhorse, 1536 dims. Default for prototyping.
text-embedding-3-large (OpenAI, API, $0.13/M) — higher quality, 3072 dims. Use when quality matters more than cost.
intfloat/multilingual-e5-large (open-source, HuggingFace) — free if you have a GPU, best-in-class for many multilingual tasks including Nepali.
sentence-transformers/all-MiniLM-L6-v2 (open-source, CPU-friendly) — small, fast, English-only. Fine for prototypes.
Cohere embed-multilingual-v3.0 (API) — very strong on multilingual, comparable to OpenAI’s small.
Voyage AI’s models — strong for domain-specific retrieval (law, medicine, code).

For Course 04 we use OpenAI’s small model because it’s simple. In production, benchmark on your actual data before choosing.

Check your understanding

Quick check

—

Your semantic-search app returns documents about 'a cheap smartphone' when the user queries 'budget mobile phone'. What just happened?

The keyword search engine matched 'phone'.
Embedding-based semantic search: 'budget mobile phone' and 'cheap smartphone' have very similar embeddings because they mean roughly the same thing. Cosine similarity captured this even though the two queries share almost no keywords. This is the fundamental advantage of embeddings over keyword search — meaning, not literal words.
The LLM generated a summary.
The database indexed synonyms.

Quick check

—

A teammate is planning a fact-checking system that compares a claim ('Sun rises in the west') against a knowledge base using embedding similarity. What's the fundamental problem with this design?

Embeddings are too slow for fact-checking.
Embedding similarity captures *topical* similarity, not truth. 'Sun rises in the west' and 'Sun rises in the east' are highly similar in embedding space — both are about the sun and rising — even though one is true and one is false. Fact-checking requires reasoning, not just similarity: retrieve related documents (using embeddings), then have an LLM read the retrieved evidence and evaluate the claim against it. Embeddings alone can't tell you a claim is wrong.
Embeddings are too expensive for fact-checking.
Fact-checking requires supervised learning.

What comes next

You know what an embedding is. The next section is what to do with them: semantic search — building an index of documents you can query by meaning, in ten lines of Python. This is the foundation of every RAG system, and one of the most useful patterns you’ll learn in this course.