Searching by meaning, not keywords · Building with Language Models

You have embeddings. Now you build the workhorse pattern of modern retrieval: semantic search. The idea is simple — turn every document in your collection into a vector, turn each search query into a vector, and return the documents whose vectors are closest to the query’s vector. The code is twenty lines. The applications are enormous. This section walks the full pipeline and gives you the shape you’ll reuse everywhere.

The pipeline, in one picture

     [ FAQ answer 1 ]      →  vector [ ... ]  ┐
     [ FAQ answer 2 ]      →  vector [ ... ]  │
     [ FAQ answer 3 ]      →  vector [ ... ]  │
        ...                        ...         │  (build once)
     [ FAQ answer N ]      →  vector [ ... ]  ┘

     [ user query ]        →  vector [ ... ]  ┐
                                              │  (each search)
     compute similarity of query vs each answer
     return top-k by cosine similarity        ┘

Two steps. Build the index once (embed all documents, store the vectors). Search many times (embed the query, compare, rank).

The minimum viable index

For our first version, keep it dead simple — no database, just a Python list of (text, vector) pairs:

import numpy as np
from openai import OpenAI

client = OpenAI()


def embed_batch(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return [d.embedding for d in response.data]


def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


class SemanticIndex:
    def __init__(self, documents: list[str]):
        self.documents = documents
        raw_vectors = embed_batch(documents)
        self.vectors = np.array(raw_vectors)   # shape (N, 1536)

    def search(self, query: str, k: int = 3) -> list[tuple[str, float]]:
        query_vec = np.array(embed_batch([query])[0])

        # Vectorised similarity to every document
        norms = np.linalg.norm(self.vectors, axis=1) * np.linalg.norm(query_vec)
        similarities = self.vectors @ query_vec / norms

        # Top-k
        top_idx = np.argsort(similarities)[-k:][::-1]
        return [(self.documents[i], float(similarities[i])) for i in top_idx]

Forty lines. Handles arbitrary text collections. Works.

Trying it on a small Nepali FAQ

Suppose we have a mini-FAQ for a shop:

faq = [
    "How do I pay with Khalti? Open the app, scan the QR code, enter the amount, and confirm with your PIN.",
    "What are your opening hours? Sunday to Friday, 9am to 8pm. Closed on Saturdays.",
    "Do you deliver to Lalitpur? Yes, we deliver anywhere in the Kathmandu Valley for Rs 100.",
    "How can I return a product? Bring it back within 7 days with the receipt for a full refund.",
    "Is the shop wheelchair accessible? Yes, we have a ramp at the main entrance.",
    "How do I contact customer support? Call 9801234567 or WhatsApp us at the same number.",
    "मैले खल्तीबाट कसरी तिर्ने? एप खोल्नुहोस्, QR स्क्यान गर्नुहोस्, रकम हाल्नुहोस्, PIN ले पुष्टि गर्नुहोस्।",
    "पसल कति बजेसम्म खुल्छ? आइतबारदेखि शुक्रबार, बिहान ९ बजेदेखि राति ८ बजेसम्म।",
    "ललितपुर डेलिभरी हुन्छ? काठमाडौं उपत्यकाभित्र रु १०० मा डेलिभरी।",
]

index = SemanticIndex(faq)

Building the index took one API call (all 9 documents batched). Now searches:

queries = [
    "how much do i pay for delivery?",
    "when are you open?",
    "क्या तपाईंको ठाउँ व्हीलचेयरका लागि पहुँचयोग्य छ?",  # Nepali about accessibility
    "the phone number?",
]

for q in queries:
    print(f"\nQuery: {q}")
    for text, sim in index.search(q, k=2):
        print(f"  {sim:.3f}  {text[:60]}...")

Output:

Query: how much do i pay for delivery?
  0.741  Do you deliver to Lalitpur? Yes, we deliver anywhere in the...
  0.523  ललितपुर डेलिभरी हुन्छ? काठमाडौं उपत्यकाभित्र रु १०० मा डेलिभरी...

Query: when are you open?
  0.821  What are your opening hours? Sunday to Friday, 9am to 8pm...
  0.639  पसल कति बजेसम्म खुल्छ? आइतबारदेखि शुक्रबार, बिहान ९ बजे...

Query: क्या तपाईंको ठाउँ व्हीलचेयरका लागि पहुँचयोग्य छ?
  0.752  Is the shop wheelchair accessible? Yes, we have a ramp...
  0.288  ललितपुर डेलिभरी हुन्छ? काठमाडौं उपत्यकाभित्र...

Query: the phone number?
  0.564  How can I contact customer support? Call 9801234567...
  0.310  Is the shop wheelchair accessible?...

Notice three interesting behaviours:

Cross-lingual works. The Nepali query about accessibility (in Devanagari + Hindi-adjacent phrasing) surfaces the English “wheelchair accessible” FAQ — with a solid 0.75 similarity. This is the killer feature.
Same-language wins. For each English query, the English FAQ ranks higher than the Nepali equivalent. That’s usually right — same language typically wins for exact matching.
Related-but-different scores lower. For “the phone number?”, the correct FAQ scores 0.56 (relevant but not extremely high) because “customer support” isn’t literally in the query. Semantic search sees the connection anyway.

Cost and speed, honestly

For a 1000-document FAQ:

Indexing: one batched embedding call, ~2 seconds, ~$0.01 (at typical document lengths).
Search: one embedding call for the query (~200ms, negligible cost), plus in-memory similarity computation (~1ms for 1000 vectors).

At 100k documents, indexing takes about a minute and costs about $1. Search latency stays under a second because NumPy’s vectorised dot product is fast. For millions of documents you need a proper vector database — see the next section.

What we don’t handle yet

The naïve pipeline above works but has real limitations:

In-memory only. The vectors live in a Python list; they disappear when the process ends.
Slow at scale. At millions of vectors, brute-force cosine similarity slows down; you want an approximate-nearest-neighbour index.
No metadata filtering. You can’t filter by date, category, or user in the naïve version.
No chunking. If your documents are long (a 20-page PDF), embedding the whole thing produces a fuzzy average. Long documents need to be split into meaningful chunks first — the topic of Chapter 5.

For the next section we solve the persistence problem with a simple on-disk vector store. For everything else, Chapter 5 (RAG) gives you the production patterns.

Persistence, quickly

For anything beyond a toy demo, save the vectors to disk:

import json
import numpy as np


class PersistentIndex:
    def __init__(self, path: str):
        self.path = path
        self.documents: list[str] = []
        self.vectors: np.ndarray = np.empty((0, 1536))

    def add(self, texts: list[str]):
        new_vecs = np.array(embed_batch(texts))
        self.documents.extend(texts)
        self.vectors = np.vstack([self.vectors, new_vecs])

    def save(self):
        np.save(self.path + ".vectors.npy", self.vectors)
        with open(self.path + ".docs.json", "w", encoding="utf-8") as f:
            json.dump(self.documents, f, ensure_ascii=False, indent=2)

    def load(self):
        self.vectors = np.load(self.path + ".vectors.npy")
        with open(self.path + ".docs.json", encoding="utf-8") as f:
            self.documents = json.load(f)

    def search(self, query: str, k: int = 3) -> list[tuple[str, float]]:
        query_vec = np.array(embed_batch([query])[0])
        norms = np.linalg.norm(self.vectors, axis=1) * np.linalg.norm(query_vec)
        similarities = self.vectors @ query_vec / norms
        top_idx = np.argsort(similarities)[-k:][::-1]
        return [(self.documents[i], float(similarities[i])) for i in top_idx]

Now you can build the index once, save it, and load it later:

# One-time build
index = PersistentIndex("data/faq_index")
index.add(faq)
index.save()

# Later — no re-embedding
index = PersistentIndex("data/faq_index")
index.load()
print(index.search("delivery cost?"))

Fifty lines of Python. Handles up to tens of thousands of documents fine. For anything bigger, use a vector database (Chroma, LanceDB, pgvector, Qdrant, Weaviate). Same conceptual model, better scale.

Semantic search is not everything

Two honest caveats:

Hybrid search wins. In practice, the best retrieval combines semantic search with traditional keyword search (BM25). Semantic search finds meaning matches; BM25 finds exact-term matches. Neither dominates. Production systems typically use both, weighted, then rerank.

Reranking helps. After retrieving 20-50 candidates cheaply with embeddings, a smaller cross-encoder or LLM can rerank them for quality. Costs more per query but improves precision dramatically.

For most Course 04 projects, plain semantic search is enough. The hybrid + rerank pattern is standard in production systems; you’ll meet it when you build one.

Check your understanding

Quick check

—

A user's Nepali query 'ललितपुरमा डेलिभरी हुन्छ?' finds the English document 'Do you deliver to Lalitpur?' with a high similarity score. Why?

The database was preconfigured with translations.
The embedding model treats semantically similar text similarly across languages. Both the English question and its Nepali equivalent map to nearby points in embedding space, because they mean the same thing. Cosine similarity between the query vector and document vector captures this cross-lingual similarity automatically — no translation step required. This is why semantic search works well for multilingual retrieval.
The model translated the query first.
Semantic search doesn't work across languages.

Quick check

—

A production semantic-search app slows to a crawl when the document collection grows to 5 million documents. What's the appropriate response, in the language of this section?

Move to keyword search.
Move to a real vector database (Chroma, LanceDB, pgvector, Qdrant, Weaviate). At millions of documents, brute-force cosine similarity is too slow; vector databases use approximate-nearest-neighbour indexes (like HNSW) to return top-k results in milliseconds regardless of collection size. The conceptual model — embed + rank — stays the same; only the storage/index changes.
Reduce the embedding dimensions.
Only search recent documents.

What comes next

You have semantic search. The last section of Chapter 4 applies it to a real Nepali problem: searching a corpus of Nepali documents — either government circulars, news articles, or your own notes — end to end. Combined with the RAG pattern in Chapter 5, this is what turns generic LLM applications into ones that answer from your specific corpus, in Nepali, with cited sources.