Semantic search over Nepali documents · Building with Language Models

You have the pieces — embeddings, similarity, an index. This section assembles them into a complete, real project: semantic search over a folder of Nepali documents. The example is a corpus of government-style circulars, but the pattern generalises to any Nepali text collection — news articles, medical notes, your own writing. By the end of the section you’ll have code that loads a folder, chunks documents intelligently, embeds them, and answers Nepali queries with source-attributed results.

The project

We build a searchable corpus with:

Input: a folder of .txt files, each a Nepali circular or article.
Output: given a Nepali query, return the top-k most relevant passages (not entire documents), with source filenames.
Behaviour: cross-lingual (English queries find Nepali documents and vice versa), respects natural document boundaries, and caches embeddings so re-indexing is free.

The chunking problem

If a document is a 5000-word circular and you embed the whole thing, you get one vector that averages meaning across the entire document. A specific query about “Article 5 fees” won’t stand out; the vector is dominated by everything else.

The standard fix: chunking. Split each document into passages (typically 200-800 tokens each), embed each chunk separately, and index the chunks. Now the search returns “this specific paragraph in this specific document” rather than “this whole document”.

Two things to get right:

Chunk size. Too small (50 tokens) and each chunk loses context. Too large (2000 tokens) and you lose the granularity. For most text, 300-500 tokens is a sweet spot.

Chunk boundaries. Don’t cut mid-sentence or mid-paragraph. Split on natural boundaries — paragraphs, then sentences.

Here’s a simple splitter:

import re
from typing import Iterator


def split_into_paragraphs(text: str) -> list[str]:
    # Split on double-newlines (paragraph breaks), strip whitespace
    parts = re.split(r"\n\s*\n", text.strip())
    return [p.strip() for p in parts if p.strip()]


def approximate_token_count(text: str) -> int:
    # Rough heuristic: 1 token per ~3 chars for Nepali, per ~4 chars for English
    return max(1, len(text) // 3)


def chunk_document(
    text: str,
    target_size: int = 400,
    max_size: int = 700,
) -> list[str]:
    paragraphs = split_into_paragraphs(text)
    chunks: list[str] = []
    current: list[str] = []
    current_size = 0

    for p in paragraphs:
        p_size = approximate_token_count(p)

        # If adding this paragraph would exceed max_size, close the current chunk
        if current and (current_size + p_size > max_size):
            chunks.append("\n\n".join(current))
            current = [p]
            current_size = p_size
            continue

        current.append(p)
        current_size += p_size

        # If we've reached target size, close the chunk here
        if current_size >= target_size:
            chunks.append("\n\n".join(current))
            current = []
            current_size = 0

    if current:
        chunks.append("\n\n".join(current))

    return chunks

The rules: prefer keeping full paragraphs together; close a chunk when it reaches the target size; force a close when it would exceed the max. Simple, works for almost any prose.

Loading a folder

from pathlib import Path
from dataclasses import dataclass


@dataclass
class Passage:
    text: str
    source: str
    chunk_index: int


def load_corpus(folder: str) -> list[Passage]:
    passages: list[Passage] = []
    for path in sorted(Path(folder).glob("*.txt")):
        text = path.read_text(encoding="utf-8")
        chunks = chunk_document(text)
        for i, chunk in enumerate(chunks):
            passages.append(Passage(text=chunk, source=path.name, chunk_index=i))
    return passages

Every chunk knows which file it came from and where in the file. That metadata will be gold when we present results.

Embedding with caching

Embeddings cost money, and re-embedding the same text every time is wasteful. Cache by text hash:

import hashlib
import json
import numpy as np
from openai import OpenAI

_client = OpenAI()


class EmbeddingCache:
    def __init__(self, cache_path: str):
        self.cache_path = Path(cache_path)
        self.cache: dict[str, list[float]] = {}
        if self.cache_path.exists():
            with open(self.cache_path, encoding="utf-8") as f:
                self.cache = json.load(f)

    def _key(self, text: str) -> str:
        return hashlib.sha256(text.encode("utf-8")).hexdigest()

    def embed(self, texts: list[str], batch_size: int = 100) -> list[list[float]]:
        result = [None] * len(texts)
        to_embed_indices = []
        to_embed_texts = []

        for i, t in enumerate(texts):
            k = self._key(t)
            if k in self.cache:
                result[i] = self.cache[k]
            else:
                to_embed_indices.append(i)
                to_embed_texts.append(t)

        for batch_start in range(0, len(to_embed_texts), batch_size):
            batch = to_embed_texts[batch_start:batch_start + batch_size]
            response = _client.embeddings.create(
                model="text-embedding-3-small",
                input=batch,
            )
            for offset, embedding_obj in enumerate(response.data):
                overall_i = to_embed_indices[batch_start + offset]
                vec = embedding_obj.embedding
                result[overall_i] = vec
                self.cache[self._key(to_embed_texts[batch_start + offset])] = vec

        return result

    def save(self):
        with open(self.cache_path, "w", encoding="utf-8") as f:
            json.dump(self.cache, f, ensure_ascii=False)

First run embeds everything. Second run reads from the cache — near-instant, zero cost.

The complete searchable corpus

Putting it all together:

class NepaliCorpusIndex:
    def __init__(
        self,
        corpus_folder: str,
        cache_path: str = "data/embeddings_cache.json",
    ):
        self.passages = load_corpus(corpus_folder)
        self.cache = EmbeddingCache(cache_path)
        raw_vectors = self.cache.embed([p.text for p in self.passages])
        self.vectors = np.array(raw_vectors)
        self.cache.save()

    def search(self, query: str, k: int = 5) -> list[tuple[Passage, float]]:
        query_vec = np.array(self.cache.embed([query])[0])
        similarities = self.vectors @ query_vec / (
            np.linalg.norm(self.vectors, axis=1) * np.linalg.norm(query_vec)
        )
        top_idx = np.argsort(similarities)[-k:][::-1]
        return [(self.passages[i], float(similarities[i])) for i in top_idx]


def main():
    index = NepaliCorpusIndex("data/nepali_circulars/")

    queries = [
        "नयाँ शिक्षा नीति के छ?",
        "loan fees for microfinance",
        "जिल्ला विकास प्रक्रिया",
    ]

    for q in queries:
        print(f"\n=== Query: {q} ===")
        for passage, sim in index.search(q, k=3):
            print(f"\n  Similarity: {sim:.3f}")
            print(f"  Source: {passage.source} (chunk {passage.chunk_index})")
            print(f"  {passage.text[:200]}...")


if __name__ == "__main__":
    main()

Under 150 lines total. Loads any folder, handles any language mix, respects natural boundaries, caches expensive work. This is the shape of every RAG-style project you’ll build.

What to expect on real data

For a corpus of ~100 Nepali government circulars (each 2-5 pages), you’ll see:

Indexing time: ~30 seconds first run, <1 second cached.
Indexing cost: ~$0.05 for the full corpus.
Search time: ~200-500ms per query (mostly the query embedding call).
Search quality: solid on paraphrased Nepali queries, decent on cross-lingual (English → Nepali documents), variable on very specific Nepali terminology the model may not have seen.

Nepali chunking quirks

A few honest observations from Nepali-heavy applications:

Devanagari sentence-splitting is trickier than English. The Nepali sentence terminator । (danda) is common in formal writing but variable elsewhere. Splitting on । in addition to Western periods helps.
Bilingual documents (Nepali paragraphs mixed with English tables) chunk unevenly. Consider handling tables separately or accepting some noise.
Numbers and dates. Nepali documents mix Devanagari digits (१२३), Western digits (123), and Bikram Sambat vs Gregorian dates. Semantic search handles the meaning fine; if users query with one convention and documents use another, results still match.
OCR’d Nepali (scanned PDFs run through OCR) often has broken word boundaries and mixed scripts. Clean OCR output before embedding — even a 5-minute cleanup often bumps search quality significantly.

What we haven’t built

Two things you’ll add before shipping this as a real product:

Metadata filtering. Users often want “circulars from 2025” or “only from the education ministry.” Add metadata fields to the Passage class and filter before searching — most vector databases support this natively.

Reranking. For higher precision, run the top-20 retrieved passages through a small reranker model or LLM to pick the top-5. This trades cost for accuracy.

We’ll meet both in Chapter 5.

Check your understanding

Quick check

—

A teammate wants to speed up their Nepali document search by embedding entire 20-page documents (rather than chunks). What's the practical downside?

Full-document embeddings are more expensive.
When you embed a full 20-page document, the resulting vector averages meaning across everything in it — a query about a specific paragraph gets muddled with all the other content in that document. Search returns the whole document instead of the relevant paragraph, and short queries don't stand out against the document's dominant themes. Chunking preserves the ability to pinpoint what part of what document is relevant. Speed isn't the bottleneck; chunk quality is.
The model can't handle 20-page documents.
Full-document embeddings are slower to compute.

Quick check

—

You've built a Nepali FAQ search app. To measure whether it's actually useful, what should you do first, per the marginal note in this section?

Ship it and see.
Hand-write 20 realistic queries paired with the FAQ item you expect to be returned. Measure whether the correct FAQ is in the top-3 retrieved for each query, and use that as your evaluation set. Every change (chunk size, model choice, prompt) can now be A/B tested against a real measurement, rather than 'it feels better'. This is boring, essential, and the difference between real quality and lucky demos.
Trust the demo you showed the CEO.
Embedding models are always accurate; no measurement needed.

What comes next

You can retrieve relevant passages. Chapter 5 puts an LLM reader on top: retrieval-augmented generation. The reader synthesises answers from retrieved passages, cites sources, and gracefully says “I don’t know” when the corpus doesn’t have the answer. This is the single most useful pattern in modern LLM applications — most production apps you’ll build are variations of RAG.