Tokenization

Data

डाटा / तथ्याङ्क

Raw facts, numbers, text, images, or sounds that AI systems learn from and process.

Long Context

लामो सन्दर्भ

The capacity of a model to process very large amounts of input text at once.

Autoregressive

अटोरिग्रेसिभ

Generating output one token at a time, each based on all the tokens produced before it.

Bag of Words

ब्याग अफ वर्ड्स

A simple text representation counting word occurrences while ignoring order.

BERT

बर्ट (BERT)

An influential transformer model that reads text in both directions to understand context.

Byte-Pair Encoding (BPE)

बाइट-पेयर इन्कोडिङ

A tokenization method that builds subword units from frequent character pairs.

Chatbot

च्याटबोट / कुराकानी बोट

A program that converses with people through text or voice, often powered by AI.

Chunking

चन्किङ / टुक्र्याउने

Splitting long documents into smaller pieces for indexing and retrieval.