Glossary entry
Token
टोकन
Definition (English)
A small unit of text — a word or part of a word — that a language model processes.
अर्थ (नेपालीमा)
भाषा मोडेलले प्रशोधन गर्ने पाठको सानो एकाइ — शब्द वा शब्दको अंश।
Related terms
Other terms related to Token.
-
Bag of Words
ब्याग अफ वर्ड्स
A simple text representation counting word occurrences while ignoring order.
-
Corpus
कोर्पस / पाठ सङ्ग्रह
A large, structured collection of text used to train or study language models.
-
Function Calling (Tool Use)
फंक्सन कलिङ / उपकरण प्रयोग
Letting a language model invoke external tools or APIs to act, not just generate text.
-
GPT
जीपीटी (GPT)
A family of generative transformer models that produce text by predicting the next token.
-
Language Model
भाषा मोडेल
A model that predicts likely sequences of words to understand or generate text.
-
Large Language Model (LLM)
ठूलो भाषा मोडेल
A very large neural network trained on massive text to understand and generate language.
-
Machine Translation
मेसिन अनुवाद
Automatically translating text or speech from one language to another.
-
Next Token Prediction
अर्को टोकन अनुमान
The core training task of LLMs: predicting the next piece of text given prior context.