eduKate SG

Category: Artificial Intelligence

Code Tokenisation | Identifiers, Operators, Whitespace and Syntax-Aware Boundaries
Numerical Tokenisation | How Digits, Decimals, Dates and Units Enter Language Models
Entity Tokenisation | When Names, IDs, Aliases and Mentions Point to One Thing
Representation Cost Accounting | How Tokens Turn Meaning Into Compute, Memory, Latency and Money
Token Granularity | How Characters, Subwords, Words, Phrases and Concepts Create Different Model Worlds
Tokenisation in Search and Indexing | How Documents Become Searchable Terms and Query Units
Tokenizer Migration and Version Compatibility | How to Change Token Boundaries Without Breaking the Model
Discrete Latent Tokens | How Continuous Signals Become Codebook Identities
Multimodal Tokenisation | How Text, Images, Audio and Video Share One Model Context
Audio Tokenisation | How Sound Becomes Frames, Features and Discrete Acoustic Units
Visual Tokenisation | How Images Become Patches, Regions and Model-Ready Visual Units
Tokenizer Training Corpus Design | How the Data Before the Model Shapes Every Token Boundary After It
How Unigram Tokenisation Works | From Many Candidate Pieces to a Probabilistic Subword Model
How WordPiece Works | From Subword Vocabulary to Greedy Longest-Match Tokenisation
How Byte Pair Encoding Works | From Frequent Pairs to a Learned Subword Vocabulary

Subscribe to
our newsletter