Search begins by turning documents and queries into units that can be compared. Those units are not always the same tokens used by a language model.
Search tokenisation decides which character sequences become indexable terms, which differences are normalised away, which boundaries matter, which positions are stored and how a user’s query is transformed before matching. The resulting representation determines what can be found quickly—and what may become invisible.
This article extends the eduKateSingapore Representation and Tokenisation series into retrieval systems.
The Search Representation Route
DOCUMENT → TEXT EXTRACTION → NORMALISATION → SEARCH TOKENISATION → TERMS + POSITIONS + FIELDS → INVERTED INDEX QUERY → QUERY NORMALISATION → QUERY TOKENISATION → TERMS / PHRASES / FILTERS → MATCH / SCORE / RANK → RESULT
1. Search Tokens Are Retrieval Units
A search token is a unit used for indexing or matching. It can be a word, normalized term, character n-gram, edge n-gram, language-specific segment or another lexical unit.
Its job is not to minimise model sequence length. Its job is to make documents retrievable.
2. Search and LLM Tokenisation Solve Different Problems
A language-model tokenizer maps text into a finite model vocabulary. A search analyser maps text into index terms designed for matching. The same visible sentence can therefore produce two completely different token sequences.
Do not assume that “token” means one universal representation across software systems.
3. The Document Must Be Extracted Before It Can Be Indexed
PDFs, webpages, office files and scanned images can contain layout, metadata, tables and hidden text. Search tokenisation usually operates on an extracted text representation rather than the original object.
If extraction loses reading order or table structure, the index faithfully stores a damaged representation.
4. Field Structure Is Part of Search Representation
Titles, headings, body text, tags, authors and dates can be indexed as separate fields. Search systems can then weight or filter them differently.
A term in a title may carry more retrieval significance than the same term buried in a footer.
5. Lowercasing Can Improve Case-Insensitive Search
Many search indexes lowercase ordinary terms so Singapore and singapore match. This increases recall but collapses case distinctions.
Identifiers, code and proper-name applications may need a second case-sensitive field.
6. Accent Folding Can Improve Recall and Reduce Fidelity
An accent-insensitive index can make queries easier for users who omit diacritics. But two distinct forms may collapse into the same search term.
A common solution is to preserve the original field while indexing a secondary normalized field for broad matching.
7. Punctuation Rules Determine Search Boundaries
Hyphens, apostrophes, slashes and dots can either split or join terms. The query state-of-the-art may need to match state of the art, while C++ loses meaning if punctuation is stripped indiscriminately.
Search tokenisation must respect domain-specific syntax.
8. Whitespace Tokenisation Is Simple but Language-Limited
Splitting on spaces works tolerably for many English queries. It is much less sufficient for languages where words are not consistently separated by whitespace.
Multilingual search requires language-aware segmentation or character-based alternatives.
9. Dictionary Segmentation Can Recover Words in Unspaced Text
Some search systems use dictionaries or statistical models to segment languages such as Chinese, Japanese or Thai into terms.
Segmentation errors can reduce recall even when every character remains present.
10. Character N-Grams Avoid One Hard Boundary Decision
Instead of deciding exactly where words begin, an index can store overlapping sequences of characters such as bigrams or trigrams.
This improves robustness to spelling variation and unsegmented scripts but increases index size and can reduce precision.
11. Edge N-Grams Power Autocomplete
Autocomplete indexes often store prefixes such as s, si, sin, sing so partially typed queries can match full terms.
The search representation is deliberately expanded to support an interaction pattern.
12. Stemming Collapses Related Surface Forms
A stemmer can map connect, connected and connecting toward a common stem. This improves recall when the user does not type the exact document form.
But stems can be linguistically crude and can merge terms that should remain distinct.
13. Lemmatization Uses More Linguistic Structure
Lemmatization maps inflected forms to a dictionary base form using language-specific analysis. It can distinguish some cases that simple suffix stripping cannot.
The cost is more linguistic machinery and language dependence.
14. Stop Words Are a Historical Compression Choice
Traditional indexes sometimes omit highly common words such as the or and because they contribute little to ordinary keyword matching and consume index space.
Modern systems often retain more positional information because common words can matter in phrase queries, legal text and titles.
15. Phrase Search Needs Positions
To distinguish machine learning from documents where machine and learning occur far apart, the index can store term positions.
Position turns a bag of terms back into partial sequence structure.
16. An Inverted Index Maps Terms Back to Documents
Instead of scanning every document for every query, an inverted index stores a posting list for each term identifying which documents contain it.
Tokenisation determines the keys of that index.
17. A Bad Token Boundary Creates Retrieval Blindness
If the index stores e-mail as one term while the query analyser produces email, matching can fail unless normalization or synonyms bridge the forms.
Document and query analysers must be compatible enough to meet in the same representation space.
18. Query Tokenisation Can Differ From Document Tokenisation Deliberately
Autocomplete, fuzzy search and query expansion may transform user input more aggressively than document indexing. The asymmetry can improve usability if it remains controlled.
Search representation is a two-sided contract, not necessarily identical preprocessing.
19. Exact Match Fields Should Avoid Destructive Analysis
Product IDs, ISBNs, usernames, codes and exact tags often need a keyword-like field that preserves the whole string as one searchable value.
Breaking an identifier into ordinary words can create false matches.
20. One Source Can Have Several Index Representations
A name field can be indexed in original form, lowercased form, accent-folded form, n-grams and phonetic keys simultaneously. Each representation supports a different retrieval route.
Multiple representations can preserve fidelity while improving recall.
21. Synonym Expansion Changes the Search Vocabulary
Search systems can treat car and automobile as related query terms even though they are distinct surface tokens.
This moves search beyond literal token identity toward lexical meaning.
22. Synonyms Need Direction and Context
Apple can mean a fruit or a company. Expanding every occurrence into every related term can destroy precision.
Semantic expansion must remain context-sensitive.
23. Spelling Correction Is Query Representation Repair
A search engine can map a misspelled query to a likely intended form. This is not tokenisation alone; it is an inference layered above the query representation.
The original query should remain available because the correction can be wrong.
24. Fuzzy Matching Avoids Exact Token Identity
Edit-distance matching can retrieve terms similar to the query even when their spellings differ slightly.
This trades precision and compute for resilience to typos and variants.
25. Compound Words Challenge Search Segmentation
One language may write a concept as one compound while another uses several words. Even within one language, hyphenation can vary.
Search tokenisation should test whether compounds remain discoverable under realistic query variation.
26. Names Challenge Search Tokenisation
Personal names can contain spaces, hyphens, apostrophes, particles and script variants. Naive tokenisation can split or normalize away identity-critical structure.
Identity search often needs dedicated analysers and alias tables.
27. Numbers Challenge Search Semantics
Searching for “2026” can mean a year, model number or identifier fragment. Numeric fields often need typed indexing rather than ordinary lexical tokenisation.
Search works better when representation preserves data type.
28. Dates Should Be Indexed as Dates
Storing a date only as text makes range queries and chronological filtering difficult. A search system can preserve the original visible string while also indexing a normalized temporal value.
Typed representations augment lexical representations.
29. Search Tokenisation Can Be Domain-Specific
Biomedical terms, chemical formulas, legal citations and source code need boundary rules different from ordinary prose. A universal standard analyser can be a reasonable baseline but a poor specialist representation.
Receiver job determines the correct search unit.
30. Code Search Needs Symbol Awareness
Searching source code for getUserID may require camelCase splitting, identifier-preserving exact fields and language-aware syntax.
Code search often indexes one identifier through several representations simultaneously.
31. Semantic Search Uses a Different Representation Layer
Embedding-based search maps queries and passages into learned vectors and retrieves by similarity rather than exact lexical overlap.
Lexical tokenisation still matters for source extraction, chunking and hybrid systems, but the matching unit becomes a vector representation.
32. Hybrid Search Combines Lexical and Semantic Representations
Lexical search excels at exact names, rare terms and identifiers. Semantic search excels at paraphrases and concept-level similarity. Combining them gives two independent retrieval routes.
Different representations fail differently.
33. LLM Tokenisation Still Matters After Retrieval
Once search returns documents or chunks, a language model tokenizer determines how much of that evidence fits into context.
Search tokenisation decides what is found; LLM tokenisation decides how the found evidence enters the model.
34. Retrieval Chunking Adds a Third Boundary System
A document can have search terms at the word scale, retrieval chunks at the paragraph scale and LLM tokens at the subword scale.
These layers must be coordinated rather than collapsed into one concept of “token”.
35. Index Migrations Must Be Versioned
Changing the analyser changes which terms are stored. Existing documents may need reindexing, and old query behaviour may no longer be reproducible.
Search tokenisation is versioned infrastructure.
36. Search Quality Should Be Measured by Query Sets
Evaluate exact names, misspellings, phrases, multilingual queries, identifiers, dates, compounds and domain terms. One average search metric can hide catastrophic failure on a small but important query class.
The real query distribution is the receiver test.
37. The Search Tokenisation Audit
- What source formats are extracted?
- What fields are indexed separately?
- What normalization applies to each field?
- How are punctuation and whitespace handled?
- Which languages need specialised segmentation?
- Are n-grams used for autocomplete or fuzzy recall?
- Are stemming or lemmatization applied?
- Are stop words preserved for phrase needs?
- Are token positions stored?
- Which identifiers require exact fields?
- What synonym or spelling-expansion logic exists?
- How are names, numbers, dates and code treated?
- Does semantic search complement lexical search?
- Are document and query analysers compatible?
- Can the index be reproduced from a versioned analyser?
38. What Students Should Remember
- Search tokens are retrieval units, not necessarily LLM tokens.
- Document and query analysers must meet in compatible representations.
- Stemming, n-grams and normalization trade precision for recall.
- Phrase search needs positional information.
- Exact identifiers often need non-tokenized keyword fields.
- Semantic search uses learned vectors rather than only term overlap.
- Hybrid search benefits from multiple representations.
39. The Deep Principle
A search engine can retrieve only what its index representation makes addressable. Every analyzer decision decides which distinctions survive into the searchable world.
Tokenisation turns documents into handles for retrieval. The right handle is not the smallest linguistic unit; it is the unit that lets the right evidence be found when the receiver asks for it.