Normalisation decides which surface differences should survive before tokenisation begins.

That makes it one of the earliest and most consequential representation choices in a language pipeline. Lowercasing can increase sharing while erasing identity. Unicode normalisation can merge equivalent encodings while preserving visible form. Whitespace cleanup can remove noise or destroy code structure. Diacritic removal can improve rough matching or collapse distinct words.

This article continues the eduKateSingapore Representation and Tokenisation series.

The Normalisation Gate

SOURCE TEXT
→ ENCODING
→ NORMALISATION POLICY
→ CANONICALISED TEXT
→ TOKENISATION
→ TOKEN IDS
→ MODEL

The principle is simple: if a distinction is removed here, the tokenizer and model cannot recover it from the normalised string alone.

1. Normalisation Is a Representation Decision

Normalisation transforms different surface forms into a more standard representation. The goal is usually consistency, but consistency is never free. It is achieved by declaring that some differences should count as equivalent.

The right policy depends on the receiver and task.

2. Unicode Normalisation Solves One Class of Equivalence

Unicode allows some visually equivalent text to be encoded through different code-point sequences. A precomposed accented character and a base character followed by a combining mark can render similarly while remaining different internally.

The Unicode Consortium defines standard normalization forms in Unicode Standard Annex #15.

3. Canonical Equivalence Is Not Byte Equality

Two canonically equivalent strings can render the same while having different encoded bytes before normalisation. A pipeline that canonicalises them may improve matching and token consistency but no longer preserve the exact original byte sequence.

That is acceptable only if byte identity is not an invariant of the task.

4. NFC and NFD Preserve Canonical Equivalence Differently

Canonical normalisation forms can prefer composed or decomposed representations. They aim to standardise equivalent character sequences without applying broader compatibility substitutions.

The important point for tokenisation is that a consistent underlying sequence creates more stable boundaries and vocabulary matches.

5. Compatibility Normalisation Is More Aggressive

Compatibility normalisation can collapse characters that are visually or semantically related but not canonically equivalent. This can improve search and matching in some applications while destroying distinctions important for typography, mathematics or specialised notation.

A stronger normalisation rule needs a stronger justification.

6. Lowercasing Increases Sharing

Converting all text to lowercase can reduce vocabulary fragmentation between sentence-initial and sentence-medial forms. It also helps matching when case should not matter.

But it removes information. US, us and a case-sensitive variable name are not necessarily equivalent.

7. Case Folding Is More Than English Lowercasing

Unicode-aware case folding tries to support caseless matching across scripts more systematically than naive ASCII lowercasing. Even so, language-specific behaviour can complicate one-size-fits-all rules.

Multilingual systems should test case policy with the actual languages they serve.

8. Proper Names Need Case Fidelity

Capitalisation can distinguish proper names, acronyms and brands. Lowercasing before tokenisation may improve compactness while making exact reconstruction impossible.

If identity matters, preserve the canonical source form even when a secondary normalised form is used for matching.

9. Code Is Often Case-Sensitive

Programming languages can distinguish UserID from userid. Lowercasing source code therefore changes the program rather than merely normalising it.

Text preprocessing that is harmless for general prose can be destructive for code.

10. Whitespace Normalisation Is Tempting

Repeated spaces, tabs and line breaks can appear noisy. Collapsing them can make prose more regular and reduce token variation.

But whitespace may encode paragraphs, indentation, table columns, poetry, Markdown or program blocks.

11. Newlines Carry Document Structure

A blank line can separate paragraphs. Consecutive newlines can signal a section break. Removing them can flatten discourse structure before the tokenizer ever sees it.

Structure-aware normalisation preserves meaningful line boundaries while removing only accidental variation.

12. Tabs Can Be Data

Tab-separated files use tabs as field delimiters. Python uses indentation structurally. Fixed-width or aligned text can use whitespace to represent columns.

“Clean whitespace” is not a universal operation.

13. Diacritics Carry Language Information

Removing accents can make fuzzy search more forgiving, but diacritics can distinguish words, pronunciation and identity. A system that strips them may merge forms that speakers treat as different.

Normalisation must respect the linguistic distinctions of the target language.

14. Search and Generation May Need Different Normalisation

A search index may store a secondary accent-insensitive key while preserving the original text for display. A generative model may need the original form to reproduce names accurately.

One source can support multiple representations rather than forcing one destructive normalisation on every task.

15. Punctuation Standardisation Can Improve Consistency

Text from different sources may contain straight quotes, curly quotes, multiple dash characters and language-specific punctuation. Normalising selected punctuation can reduce surface variation.

But punctuation can be semantic in code, mathematics and typography-sensitive tasks.

16. Hyphens Are Not Always Interchangeable

Hyphen, en dash, em dash and minus sign can look similar while serving different functions. A normaliser that collapses them can improve rough matching and simultaneously corrupt mathematical meaning.

Visual similarity is not sufficient evidence for equivalence.

17. Apostrophes and Quotes Can Affect Token Boundaries

Curly and straight apostrophes can cause different tokenisations if the vocabulary treats them differently. Standardising them may reduce fragmentation in ordinary text.

Yet exact quotations and editorial work may require preserving the original typography.

18. Full-Width and Compatibility Characters Need Deliberate Policy

East Asian text can contain full-width forms of characters that resemble common ASCII symbols. Compatibility normalisation can map some of these into standard forms.

This can improve interoperability while changing the original representation.

19. Zero-Width Characters Are Invisible but Not Necessarily Meaningless

Zero-width joiners and related characters can affect grapheme formation and text rendering. Removing all invisible characters as “noise” can break emoji sequences or script behaviour.

Invisible does not mean irrelevant.

20. Bidirectional Marks Carry Display Semantics

Right-to-left and left-to-right text can use directional marks that influence display order. Stripping them may make logical text harder to render correctly.

Multilingual normalisation must account for bidirectional text rather than treating it as ordinary Latin prose.

21. HTML Entity Decoding Is Another Representation Transformation

Web text may contain entities such as & representing visible characters. Decoding entities before tokenisation can align the representation with what the user sees.

But raw-source analysis may need to preserve the markup exactly. Again, the receiver decides.

22. Markup Removal Can Destroy Structure

Stripping HTML or Markdown tags may remove layout noise while also erasing headings, links, emphasis, table boundaries and list hierarchy.

A better pipeline often parses markup into structured content rather than flattening it blindly.

23. URLs Should Usually Stay Exact

Lowercasing or decoding parts of URLs without understanding URL semantics can change the resource addressed. Domain names and paths have different case rules in practice.

Exact identifiers need domain-aware normalisation.

24. Email Addresses Also Have Identity Constraints

An email address looks like ordinary text but functions as an identifier. Normalising it casually can alter the canonical string or create mismatches across systems.

Identifiers deserve stricter representation rules than prose.

25. Numbers Should Not Be Normalised Blindly

Removing commas from 1,234 may be useful in one locale and wrong in another where commas represent decimals. Currency and percentage symbols can carry essential units.

Numeric normalisation requires locale and domain context.

26. Dates Need Semantic Normalisation, Not String Cleanup Alone

04/09/2026 cannot be safely converted into an ISO date without knowing the convention. A normaliser that guesses can create false precision.

Representation transformation must distinguish syntax from interpretation.

27. Repeated Characters Can Be Noise or Meaning

“soooo good” contains repeated letters that may express emphasis. Collapsing them to “so good” improves lexical regularity while losing affective information.

Informal language normalisation should preserve the user signals the task needs.

28. Emoji Normalisation Can Lose Social Meaning

Skin-tone modifiers, gender combinations and variation selectors can change how an emoji is perceived. Mapping every variant to one base symbol may simplify classification while erasing identity or tone.

There is no neutral collapse when the variants matter to the receiver.

29. Abbreviation Expansion Is Not Pure Normalisation

Expanding “Dr.” to “Doctor” or “SG” to “Singapore” introduces interpretation. The abbreviation may be ambiguous. This crosses from canonicalisation into semantic transformation.

Pipelines should label when they move from form normalisation to meaning inference.

30. Spelling Correction Is Also Semantic Intervention

Correcting a typo can improve tokenisation and retrieval, but the “typo” may be a name, dialect form or specialist term. Automatic correction can replace the user’s intended string with a more common but wrong one.

Keep the original source and record the correction as a derived representation.

31. Normalisation Changes Token Counts

Collapsing surface variation can make strings match more common vocabulary entries and reduce token counts. Conversely, preserving specialised punctuation or uncommon forms can increase fragmentation.

Token efficiency is therefore partly a preprocessing result.

32. Normalisation Changes Training Statistics

If several surface forms are collapsed before tokenizer training, their frequencies combine. That can change which subword pieces are selected into the vocabulary.

Normalisation policy therefore shapes vocabulary design, not merely inference-time input.

33. Training and Inference Must Use Compatible Normalisation

If training text was normalised one way and production text another, the tokenizer can encounter patterns whose statistics differ from what the model learned.

The preprocessing contract belongs with the tokenizer and model version.

34. Preserve the Raw Source

A strong data pipeline stores raw input whenever lawful and practical, then records normalized derivatives separately. This makes later audits, corrections and alternative tokenisation possible.

Irreversible normalisation should not silently replace the authoritative source.

35. Keep a Transformation Receipt

Record which Unicode form, case policy, whitespace policy, punctuation rules and other transformations were applied. A token sequence without its preprocessing history is difficult to reproduce.

Provenance belongs at every representation layer.

36. The Normalisation Audit

1. What is the authoritative source representation?
2. Which Unicode normalisation form is used?
3. Is case preserved, lowered or case-folded?
4. Are diacritics preserved?
5. What happens to whitespace and newlines?
6. Which punctuation variants are collapsed?
7. Are compatibility characters transformed?
8. How are zero-width and directional characters handled?
9. Are URLs, emails, code and identifiers exempt from destructive rules?
10. Are dates and numbers interpreted with locale context?
11. Is spelling correction separated from normalisation?
12. Can every transformation be reproduced from a versioned policy?

37. What Students Should Remember

• Normalisation happens before tokenisation in many pipelines.
• It decides which surface differences count as equivalent.
• Every collapse trades variation for information loss.
• Unicode equivalence, case, whitespace and diacritics matter.
• Code, identifiers and multilingual text need special care.
• The raw source should be preserved when exactness matters.

38. The Deep Principle

Normalisation is the first act of editorial judgement in a token pipeline. Before the system decides where to cut, it decides which visible differences deserve to remain distinct.

Canonicalisation can make a representation cleaner, but every cleaned distinction must be one the receiver can afford to lose.

Continue the Representation & Tokenisation Series

• Token Vocabulary Design | How Systems Choose Which Pieces Deserve Their Own Identity
• Byte-Level Tokenisation | How Bytes Give Models a Route to Almost Any Text
• Tokenizer Evaluation and Benchmarking | How to Measure Coverage, Efficiency, Fidelity and Task Fit
• Canonical owner: World Representation & Cognitive Tools