Source code looks like text, but it is text governed by a formal grammar. Tokenising code therefore has to preserve not only spelling, but operators, identifiers, delimiters, indentation and structures whose smallest errors can change program behaviour.
A language-model tokenizer and a compiler lexer can both divide code into units, but they do different jobs. The language-model tokenizer optimises a sequence for neural computation. The lexer identifies formal lexical units defined by the programming language.
This article continues the eduKateSingapore Representation and Tokenisation series and develops the distinction introduced in Segmentation vs Parsing.
The Code Representation Stack
SOURCE FILE → CHARACTER / BYTE REPRESENTATION → LANGUAGE-MODEL TOKENISATION → MODEL TOKENS AND, IN A COMPILER ROUTE: SOURCE FILE → LEXER TOKENS → PARSER → ABSTRACT SYNTAX TREE → SEMANTIC ANALYSIS → EXECUTION / COMPILATION
1. Code Is Not Ordinary Prose
Natural language tolerates paraphrase. Code often does not. Replacing one operator, quote or indentation level can change a program from valid to invalid or from correct to dangerous.
Representation fidelity is therefore stricter for many coding tasks.
2. A Compiler Lexer Uses Language Rules
A lexer identifies formal units such as keywords, identifiers, numeric literals, string literals and operators according to the programming language specification.
Its tokens are designed to support parsing.
3. An LLM Tokenizer Uses Statistical Vocabulary Rules
A language-model tokenizer may split one identifier into several subwords, keep a common keyword as one token or combine whitespace with a following fragment depending on its vocabulary.
Its units need not align with compiler tokens.
4. One Lexer Token Can Span Several Model Tokens
The identifier calculateMonthlyRevenue is one identifier token to a compiler lexer. A language model may represent it as several subword pieces.
The model must compose those pieces before reasoning about the identifier as one program object.
5. One Model Token Can Contain Several Surface Characters
Common keywords or punctuation patterns can become compact model tokens. The exact boundary depends on the tokenizer corpus and vocabulary.
Compactness reflects representation frequency, not syntax authority.
6. Identifiers Carry Semantic Hints
Names such as userCount, isValid and fetchInvoice communicate intent to human programmers. Splitting identifiers into meaningful subwords can help models share information across related names.
Code tokenisation therefore benefits from internal identifier structure.
7. CamelCase Creates Implicit Boundaries
getUserName contains no spaces, yet humans perceive get, user and name. Tokenizers trained on code can learn these recurring fragments statistically.
A code-aware search engine may also split camelCase explicitly for retrieval.
8. snake_case Makes Boundaries Explicit
get_user_name uses underscores as visible separators. A tokenizer may preserve the underscore separately, attach it to neighbouring text or include common multi-character patterns.
The representation should retain enough structure for exact reconstruction.
9. Keywords Are Small but High-Impact
if, else, return, class and async are short strings with structural meaning.
Token length has little relationship to program importance.
10. Operators Encode Computation
=, ==, ===, +, **, && and other operators can differ by one character while expressing different operations.
Tokenizer and decoder fidelity around punctuation is therefore critical.
11. Multi-Character Operators Must Stay Distinguishable
A model tokenizer can represent == as one token or two, but the reconstructed source must preserve the two-character operator exactly.
Compiler semantics live above model token boundaries.
12. Brackets Encode Hierarchy
Parentheses, braces and square brackets establish grouping, calls, blocks, lists, indexing and other structures depending on the language.
A missing closing bracket can invalidate an otherwise sensible program.
13. Matching Delimiters Create Long-Range Dependencies
An opening brace can correspond to a closing brace hundreds of tokens later. Correct modelling therefore requires relationships across long code spans.
Local token adjacency is not enough to represent block structure.
14. Whitespace Can Be Syntax
In Python, indentation defines blocks. In other languages whitespace may be mostly optional but still important for formatting and readability.
A preprocessing rule that collapses all whitespace can destroy Python semantics.
15. Tabs and Spaces Can Behave Differently
Editors can display tabs and spaces similarly while compilers, linters or style tools treat them differently.
Exact code reconstruction must preserve the source’s whitespace contract when it matters.
16. Newlines Can Terminate Statements
Some programming languages use newlines syntactically or allow them to influence automatic semicolon insertion.
Line boundaries are therefore part of the code representation.
17. String Literals Contain a Language Inside the Language
A string can contain ordinary prose, JSON, SQL, a regular expression, a URL or another programming language fragment.
Code tokenisation encounters nested representational systems.
18. Escaping Changes Character Meaning
Inside a string, \n can represent a newline escape rather than two ordinary characters. Quotes and backslashes can switch between syntax and content depending on escaping.
Parsing must interpret these relationships above tokenisation.
19. Raw Strings Change the Escape Rules
Some languages provide raw-string syntax where backslashes receive different treatment. The same visible substring can therefore have different semantics under different literal forms.
Context determines syntax.
20. Comments Are Semantically Invisible to Execution but Important to Humans
Comments often explain intent, assumptions and warnings. A compiler can remove them before execution, but a code model may benefit from retaining them as natural-language context.
Whether comments are “noise” depends on the receiver.
21. Documentation Strings Are Both Code Objects and Text
Docstrings can be part of the program object model while also serving as human documentation. They bridge formal and natural language.
Code models need to coordinate both representations.
22. Numeric Literals Have Language-Specific Syntax
Languages may support hexadecimal, binary, underscores in numbers, exponent notation and suffixes indicating numeric type.
Text tokenisation alone does not reveal the parsed numeric value.
23. Type Annotations Add Structural Meaning
name: str and generic forms such as list[int] use punctuation and identifiers to represent type relationships.
The tokenizer sees surface pieces; the parser builds a type-expression structure.
24. Imports Establish External Identity
An import statement connects a local name to an external module or package. The string itself is only one part of the identity relationship.
Repository context and dependency metadata resolve what code object the name actually refers to.
25. Scope Changes Identifier Meaning
The variable x can refer to different objects in different functions or nested scopes.
Lexical identity does not equal program identity.
26. Symbol Tables Create Entity Identity for Code
Compilers and language servers track declarations and references so one identifier mention can be linked to its defining symbol.
This is code’s version of entity resolution.
27. Renaming Requires Identity, Not String Replacement
A safe refactoring tool renames references to one symbol without replacing every matching string in comments, unrelated scopes or string literals.
Structure-aware representation beats surface-token replacement.
28. Abstract Syntax Trees Compress Surface Code Into Structure
An AST represents expressions, statements and declarations as a tree. It omits some formatting while preserving syntax relationships.
The AST is a coarser semantic representation than the original token stream.
29. ASTs Usually Do Not Preserve Every Source Detail
Comments, exact whitespace and some punctuation can disappear in an abstract syntax tree.
A code editor that needs exact round-trip formatting may instead use a concrete syntax tree or retain source spans alongside the AST.
30. Concrete Syntax Trees Preserve More Surface Fidelity
A concrete syntax tree can keep tokens and punctuation needed to reconstruct the original form more exactly.
More fidelity requires a richer representation.
31. Repository-Level Context Goes Beyond One File
Correct interpretation of a function call can depend on another file, generated code, configuration or dependency version.
File tokenisation is only the local entry layer.
32. Chunking Code by Fixed Tokens Can Break Functions
A retrieval system that splits code every 500 tokens can cut a function halfway through or separate a class from its methods.
Syntax-aware chunking can use functions, classes and AST nodes as boundaries.
33. Function-Level Chunks Can Be Better Retrieval Units
A whole function often contains a coherent unit of behaviour. Attaching its signature, comments and surrounding class context can improve code retrieval.
The best chunk boundary follows the downstream coding task.
34. Very Large Functions Need Hierarchical Representation
A thousand-line function is too large to treat as one retrieval unit. Systems can preserve the function identity while representing internal blocks at finer resolution.
Hierarchy lets structure and token budgets coexist.
35. Code Search Uses Several Tokenisations
Search may index exact identifiers, camelCase fragments, comments, strings and AST symbols separately.
One source file can support several retrieval representations at once.
36. Language Models Can Learn Syntax Above Subword Tokens
A model does not need one compiler token per neural token to learn program structure. Contextual layers can reconstruct identifier, expression and block relations across several subword positions.
Subword tokenisation is compatible with structural code reasoning.
37. But Token Efficiency Still Matters for Large Repositories
If code tokenises compactly, more files or longer functions can fit inside the same context window.
Tokenizer training corpora containing substantial code can improve this efficiency.
38. Exact Output Must Be Parsed, Not Merely Read
Generated code that looks plausible can still contain syntax errors. Parse it, compile it or run tests.
Textual fluency is not program validity.
39. Code Execution Is World Return
The strongest test of many code representations is whether the resulting program executes correctly under the intended environment.
Runtime behaviour decides whether the representation survived contact with the machine world.
40. Security Raises the Cost of Small Errors
A missing validation call, changed quote or wrong operator can create vulnerabilities. Code-generation systems should use static analysis, tests and permission boundaries appropriate to the risk.
Representation fidelity becomes operational safety.
41. The Code Tokenisation Audit
- Which programming language and version apply?
- What source encoding and line-ending policy are used?
- How does the model tokenizer split identifiers?
- Are operators and delimiters reconstructed exactly?
- Is whitespace syntactic in this language?
- Are comments and docstrings preserved?
- How are string escapes represented?
- Can lexer tokens and model tokens be cross-walked when needed?
- Can identifiers be resolved to declarations and scopes?
- Are AST or concrete-syntax representations available?
- Does retrieval chunk along structural boundaries?
- How much code fits into the model context?
- Is generated output parsed or compiled?
- Are tests run before high-impact execution?
42. What Students Should Remember
- LLM tokens and compiler tokens are different units.
- Identifiers can span several subword tokens.
- Whitespace and punctuation can be program syntax.
- Lexing identifies formal units; parsing builds program structure.
- ASTs represent code more abstractly than source text.
- Repository context resolves names beyond one file.
- Generated code should be parsed, compiled and tested.
43. The Deep Principle
Code shows why tokenisation can never be the whole story. The sequence must preserve exact surface symbols while higher layers rebuild scope, syntax, types, control flow and behaviour.
A code token is only an entry unit. Program meaning lives in relationships among identifiers, operators, scopes and execution states—and one character can change the entire graph.