A molecule can be represented as a name, formula, line notation, graph, 3D geometry or sequence of model tokens. Chemical tokenisation is the problem of choosing which symbols and structural units should become machine-addressable while preserving enough chemistry for the task.
This matters because chemistry is relational. Carbon is not meaningful in isolation from its bonding environment, charge, stereochemistry and surrounding structure. A token sequence is useful only if it preserves a reliable route back to the molecular graph.
This article continues the eduKateSingapore Representation and Tokenisation series. It is an educational representation article, not a laboratory synthesis protocol.
The Chemical Representation Route
MOLECULAR STRUCTURE → NAME / FORMULA / GRAPH / STRING → TOKENISATION → ATOM / BOND / FRAGMENT / STRING TOKENS → MODEL → PREDICTION / GENERATION → VALIDITY + CHEMISTRY CHECK → MOLECULAR GRAPH → EXPERIMENTAL OR DATABASE RETURN
1. Molecular Formula Is a Very Coarse Representation
A formula such as C₂H₆O tells us element counts but not full connectivity. Ethanol and dimethyl ether share the same molecular formula and have different structures and properties.
Formula tokens preserve composition while omitting graph structure.
2. Structural Formula Adds Connectivity
Writing CH₃CH₂OH reveals more relational structure than C₂H₆O.
The representation becomes more informative and longer.
3. Molecular Graphs Are Natural Chemical Representations
Atoms can be nodes and bonds edges. Atom type, charge and stereochemical attributes can live on nodes or edges.
This makes chemistry a direct application of Graph Tokenisation.
4. Atoms Are Primitive Chemical Tokens
A model can represent each atom as one token-like node with element, charge, aromaticity and other features.
Atom identity is only the beginning; chemical behaviour depends on local structure.
5. Bonds Are Relationship Tokens
Single, double, triple and aromatic bonds encode different relationships among atoms.
A molecular representation that preserves atoms while losing bond type is chemically incomplete.
6. SMILES Serialises a Molecular Graph Into a String
SMILES is a widely used line notation that walks a molecular graph and encodes atoms, branches, ring closures and bond information in a linear string.
It makes molecules compatible with string-processing systems while introducing a serialization grammar.
7. One Molecule Can Have Multiple Valid SMILES Strings
Different graph traversal orders can produce different SMILES strings for the same molecular graph.
String identity is therefore stricter than molecular identity.
8. Canonical SMILES Chooses One Standardised Serialization
Software can apply canonicalisation rules to generate a reproducible string for a molecular graph.
Canonicalisation helps indexing and deduplication while remaining software- and rule-dependent.
9. Randomised SMILES Can Be Data Augmentation
Training on several valid SMILES traversals for one molecule can reduce overdependence on one serialization path.
This parallels subword regularisation: several surface representations can point to one underlying object.
10. SMILES Tokens Need Chemical Grammar Awareness
Cl is one chlorine atom symbol, not C followed by l. Bracket expressions can contain charges, isotopes and stereochemical information.
Naive character tokenisation can split chemically meaningful units awkwardly.
11. Regex or Grammar-Aware Tokenisers Preserve Atomic Symbols
Chemical language models often use tokenisation rules that recognise multi-character atom symbols, bracketed atoms and ring or branch syntax.
The tokenizer can therefore better align surface units with the notation grammar.
12. BPE Can Learn Larger Chemical Fragments
Subword algorithms can merge frequent SMILES substrings into reusable pieces.
Those pieces may correlate with common functional groups or may simply be statistically frequent serialization fragments.
13. Learned Chemical Subwords Are Not Automatically Functional Groups
A token can cross boundaries chemists would normally use when describing functional groups.
Statistical compression and chemical ontology remain different layers.
14. Ring Closure Tokens Encode Nonlocal Graph Edges
SMILES uses matching ring labels to connect atoms that can be far apart in the linear string.
The model must reconstruct a graph edge from nonlocal sequence syntax.
15. Branch Parentheses Encode Graph Topology
Parentheses indicate branches away from the current traversal path.
One missing parenthesis can alter or invalidate the molecular representation.
16. Stereochemistry Is Small Syntax With Large Consequence
Chiral and geometric stereochemical markers can distinguish molecules with different biological or physical behaviour.
Tokenisation and decoding must preserve these symbols exactly when stereochemistry matters.
17. Charges and Isotopes Need Structured Atom Tokens
Bracketed atom expressions can specify formal charge, isotope or hydrogen count.
Flattening them into generic element tokens loses chemical state.
18. SMILES Has a Validity Problem in Generative Modelling
Arbitrary token sequences are often not valid SMILES strings or do not map cleanly to chemically valid graphs.
A language model can therefore generate syntactically plausible-looking strings that fail molecular parsing.
19. SELFIES Was Designed for Robust Molecular Strings
SELFIES introduced a self-referencing molecular string representation designed so valid SELFIES strings decode to valid molecular graphs under its grammar.
The important representation move is that chemical validity constraints are built into the language rather than left entirely to the generative model.
20. Robust Syntax Changes the Search Space
If every generated token sequence maps to a valid graph, a generative search can spend less effort producing unparseable strings.
Representation design can therefore enforce useful constraints before reasoning begins.
21. Valid Molecule Does Not Mean Useful Molecule
A structurally valid graph can still be unstable, impractical, toxic or irrelevant to the task.
Syntax validity is only one gate in chemical evaluation.
22. Group SELFIES Shows Another Granularity
Later work such as Group SELFIES explored robust fragment-level tokens, allowing functional groups or larger substructures to become representation units.
This moves chemical tokenisation upward from atoms toward chemically meaningful groups.
23. Fragment Tokens Can Improve Interpretability
A token corresponding to a known substructure can be easier for chemists to interpret than arbitrary character fragments.
The trade-off is a larger, domain-specific vocabulary.
24. Functional Groups Are Higher-Level Chemical Tokens
Hydroxyl, carbonyl, amine and aromatic ring structures recur across many molecules.
Representing these groups directly can align model units with common chemical reasoning.
25. Molecules Are More Than 2D Graphs
Conformation, geometry, solvent and environment can change molecular behaviour while connectivity remains unchanged.
String and graph tokens therefore preserve only part of molecular reality.
26. 3D Coordinates Add Conformation
Atomic positions can be represented as a 3D point set or molecular geometry.
This connects chemical representation to 3D Tokenisation.
27. One Molecule Can Have Several Conformers
Rotatable bonds allow different spatial arrangements with the same connectivity.
Canonical graph identity and conformational state should remain separate.
28. Protein–Ligand Context Adds Relational Meaning
A molecule’s usefulness can depend on interactions with a biological target.
Isolated molecular tokens are not enough for interaction prediction.
29. Reaction Tokenisation Adds Before-and-After Structure
A reaction can be represented through reactants, products, reagents and transformations.
The central token-like unit can become a changed bond or reaction centre rather than a whole molecule.
30. Atom Mapping Creates Identity Across a Reaction
Mapping labels can show which atom in the reactant corresponds to which atom in the product.
This preserves entity identity through structural transformation.
31. Reaction Strings Can Be Tokenised Like Language
SMILES representations of reactants and products can be separated by special delimiters and passed to sequence models.
But the model must learn chemical conservation and transformation rules above the string syntax.
32. Property Prediction Needs Task-Aligned Representation
Some properties depend mostly on local functional groups; others on global topology or 3D conformation.
No one tokenisation is universally sufficient.
33. Molecular Embeddings Compress Whole Structures
A molecule can be mapped into one vector for similarity search or prediction.
This is far coarser than atom or string tokens and needs a route back to the source structure for interpretation.
34. Similarity Depends on Representation
Two molecules can be similar under fingerprints, graph embeddings or 3D shape while differing under another metric.
“Chemically similar” should always name the representation and task.
35. Token Frequency Is Not Chemical Importance
A common fragment can be chemically mundane while a rare motif is biologically crucial.
Tokenizer statistics should not be mistaken for scientific significance.
36. Validity Checks Need a Chemistry Toolkit
Generated strings should be parsed into molecular graphs and checked under formal valence and representation rules before scientific interpretation.
Text fluency is not molecular validity.
37. Scientific Claims Need Experimental or Database Return
A generated molecule predicted to have a property remains a model claim until supported by appropriate computation, trusted reference data or experiment.
Representation confidence must not become scientific certainty.
38. The Chemical Tokenisation Audit
- What molecular object is being represented?
- Is the source formula, SMILES, SELFIES, graph or 3D geometry?
- What tokenizer preserves multi-character atom symbols and bracket syntax?
- How are branches and ring closures represented?
- Is stereochemistry preserved?
- Are charge and isotope states preserved?
- Are tokens atoms, characters, subwords or fragments?
- Does one molecule have several valid serialisations?
- Is canonicalisation required?
- Can generated strings always be parsed?
- Does syntactic validity imply meaningful chemistry? It does not—what further checks are required?
- Is 3D conformation relevant?
- Are reaction atom identities mapped across transformations?
- Can every model token return to the molecular graph?
- What external evidence validates the scientific claim?
39. What Students Should Remember
- Molecules can be represented as formulas, strings, graphs or 3D structures.
- SMILES linearises a molecular graph.
- One molecule can have several valid SMILES strings.
- Chemical tokenisers should preserve atoms, bonds and stereochemical syntax.
- SELFIES builds validity constraints into the molecular string representation.
- Fragments and functional groups are higher-level chemical tokens.
- 2D connectivity does not capture all 3D molecular behaviour.
- Generated structures require chemistry validation and scientific evidence.
40. The Deep Principle
Chemical tokenisation succeeds when a finite symbolic language can move cleanly back and forth to molecular structure without pretending that syntax alone is chemistry.
The token can name an atom, a bond or a fragment. The chemistry lives in the valid structure, spatial arrangement and physical behaviour that those tokens jointly represent.