A library book can be available, on loan, overdue, lost, returned and available again. A machine can be healthy, degraded, failed and repaired. A learner can move among levels of mastery rather than improve in one smooth line. In each case, the next state is uncertain, but it is not arbitrary.
Stochastic processes are mathematical models for quantities or states that evolve under uncertainty. Markov chains are a particularly important family in which the distribution of the next state depends on the present state, not on the entire recorded path once that present state is known.
This idea is powerful because it turns a changing system into a map of states and transitions. It is also easy to misuse. A model can be perfectly Markovian on paper while the real system carries hidden memory that the chosen state description failed to include.
This article uses constructed examples for explanation. It does not claim that any eduKate learner, school, library, machine fleet or public system actually follows the probabilities shown.
1. A stochastic process is a random quantity indexed by time
A single random variable describes one uncertain quantity. A stochastic process describes a sequence or collection of uncertain quantities indexed by time, position or another ordering variable. Daily demand, queue length, machine condition and weather state can all be represented this way.
The index can be discrete, such as day 1, day 2, day 3, or continuous. The state space can also be discrete or continuous. A Markov chain usually refers to a discrete-time, discrete-state Markov process, although Markov models exist in broader forms.
2. The state is the information the model carries forward
Suppose a machine is represented as Healthy, Degraded or Failed. That classification is not the machine itself. It is a model state: the summary of information chosen to predict what happens next.
If time since last maintenance matters but is not included, the three-state model may hide important memory. Expanding the state to include maintenance age can restore information the simpler model omitted. State design is therefore an epistemic decision, not clerical labelling.
3. The Markov property is conditional memorylessness
The Markov property says that, conditional on the current state, the model does not need the earlier sequence to determine the distribution of the next state. NIST’s Dictionary of Algorithms and Data Structures describes a Markov chain as a state system with probabilities attached to transitions.
This does not mean the physical system has no history. It means the chosen state is supposed to contain all history relevant to the transition law. That distinction prevents the phrase “memoryless” from becoming misleading.
4. Transition probabilities are a model of movement between states
Consider a fictional two-state borrowing system: Available and On Loan. Suppose an available book has a 0.30 probability of being on loan the next day and a 0.70 probability of remaining available. A book already on loan has a 0.40 probability of being returned by the next day and a 0.60 probability of remaining on loan.
| Current state | Available next | On Loan next |
|---|---|---|
| Available | 0.70 | 0.30 |
| On Loan | 0.40 | 0.60 |
Every row sums to one because the next state must be one of the listed possibilities under this model.
5. One step becomes many steps through matrix multiplication
If the initial probability vector is [1, 0], the book begins certainly available. Multiplying by the transition matrix gives [0.70, 0.30] after one day. Repeating the multiplication propagates the distribution forward.
After two days, the probability of being available is 0.70×0.70 + 0.30×0.40 = 0.61. The probability of being on loan is 0.39. The calculation does not claim which particular path occurred; it gives the distribution over possible states.
6. A path is different from a state distribution
One realised path might be Available → On Loan → On Loan → Available. Another might remain Available for several days. The transition matrix describes the probabilities governing such paths, while the state distribution describes uncertainty about where the process is at a particular step.
Confusing path and distribution is a common reasoning error. A 60% probability of remaining on loan does not mean every loan lasts 0.6 days or that exactly 60% of every small sample must behave that way.
7. Time-homogeneous is an additional assumption
A time-homogeneous chain uses the same transition probabilities at every step. That may be reasonable for a stable laboratory process and poor for a library with school-holiday surges, examination periods or changing opening hours.
The Markov property and time homogeneity are different assumptions. A process can be Markov while its transition probabilities change with time.
8. Stationary distributions describe long-run balance under specific conditions
A stationary distribution is a probability vector that remains unchanged after applying the transition matrix. For the fictional borrowing chain, let a be the stationary probability of Available. Then a = 0.70a + 0.40(1−a), which gives a = 4/7, about 0.571. The stationary On Loan probability is 3/7, about 0.429.
This is not a prediction that every week will contain exactly those proportions. It is a long-run distribution of the model under its assumptions.
9. Long-run behaviour requires structural conditions
Some chains have multiple closed classes, periodic cycles or absorbing states. In such cases, one simple stationary story may not describe convergence from every starting point.
MIT’s public Markov Chains lecture notes provide a formal route into state distributions and transition powers. A more recent tutorial in MIT Press’s Neural Computation develops the spectral view of finite Markov chains.
10. Absorbing states represent one-way arrival
An absorbing state cannot be left once entered. In a simplified equipment model, Retired might be absorbing. In a project model, Cancelled might be absorbing if restart is outside the model boundary.
The modelling question is whether that one-way representation is actually appropriate. A “Dropped Out” learner state should not be made absorbing if re-entry is possible and relevant.
11. Hitting times ask how long until a state is reached
A hitting time is the number of steps required to reach a specified state for the first time. This turns a transition model into questions such as expected time to failure, expected time to return, or probability of reaching a target before a deadline.
These quantities often matter more to decisions than the one-step transition probability itself.
12. Return times ask when a process comes back
For recurrent states, the return time measures how long it takes to revisit a state after leaving it. In networks and service systems, return behaviour can reveal persistent circulation rather than simple one-way progression.
13. Communicating classes reveal which states can reach one another
Two states communicate if each can eventually reach the other with positive probability. This partitions a chain into structural regions. Some are transient; some trap the process once entered.
This graph view connects directly to How Network Science and Graph Analysis Work: the transition matrix is also a weighted directed graph.
14. Eigenvectors appear because equilibrium is a fixed point
A stationary distribution π satisfies πP = π. In linear-algebra language, it is a left eigenvector associated with eigenvalue one, normalised to sum to one.
This is one reason Markov chains link probability, graphs and linear algebra so deeply. Long-run behaviour can often be studied through the spectrum of the transition operator.
15. Hidden states create hidden Markov models
Sometimes the underlying state cannot be observed directly. A learner’s mastery, machine health or speech phoneme may be latent while we observe test answers, sensor signals or acoustic features.
A hidden Markov model adds an observation process: hidden states transition according to one model and emit observable signals according to another. The task becomes inference about the hidden path or state distribution.
16. Estimating a transition matrix is a statistical problem
If transitions are observed, a simple estimate divides the number of transitions from state i to state j by the total observed departures from i. But this estimate has uncertainty, especially for rare states or short histories.
Missing observations, measurement error and changing transition rules can bias the estimate. Relevant routes include Missing Data Analysis and Measurement Error and Misclassification.
17. The state space can be too coarse
If every learner is classified only as “understands” or “does not understand”, two people with very different histories may be forced into the same state even though their next-step probabilities differ. The model then appears non-Markov because the state lost necessary predictive information.
Refining the state can improve adequacy, but excessive refinement creates sparse data and a huge transition matrix. The right state is the smallest representation that carries the information needed for the decision job.
18. The state space can also be too fine
Tracking every tiny distinction can make the model impossible to estimate and interpret. If each state is rarely revisited, empirical transition probabilities become unstable.
This is a general modelling trade-off between fidelity and tractability.
19. Non-Markov behaviour is sometimes the truth
Duration dependence is a classic example. The probability that a machine fails tomorrow may depend on how long it has already been degraded. If the state records only Degraded and not duration, the next step genuinely depends on history omitted from the state.
One repair is to enlarge the state. Another is to use a semi-Markov or other duration-sensitive model. The correct choice depends on the mechanism and evidence.
20. Markov models are not causal by default
A transition probability describes how states are associated over time under the observed or modelled process. It does not automatically tell us what would happen if an intervention forced the system into a different state.
For intervention questions, the causal structure must be justified separately. See How Causal Inference Works.
21. Forecasting with a transition model inherits structural uncertainty
Even exact multiplication of P many times cannot rescue an incorrect P. Long-horizon forecasts compound uncertainty about transition probabilities, state definitions and possible regime changes.
This connects to Forecasting and Prediction and Sensitivity Analysis and Robustness Checks.
22. Simulation turns the transition law into sample paths
Instead of multiplying distributions analytically, a simulation can repeatedly sample the next state according to the transition probabilities. Many simulated paths approximate distributions of hitting times, occupancies and other quantities.
Simulation introduces Monte Carlo error in addition to model uncertainty. The distinction is developed in Monte Carlo and Simulation-Based Inference.
23. Markov chains can support decisions, but decisions change the transition law
If a controller can choose maintenance, teaching intervention or inventory action, the transition probabilities may depend on that action. The model then becomes a Markov decision process rather than a passive Markov chain.
This is the bridge to dynamic programming: choose the action now while accounting for how it changes future states.
24. A practical model-audit route
- Define the real question and time scale.
- Choose states that carry relevant predictive information.
- Identify which transitions are possible.
- Estimate or justify transition probabilities.
- Check whether probabilities vary with time, duration or hidden conditions.
- Test sensitivity to state aggregation and transition uncertainty.
- Validate short- and long-run behaviour against held-out evidence where possible.
- Keep descriptive transition models separate from intervention claims.
- Record the model edition, data period and domain of validity.
25. The deepest lesson
A Markov chain is not a magical claim that the past does not matter. It is a disciplined proposal that the present state contains the part of the past needed for the next probabilistic step.
When that proposal is adequate, complicated histories become tractable. When it is inadequate, the model’s apparent simplicity hides memory that the world is still carrying.
Sources and further reading
- NIST Dictionary of Algorithms and Data Structures — Markov chain.
- MIT CSAIL — Markov Chains lecture notes.
- MIT Press — A Tutorial on the Spectral Theory of Markov Chains, 2023.
Continue through eduKate: Network Science and Graph Analysis → Monte Carlo and Simulation-Based Inference → Models and Simulations → Research Collections Directory.