The Occam's Razor Machine
The 14th-century friar William of Ockham had a principle: it is futile to do with more what can be done with fewer. In statistics, this creates a genuine tension — a more complex model always fits the data better, so how do you decide when the extra parameters are truly warranted? AIC, BIC, and Bayesian evidence each answer this with a different penalty on complexity. Tune the sliders and watch them agree — or argue.
BIC penalty = k·ln(N) — grows with N, unlike AIC.
High noise hides signal — simpler models win.
Background: The Three Criteria
Penalises the maximum log-likelihood L̂ by twice the number of free parameters k:
AIC = −2 ln L̂ + 2k
The penalty is a fixed 2 per parameter, regardless of sample size. Because AIC only requires evaluating L̂, it can be estimated from any MCMC sampler — the posterior mean log-likelihood or the maximum log-likelihood extracted from the chain serves as a direct plug-in.
Replaces the fixed penalty with one that grows logarithmically with sample size N:
BIC = −2 ln L̂ + k ln N
For N > e² ≈ 7, ln N > 2, so BIC always penalises more heavily than AIC and grows without bound as the dataset expands. Like AIC, it only needs the maximum likelihood value and is therefore straightforward to compute from standard MCMC output.
The probability of the data D under model M, integrated over all parameter values:
p(D | M) = ∫ p(D | θ, M) p(θ | M) dθ
This integral is analytically intractable in general. This visualisation uses the Laplace approximation — ln p(D|M) ≈ ln L̂ − (k/2) ln(N/2π) — which assumes the posterior is approximately Gaussian near its peak. For an exact computation, standard MCMC chains are insufficient: dedicated algorithms such as nested sampling (e.g. MultiNest, dynesty, nautilus) or thermodynamic integration are required. Ratios of evidences between two models give the Bayes factor, interpreted via the Jeffreys scale shown in the insights above.