Audit of grokking metrics showed: early estimates of representation compression are overestimated several times

Researchers in July 2026 published an audit of grokking measurement methods on arXiv — the effect where a neural network suddenly generalizes results after stopping improving accuracy on the training set — and showed that the popular effective rank metric overstates the degree of representation compression in models by orders of magnitude when measured at the exact moment of the grokking transition.

What is grokking and why is it hard to measure

Grokking is a phenomenon in which a neural network trained on tasks like modular arithmetic first memorizes the training set, then — after tens of thousands of training steps — suddenly generalizes the pattern and begins correctly solving examples it has never seen. Researchers often associate grokking with "compression" of the network's internal representation (embedding), and use the effective rank metric to measure this compression.

The audit authors show that measuring effective rank at the exact moment of the grokking transition — a common practice in the field — produces a distorted picture: embedding compression continues for tens of thousands of steps after the network has already generalized the task.

How distorted are the estimates

• On MLP architecture, effective rank measured at the grokking transition overstates the final (converged) value by 3-5 times
• On a transformer trained to full convergence, the overestimate is smaller — 1.3-1.5 times
• On MLPs, such early measurement additionally "erases" information about which representation cells are even being compressed
• The lag between compression and transition to correct answers is roughly equivalent to the time spent on grokking itself — at least 10,000 training steps
• Adding LayerNorm to the transformer reduces the fraction of compression completed by the grokking moment from 0.87 to 0.25

What the architecture verification revealed

A single-variable ablation experiment — changing only one architecture parameter to isolate its effect — revealed that LayerNorm specifically controls the size of this lag: without it, the transformer compresses representation nearly synchronously with the grokking transition (0.87), while with LayerNorm most compression occurs afterward (0.25). A pre-registered control experiment excluded scale invariance as an alternative explanation for the effect.

The authors also tested a narrower, MLP-specific "depth law" linking the normalization budget to the converged compression value — and found that the law does not transfer to transformers and even reverses sign with free weight decay, meaning it works unstably outside the narrow set of conditions for which it was originally derived.

What this means

This work is not just a new metric, but a methodological audit: the authors propose a protocol that separates the moment generalization begins from the moment compression completes, flags censoring cases (when the metric hits a measurement ceiling), and excludes "boundary" representation cells that never fully generalize at all. Their own adversarial test suite caught a false-confidence error in their own code branch — underscoring how easy it is to obtain deceptively confident but incorrect results from shallow grokking measurement. The audit code and instruments are released publicly.

For interpretability researchers, this means reconsidering practice: if a paper on neural network architecture relies on effective rank measured at a single moment in time — right after the model began solving the task correctly — conclusions about representation compression are likely inflated and need rechecking on the converged model. A separate lesson concerns architectural components like LayerNorm: their presence or absence can dramatically change not the fact of grokking itself, but the dynamics of what happens to network representation afterward, meaning comparing experimental results on MLPs and transformers directly, without architectural correction, is risky.