Latest
Product updates, essays and research from Grounds, newest first.
-
Essay
Interpretability as a Guiding PrincipleWhy understanding the internals of neural networks matters, why it is hard, and what Grounds is doing about it.
-
Update
Grounds goes liveWe are privately working with early customers and exploring core research directions.
Attention head 5 in layer 5 of GPT-2 small, reading the same sentence twice. A script found this head in the released weights.
How this was made
The script scored all 144 attention heads of GPT-2 small (124M parameters; layers and heads counted from zero) for induction behavior on 64 random repeated token sequences: how much of each second-pass token's attention lands on the token that followed its first occurrence. This head, layer 5 head 5, scored 0.9254, the highest; the next four were layer 6 head 9 (0.9112), layer 7 head 10 (0.9109), layer 5 head 1 (0.9065), layer 7 head 2 (0.8341).
It then read "Neural networks can be understood completely." written twice, with a leading space so both passes tokenize identically into 7 tokens each. Rows are the token being read; columns are the earlier tokens it can look back at; the darkness of a cell is the attention weight from 0 (an empty outline) to 1 (solid). The upper right is empty because a token cannot look forward. The dark first column is an attention sink, where heads park attention they are not using. The blue outline marks the stripe: for each second-pass token, the cell for the token that followed its first appearance. On this sentence the stripe averages 0.59 and is the darkest cell in 4 of the 7 second-pass rows; the strongest cell in the stripe is "completely" looking back at "." at 0.91. With a pointer, hover any cell for its weight.
Weights: gpt2 at commit 607a30d7, run with transformers 4.56.1 on 2026-09-03. Script: scripts/hero_artifact.py in the site's source; rerunning it regenerates this figure and every number above.