AI systems that are custom, confidential, and understandable.

Grounds researches how to understand the internals of neural networks, and builds custom AI products for businesses.

Attention head 5 in layer 5 of GPT-2 small, reading the same sentence twice. Nobody designed this head. It was found in the released weights by a script, not drawn by hand.

How this was made

The script scored all 144 attention heads of GPT-2 small (124M parameters; layers and heads counted from zero) for induction behavior on 64 random repeated token sequences: how much of each second-pass token's attention lands on the token that followed its first occurrence. This head, layer 5 head 5, scored 0.9254, the highest; the next four were layer 6 head 9 (0.9112), layer 7 head 10 (0.9109), layer 5 head 1 (0.9065), layer 7 head 2 (0.8341).

It then read "Neural networks can be understood completely." written twice, with a leading space so both passes tokenize identically into 7 tokens each. Rows are the token being read; columns are the earlier tokens it can look back at; the darkness of a cell is the attention weight from 0 (an empty outline) to 1 (solid). The upper right is empty because a token cannot look forward. The dark first column is an attention sink, where heads park attention they are not using. The blue outline marks the stripe: for each second-pass token, the cell for the token that followed its first appearance. On this sentence the stripe averages 0.59 and is the darkest cell in 4 of the 7 second-pass rows; the strongest cell in the stripe is "completely" looking back at "." at 0.91. With a pointer, hover any cell for its weight.

Weights: gpt2 at commit 607a30d7, run with transformers 4.56.1 on 2026-09-03. Script: scripts/hero_artifact.py in the site's source; rerunning it regenerates this figure and every number above.

Latest

2026-09-12
Interpretability as a Guiding Principle
Why understanding the internals of neural networks matters, why it is hard, and what Grounds is doing about it.

Grounds Workspace

We make Grounds Workspace, a private and customizable agentic AI system for businesses. It learns from your own best work product and drafts documents in your format. It also shows source documents beside each assertion the AI makes, so humans can easily verify everything.