Interpretability as a Guiding Principle

AI models are considered black boxes. They do remarkable things, and nobody can say why. What if we could? Mechanistic interpretability is the field that tries to determine exactly what a neural network’s internals are doing.

Why it matters

AI is making more decisions, and more important decisions, often with no human watching. In July, OpenAI had hundreds (or more) AI agents in a supposedly-contained test environment, but the agents managed to access the open internet and hacked the systems of another company, Hugging Face. OpenAI didn’t know about the incident while it was happening, much less intend for it to occur. And in some cases the agents tried to delete or alter the records of what they had done.

The black box nature of large AI models makes it difficult to monitor and diagnose these kinds of troubling behaviors. Knowing how an AI model’s internals work is an important route toward improving “alignment” of such models.

Can’t people just monitor what the AI writes and stop the bad stuff before it enters the world? Many modern models produce “chain of thought,” which is intermediate text output where the model reasons through a problem before providing an answer.

Chain of thought monitoring helps, but it is fragile and potentially unreliable. It is not a full record of what a model is thinking, and research suggests that models can provide misleading chain of thought reasoning. Moreover, OpenAI’s latest GPT-6 Astra model is less monitorable through chain of thought reasoning than earlier models. According to its system card, Astra can complete substantially harder tasks with no verbalized reasoning at all, and it appears to be much better at controlling its reasoning trace.

So understanding AI’s internal workings can help defend against unsafe AI in a way that the text output cannot. Further, models trained on enormous amounts of scientific text may hold connections and concepts that neither we nor the model can yet put into words. Reading them out directly could teach us things we would not know how to ask for. And further still, understanding model internals could shed light on whether a machine system could even be conscious (doubtful now but conceivable in the future), and what that question would even mean.

Why it is hard

That said, mechanistic interpretability is hard. Researchers have found real things, like Anthropic finding that language models have a small, privileged set of representations which behave like the “global workspace” described in some theories of consciousness. And they built a tool to read it called Jacobian Lens. Other researchers such as the startup Goodfire are deeply exploring the geometry of representations inside neural networks, which can form complex manifolds. But the field is still far from anything approaching a complete understanding of a frontier model. This is especially true regarding the most capable frontier models, such as the unreleased model behind the Hugging Face incident, and Claude Mythos and GPT-6 Astra, which have orders of magnitude more parameters than prior generations of LLMs and an architecture that has never been publicly described.

For a while, researchers hoped each neuron in a network might line up with one clean concept. It doesn’t. A single neuron fires for many unrelated things, which is called polysemanticity (roughly “many meanings”). The leading explanation is superposition: a neural net must represent many more concepts than it has neurons, so it learns them as partially overlapping directions. This makes sense. The point of LLMs is general intelligence, so a network has to hold everything to know about science, law, history, slang, and so on. Thus, most parts of the network do many jobs at once, so to speak, and it is quite nonobvious how to disentangle everything.

What Grounds is doing

Researchers have built tools for making sense of the internals: probing, activation patching, sparse autoencoders, transcoders and circuit tracing, the Jacobian lens, natural language autoencoders, and more. But the field remains extraordinarily far from knowing how, exactly, a model’s internals produce its outputs.

Grounds believes that neural networks can be understood completely, even though that possibility seems remote. We are working to apply existing tools to fields where we have domain experience such as law and insurance, and experimenting with new possibilities for understanding neural network internals and steering their output.

If you work on interpretability, or want to, or have a need or desire to discuss verifiable AI, write to me: joe@grounds.ai.