Reading the Mind We Made, Part 2: Two Ways to Read a Mind
Two roads lead into the machine. Frontier labs read their own closed models from inside; open source AI interpretability makes the microscope public.
Part 1 established what interpretability even is: the attempt to look past a model’s words and read the mechanism underneath, the features and circuits that actually decide what it says. It ended on a promise. Now we have to cash it, because there is a fact about how that reading gets done that almost nobody says out loud, and once you see it you cannot unsee it.
There are two roads into the machine, and they are diverging.
On one road, a frontier lab reads its own closed model from the inside, with the weights, the activations, and the compute that only it possesses. On the other, anyone with a laptop and a Hugging Face account downloads an open model and runs the same microscope over it. Both roads are real. Both are producing genuine science. But they are drifting apart in a way that defines the whole field: the minds we most need to understand are the ones only their makers can see, and the minds we can see clearest are not the ones making the biggest decisions.
That gap is the story of this post.
Road one: reading a mind you own
Start with the frontier road, because it produced the results you have probably heard about.
When Anthropic published Scaling Monosemanticity in May 2024, it was a landmark. They trained a sparse autoencoder on Claude 3 Sonnet, a real production model, and pulled out on the order of tens of millions of interpretable features: a feature for the Golden Gate Bridge, one for genetic code, one for the concept of inner conflict, one that fired on security vulnerabilities in code. This was interpretability at the scale of a model people actually use, not a toy.
Then they did the thing that made it visceral. They found the Golden Gate Bridge feature, turned its activation up, and released Golden Gate Claude. Ask it anything and it would steer the conversation back to the bridge, insisting on it, working it into answers about soup recipes and its own identity. It was funny, and it was proof. You do not get to clamp a feature and watch the behavior bend unless the feature is real and causal. Correlation does not let you do that. Intervention does.
The work kept going. In 2025 Anthropic’s interpretability team, led by Chris Olah, moved from single features to circuits: how features connect and compute. They built attribution graphs, traced them through the model, and published On the Biology of a Large Language Model, reverse-engineering how the model plans a rhyme before it writes the line, or does arithmetic through a set of parallel approximations rather than the algorithm you were taught in school. Google DeepMind runs a serious mechanistic-interpretability effort of its own, with Neel Nanda among the people driving it. OpenAI has published on sparse autoencoders too. The frontier labs are, genuinely, trying to read their own creations.
Here is the catch, and it is the whole point.
“The public gets the paper. The lab keeps the patient. You can read the writeup of what they found inside the model. You cannot open the model and check.
”
Anthropic even open-sourced its circuit-tracing tooling so others can build attribution graphs. Read that carefully. The tool is open. The model stays closed. You can have the microscope. You cannot have the specimen that matters most. The weights of Claude, of GPT, of Gemini live behind an API, and the activations you would need to run the microscope over them never leave the building. Only the lab holds all three of the things interpretability requires at once: the weights, the internal activations, and the compute to trace them.
So on the frontier road, interpretability is a report from inside a locked room. You are asked to trust the transcript.
Road two: reading a mind anyone can open
Now the other road, which is where interpretability stops being a lab privilege and becomes a public science.
The move is almost stupidly simple. Take a model whose weights are actually published, Llama from Meta, Gemma from Google, Qwen from Alibaba, DeepSeek, GLM, Mistral, and you no longer need permission to look inside. You have the weights. You can run any input and record every activation. You can train your own sparse autoencoders, clamp your own features, trace your own circuits. The specimen is on your bench.
The clearest signal of how far this road has come arrived in December 2025, when Google DeepMind released Gemma Scope 2.
The first Gemma Scope, in 2024, was already a gift: a full suite of open sparse autoencoders for the Gemma family, so anyone could probe the model’s internal features without training the SAEs themselves, the most expensive part. Gemma Scope 2 extended that to the Gemma 3 family and did it at a scale that reframes what “open” can mean. It is described as the largest open-source interpretability release to date: on the order of 110 petabytes of activations processed, with over a trillion parameters trained across the sparse autoencoders. The weights sit on Hugging Face. There is an interactive demo on Neuronpedia. DeepMind’s own framing is the one that sticks: a microscope for LLMs, handed to everyone.
And Gemma Scope 2 is one instrument in a growing kit. The open road has a stack now.
- Pythia, from EleutherAI, is a suite of models released with their training checkpoints, built specifically so researchers can study how capabilities and internal structure emerge over training, not just inspect a finished model. You can watch a circuit form.
- OLMo, from the Allen Institute for AI, goes further than open weights. It is fully open, including the training data. That closes a loop nothing else can: when you find a behavior inside the model, you can trace it back toward the data that produced it. Open weights let you ask what the model does. Open data lets you ask why.
- TransformerLens, from Neel Nanda and the community, is the workhorse library for mechanistic interpretability on GPT-style models, the thing a huge amount of published open interpretability actually runs on.
- nnsight and NDIF, from David Bau’s lab at Northeastern, let researchers remotely inspect the internals of very large models without hosting them locally, so the compute wall is lower. nnterp (2025) adds a standardized interface across model architectures so results transfer.
- Neuronpedia is the platform that ties it together: an open interpretability hub with, from March 2024, the first interpretability API, where features from open models are named, visualized, and queried programmatically.
Put those together and you get something the frontier road structurally cannot offer: interpretability you can reproduce. Someone publishes a claim about a feature in Gemma, and you can download the same weights, the same SAEs, and check it yourself by dinner. That is not a nicety. Reproducibility is the thing that turns a set of impressive results into a science.
On the frontier road you evaluate a claim by trusting the authors. On the open road you evaluate it by rerunning it. Same method, completely different standard of proof.
Layer 2 · Mechanismhow it actually works
To run the microscope you need three things together: the weights (to reconstruct the network), the activations (the intermediate values a specific input produces), and enough compute to record and analyze them. A closed API gives you outputs and nothing else. The activations are computed on the lab’s servers and discarded before anything reaches you.
Open weights hand you all three at once. You run the forward pass yourself, so every activation is yours to capture. You can insert hooks at any layer, clamp any feature, ablate any head, and diff the results. A published SAE suite like Gemma Scope 2 removes the one remaining expensive step, training the feature dictionary, so the barrier to entry drops from “a lab” to “a good GPU.”
Layer 3 · Math & where it breaksgo deeper
Interpretability’s strongest evidence is causal, not correlational. Finding that a feature lights up when the model discusses the Golden Gate Bridge is suggestive. Clamping that feature and watching the behavior change is proof, because you intervened on the variable and measured the effect. Every such intervention needs write access to the model’s internal activations. Closed models permit this only for their owners. Open weights make the causal experiment available to anyone, which is exactly why the open road can accumulate checkable results while the closed road accumulates reports.
You can stop after Layer 1 and still be correct about why open weights change the epistemics, not just the access, just less complete.
The two roads, side by side
Lay them next to each other and the divergence is stark.
| Frontier interpretability | Open-weight interpretability | |
|---|---|---|
| Who can do it | Only the lab that owns the model | Anyone who can download the weights |
| What you can access | Weights, activations, compute (all held internally) | Weights, activations, compute (all in your hands) |
| The specimen | Stays closed; you read the paper, not the model | On your bench; you run the microscope yourself |
| Reproducible by outsiders | No, you trust the writeup | Yes, rerun the same weights and check |
| Examples | Scaling Monosemanticity, Golden Gate Claude, attribution graphs on Claude | Gemma Scope 2, Pythia, OLMo, TransformerLens, nnsight/NDIF, Neuronpedia |
| On the capability frontier | Yes, these are the strongest models | Trailing, but closing |
| Legible to outsiders | Barely | Fully |
Read the last two rows together. That is the entire tension of the field in two lines.
Model organisms: where the methods actually grow up
There is a deeper reason the open road matters, and biology named it a century ago.
You do not learn genetics by starting on a human being. You start on a fruit fly and E. coli, organisms that are small, fast, cheap to run, and shared across every lab so results can be compared. You develop the method on the model organism, and only then carry it to the harder subject. Interpretability borrowed the idea directly. Small, open models are the field’s fruit flies: the place where a new technique gets invented, stress-tested, and understood before anyone points it at a frontier system where the stakes are higher and the doors are locked.
This is not a loose analogy. Hubinger and colleagues made it a research program in 2023 with Model Organisms of Misalignment, arguing for deliberately building small models that exhibit the failure you want to study, so you can dissect it in a system you fully control. If you want to understand deception, you build a tractable model that deceives, and you take it apart. The open road is where that dissection happens, because it is the only road where you hold the whole animal.
The legibility gap
Now name the thing, because naming it is half of seeing it.
Call it the legibility gap: capability and legibility are pulling apart. The models with the most capability are the least legible to anyone outside the lab that built them. The models most legible to everyone are not the ones on the capability frontier. The two properties we most want in the same system, being powerful and being understandable, are drifting onto different machines.
The gap is not a scandal. It is a natural consequence of two things being true at once: the frontier costs a fortune to build, so it lives inside a few companies, and interpretability needs total internal access, which only the owner has. Nobody chose the legibility gap. It fell out of the economics. But it is the fact that shapes everything downstream, and pretending it away helps no one.
“The most powerful minds we have built are the least legible to outsiders. The most legible minds are not the most powerful. Closing that gap is the real work of the next few years.
”
Why now
Why is this the moment the two roads split into view, rather than five years ago?
Because until recently, open models were too far behind for the question to bite. When the best downloadable model was a curiosity, “we can fully inspect it” was a consolation prize. That has changed. The open-weight families are now close enough to the frontier that the gap between what we can understand and what we deploy has become the interesting variable. And the tooling crossed a threshold in the same window: a published SAE suite at the scale of Gemma Scope 2, a working interpretability API in Neuronpedia, a standardized inspection stack in TransformerLens and nnsight, all landing within roughly two years. The method got cheap and shared at the exact moment the open models got good. That is why the legibility gap is visible now and was invisible before.
Where this goes (labeled predictions)
The rest is forecast, so I will label it as forecast. These are my predictions, not established fact.
Prediction: open interpretability tooling becomes standard infrastructure. The way TensorFlow and PyTorch became the assumed substrate for training, a shared interpretability stack, published SAEs, standardized hooks, an inspection API, becomes the assumed substrate for understanding. A microscope any lab can point at any open model, off the shelf. Gemma Scope 2 and Neuronpedia look like the early shape of it.
Prediction: a common open feature standard emerges. Something Neuronpedia-like becomes the shared vocabulary, a public registry where features are named, described, and referenced across models, so “the deception feature in this model” is a thing you can cite and compare rather than rediscover. Interpretability gets a shared namespace.
Prediction: interpretability access becomes a deployment requirement. For high-stakes, audited frontier systems, I expect the demand to shift from “publish a safety paper” to “grant an auditor real interpretability access.” Not open weights for everyone, but structured internal access for accountable review. The legibility gap is not only a scientific problem. Once these systems make consequential decisions, someone will insist on being able to look inside, and the open road is where the methods for doing that are being built.
The open microscope
Here is what changes for you, whoever you are.
Interpretability used to be something you read about. A lab published, you nodded, you moved on. That era is ending. The microscope is on Hugging Face. The SAEs are trained and downloadable. The API answers queries. If you have ever wanted to know what a model is actually doing when it answers you, the honest situation is that for a growing class of capable models, you can now go and find out yourself. Not trust the paper. Check the model.
That is the shift the open road represents. Interpretability is becoming everyone’s science instead of a lab secret, and the legibility gap is the frontier that public science now gets to push on. The frontier models will stay ahead on capability for a while. But the tools to read a mind are escaping the buildings that built them, and tools, once loose, do not go back.
Part 3 takes the next step, from reading to steering: once you can find the feature, you can turn it up, turn it down, and change what the model does. Golden Gate Claude was the party trick. The real question is what happens when clamping a feature becomes a control surface, and who gets to hold it.
