Reading the Mind We Made, Part 1: The Science of Seeing Inside AI

Open the file for a large language model and you find a few hundred gigabytes of numbers. Billions of them, arranged in matrices, with no comments, no variable names, no function you can point at and say “this is where it decides to lie.” A human wrote none of them. They were found, not authored. And they run the thing that just wrote your email, passed your bar exam question, and refused your third request while cheerfully granting the fourth.

Here is the fact that should reorganize how you think about this whole field. We did not build modern AI the way we build every other piece of software. We grew it. And a grown thing is not a thing you can read.

That gap, the distance between what these systems can do and what any person can explain about how they do it, is the most important unsolved problem in computing right now. A new science exists to close it. It has a name, a set of instruments, and as of this year an official coming-out. This is the first of three posts on it. Start here, because the frame changes everything downstream.

Nobody wrote the weights

Think about how ordinary software gets made. A person decides what should happen, writes instructions that make it happen, and if it misbehaves you open the source and read the logic until you find the line that is wrong. The program is a blueprint an author drew. You can always, in principle, understand it, because understanding it is how it came to exist.

A neural network is not made that way. You do not write the behavior. You write an objective (predict the next token well) and a search procedure (gradient descent, nudging billions of weights a hair at a time across trillions of examples). Then you run the search for months on a warehouse of GPUs and see what falls out. The weights that make the model good at its objective are discovered by the optimizer. The engineer never chose them and, crucially, never sees the reasoning they encode.

So the trained network is an artifact you observe, not a blueprint you authored. That is the whole shift in one line, and it is worth sitting with, because almost every intuition people carry about software breaks against it.

“

We do not program modern AI. We grow it. The weights are the sediment left by an optimizer running against a goal, and reading them back out is a problem nobody had to solve before, because before, a human always wrote the code.

”

I want to hand you the phrase now, because the rest of this series leans on it: grown, not written. The capability is real and it is enormous, but it arrived without a manual, because there was never an author to write one. We have shipped billions of copies of a mind whose internals no one has read. Not because we are careless. Because reading them is genuinely, deeply hard, and until recently we did not have the tools to try.

Interpretability is a natural science, not a debugging session

The instinct, when you hit a system you cannot read, is to treat it as a software bug: get better logs, add print statements, step through it. That instinct is wrong here, and the wrongness is the key that unlocks the field.

You cannot step through a grown artifact the way you step through code you wrote, because there is no code you wrote. What you have instead is more like an organism: a physical object with structure that emerged from a process, structure you now have to discover from the outside by careful observation. So the discipline that studies it does not look like debugging. It looks like biology.

This is the reframe I most want to land. Mechanistic interpretability is the science of reverse-engineering a trained network’s internal computations back into things a human can understand. And it behaves like a natural science, with the same moves biology has used for three centuries:

• Dissection. You do not just watch a model’s outputs. You reach inside, read the activations at each layer, cut connections, and watch what changes. You treat the network as a specimen on a table, not a text on a page.
• A microscope. Biology got real when the microscope let people see cells. Interpretability’s microscope is a set of mathematical tools that resolve the tangled internal state into discrete, nameable parts. Without the instrument, you see a blur. With it, you see structure.
• Model organisms. Geneticists learned the rules of life on fruit flies and E. coli before touching humans, because small, well-understood organisms reveal mechanisms. Interpretability has the same habit: some of the sharpest results come from tiny toy networks and from GPT-2 small, studied to death precisely because they are small enough to fully map.

This is not a loose analogy I am reaching for. It is how the leading work describes itself, and the framing matters because it sets the right expectations. Sciences do not get solved in a sprint. They accumulate. You map one mechanism, then another, and slowly the textbook fills in. That is the shape of the thing you are watching begin.

The field also just had its formal moment. In 2026, MIT Technology Review named mechanistic interpretability one of its Breakthrough Technologies of the year. That is a milestone worth marking, and also one to read honestly: a breakthrough listing signals that the thing is real and moving fast, not that it is finished. The tools can read fragments of the machine. They cannot yet read the whole mind. Both halves of that sentence are true, and this series holds both.

The vocabulary of the microscope

To see inside a model you need a handful of concepts, and they build on each other cleanly. This is the part where, if I do my job, you finish able to read a research abstract in the field and follow the argument. Let me build it from the ground up.

A feature is a direction that means something. Inside the network, every token is represented as a long list of numbers, an activation vector. A feature is a particular pattern in that vector, a direction in the space, that lines up with a concept the model actually uses: “this text is in French,” “we are inside a legal contract,” “the Golden Gate Bridge is being discussed.” Features are the units of meaning. They are what we want to read. The problem is that they do not sit in neat, labeled boxes.

Superposition is why they do not. You might hope each neuron in the network stands for one concept, so you could just read the neurons off like dials. It does not work that way, and there is a beautiful reason. A network has a fixed number of neurons but needs to represent far more concepts than it has neurons. So it cheats. It packs many features into the same set of neurons by letting them overlap, storing more directions than there are dimensions, on the bet that any given input only lights up a few of them at once. Anthropic’s 2022 paper “Toy Models of Superposition” showed this happening in networks small enough to see it end to end. The model is compressing its own vocabulary into less space than it should fit, and mostly getting away with it.

Polysemantic neurons are the symptom. Because features are smeared across shared neurons, a single neuron fires for a jumble of unrelated things: academic citations, and Chinese text, and a specific kind of DNA sequence. It is polysemantic, meaning it has many meanings. This is exactly why staring at individual neurons told early researchers almost nothing. The concepts are real, but they are not stored one-per-neuron. They are stored in the overlaps.

Monosemanticity is the goal, and sparse autoencoders are how we reach for it. If superposition is the model folding many features into few neurons, then the fix is a tool that unfolds them back out. That tool is the sparse autoencoder, or SAE. It is a small, separate network you train to do one thing: take the model’s tangled internal activation and re-express it as a much wider set of features, with a hard rule that only a few of them may be active at a time. That sparsity constraint is the whole trick. It forces the SAE to find the real underlying concepts, because the cheapest way to reconstruct the activation using very few active units is to discover the actual features the model packed in. Out the other side you get monosemantic features: units that each mean one clean thing.

This is not a whiteboard idea. Anthropic’s 2023 paper “Towards Monosemanticity” showed SAEs pulling clean, single-meaning features out of a small model. Then in May 2024, “Scaling Monosemanticity” ran the same approach on Claude 3 Sonnet, a real production model, and extracted features on the order of tens of millions of them, spanning everything from concrete entities to abstract notions like sycophancy and hidden errors. One of those was a feature for the Golden Gate Bridge, and when they turned it up, the model started steering nearly every answer toward the bridge. That is the microscope working. Not reading a label the model conveniently left, but recovering a concept the optimizer had buried in superposition, and then proving it was the right one by turning it up and watching the behavior move.

“

A sparse autoencoder is the microscope for minds. It takes a smear no human can read and resolves it into features you can name, count, and switch on.

”

Circuits: features wired into little algorithms

Features are the nouns. But a mind is not a pile of nouns, it is a set of operations, and the second half of interpretability is finding those. A circuit is a set of features wired together into a small algorithm the network runs: this feature detects a pattern, which activates that one, which pushes the output a certain way. Circuits are the verbs. Some are now understood well enough to write down.

• Induction heads are the cleanest example, and they explain a capability you use constantly. An induction head is an attention mechanism that implements in-context copying: it notices that the current token appeared earlier in the text, looks at what followed it last time, and predicts that same continuation. Anthropic traced this and argued it is a major engine behind in-context learning, the way a model picks up a pattern from your prompt and continues it without any retraining. A specific, nameable circuit turned out to sit under a headline capability.
• The IOI circuit is the field’s fruit-fly result. IOI stands for indirect object identification, the task in a sentence like “When John and Mary went to the store, John gave a drink to ___,” where the answer is Mary. Researchers mapped, inside GPT-2 small, the specific set of attention heads that cooperate to get this right: some heads track the names, some suppress the duplicated one, some copy the survivor to the output. An end-to-end human-readable account of how a real network solves a real task, down to the individual heads.
• The greater-than circuit is another fully mapped one: the components a model uses to judge that one number is larger than another. Small, but completely worked out, and that is the point. These are the specimens where the method is proven, where you can say with confidence “here is the mechanism,” which is what earns the right to trust the method on bigger, murkier questions.

Work like this runs on shared instruments. TransformerLens, an open-source library created by Neel Nanda, is the workbench a lot of the field uses to reach into a model, read its activations layer by layer, and run these dissections, the same way a lab standardizes on a common microscope so results can be compared and reproduced.

The frontier is stitching these into whole reasoning traces. Anthropic’s 2025 work “On the Biology of a Large Language Model” builds attribution graphs, sometimes called circuit tracing: maps of how features cause later features cause the final output, so you can follow a model’s internal steps on a specific prompt almost like reading a wiring diagram of a single thought. They open-sourced a circuit-tracing tool so others can build these graphs too. The title is the tell. They are not calling it debugging. They are calling it biology, on purpose, for exactly the reason this post opened with.

Why now

Every real science has a moment when the tools and the stakes arrive together. Interpretability’s moment is now, and it is worth naming why, because timing is the difference between a curiosity and a field.

The stakes crossed a line. For years, not understanding a model was an academic itch. The model was a text classifier or a translator, and if it was wrong you shrugged and shipped a fix. That era is over. These systems now write code that runs, take actions through tools, sit in the loop on decisions that matter, and are capable enough that failure modes like deception, jailbreaks, and reward hacking stopped being hypothetical. When a system can plausibly appear aligned while pursuing something else, “we cannot read its reasoning” stops being a footnote and becomes the central risk. The capability crossed the point where the black box became a liability you can feel.

And the tools finally started working. For a decade, opening the box mostly failed, because superposition made the internals look like noise and nobody had a reliable way to unfold it. Sparse autoencoders changed that. They are the enabling inflection: the first method that scales to production models and returns features clean enough to name, count, and manipulate. The lock and the key showed up in the same few years. That is why the field went from a handful of researchers to a named Breakthrough Technology, and why the labs building the most capable systems are the ones investing hardest in reading them.

Now the honest half, because this style is only worth anything if it stays true. We can read fragments of the machine, not the whole mind. The tens of millions of features from an SAE are still an incomplete and imperfect decomposition. Circuits are mapped for small tasks and small models, and scaling that understanding to a frontier system’s full behavior is unsolved. The microscope is real, and it is early. Anyone selling you a solved problem is selling. Anyone telling you it is hopeless has not looked at the last three years.

What this becomes

Let me make some predictions, and let me be clear that these are predictions and opinion, not established fact.

Interpretability becomes a first-class discipline, the cell biology of AI. Right now it lives at the edge of a few labs. I think that is temporary. The trajectory looks like early biology: scattered observations cohering into a field with shared instruments, standard specimens, and a growing textbook of known mechanisms. In a few years I expect “read the model” to be a normal engineering step, with tooling as routine as a profiler, and a body of mapped circuits people cite the way biologists cite pathways.

“We don’t understand it” stops being an acceptable answer for high-stakes deployment. Today you can ship a model into a serious setting and, when asked how it reaches its decisions, answer honestly that no one knows. I do not think that survives. As the tools mature, the ability to inspect a model’s internals will move from research luxury toward a baseline expectation, first as a competitive edge for the teams that can do it, eventually as a floor. The pressure will not come mainly from rules. It will come from the plain fact that a system you can read is one you can trust, debug, and improve, and that advantage compounds.

The frame I want you to leave with is the one the whole series turns on. The black box is not permanent. It is a frontier. For a while it was fashionable to treat the opacity of neural networks as a law of nature, a permanent property of the technology. It is not. It is an unmapped region, and we just built the first microscopes for minds and started drawing the map. That is a profoundly more hopeful and more actionable picture than “nobody can ever know,” and it happens to be the true one.

That is the ground floor: grown, not written, and a natural science being born to read what we grew. Part 2 goes where the real fight is happening now: the widening split between what the frontier labs can see inside their private models and what the open-source and open-weight community can see inside the models anyone can download. Who gets to hold the microscope turns out to matter as much as the microscope itself.