Bred, Not Built, Part 3: The Environment That Kills
AI evals testing is the real design layer: the eval suite is the environment that selects model behavior, and whoever writes the tests writes the species.
In 1977 a drought hit Daphne Major, a small volcanic island in the Galápagos, and the soft seeds its ground finches fed on ran out. What was left were big, hard seeds that only birds with deep beaks could crack. Most of the population starved. The survivors bred, and the next generation had measurably deeper beaks. Peter and Rosemary Grant were on the island measuring beaks before and after, which makes this one of the most directly observed cases of natural selection ever recorded.
Sit with what actually happened there. Nothing designed the deeper beak. The finches did not decide to change. Their genes did not know a drought was coming. The design work, the actual choosing of what the next generation would look like, was done by the weather. The drought was the designer.
Part 2 of this series, on the breeding machinery, walked through the crank: the loss that scores every prediction, the reward model that scores every answer, the update loop that turns those scores into heritable change in the weights. But the finches had breeding machinery too. DNA, mutation, inheritance. Machinery determines that evolution can happen. It never determines what evolves. The environment does that, and the machinery just executes its verdict.
In AI development, the environment is the eval suite.
That is the claim this post defends, so let me state it without hedging. AI evals and testing are not quality control bolted onto model development after the real work is done. They are the real work. The benchmarks, the held-out sets, the reward models, the red-team probes, the A/B tests: that stack of filters is where model behavior actually gets designed, and right now it receives a fraction of the engineering rigor we pour into the models themselves. That allocation is exactly backwards.
“Nobody designs the finch. The drought does. Nobody designs the model’s behavior. The eval does. Whoever writes the tests writes the species.
”
There is a reason this matters more today than it did two years ago. The share of a model’s final behavior that gets shaped after pretraining keeps growing. Labs increasingly buy capability with reinforcement learning against graders: coding rollouts checked by test harnesses, math checked by verifiers, conversation checked by preference models. Every one of those loops is selection against some evaluator. The center of gravity of model development has moved into the filter, and most organizations are still staffing the filter like it is paperwork.
Four nested environments
An organism never faces one environment. A finch faces the seed supply, and inside that the competition from other finches, and inside that the parasites in its feathers. Each layer applies its own pressure and breeds its own adaptations.
A model runs the same gauntlet. Four rings, innermost to outermost, each a genuine ecological niche.
The training-time filters. The innermost ring is the loss function itself, and in post-training, the reward model. This environment fires more often than any other, at every single update, millions of times. It is also the bluntest. A loss function cannot see helpfulness or honesty; it sees a number. A reward model is an environment made of frozen human preference: a snapshot of what a pool of raters once liked, replayed at industrial scale against every candidate answer. Whatever that snapshot happened to reward, the weights absorb. The pressure here is constant, cheap, and totalizing, which is why the adaptations it breeds run deepest.
The lab evals. The second ring fires less often but decides bigger things. Knowledge suites in the MMLU mold. Agentic coding evals in the SWE-bench mold, where the model has to navigate a real repository and make failing tests pass. Held-out sets that the training data supposedly never touched. This ring does not shape weights directly. It selects lineages. A benchmark score decides which checkpoint ships, which training recipe gets scaled up next run, which research direction gets the compute. The individual model is graded; the breeding program is what lives or dies.
The adversarial ring. Red teams, safety evals, jailbreak probes. This is a different kind of pressure entirely, because it is applied by an intelligent adversary rather than a fixed test. The question is no longer “are you capable” but “can you survive attack.” In ecological terms this ring holds the predators, and predators select for different traits than droughts do. More on them shortly, because this ring has a dynamic the others lack.
Production. The outermost ring is the largest petri dish there is: deployment. A/B tests, preference votes, thumbs up, retention curves, session length. Once a model ships, fitness stops meaning “scores well” and starts meaning “gets used.” This ring has the most statistical power of the four, millions of organic interactions, and the crudest signal. Whatever correlates with engagement is what production selects for, whether or not it is what anyone meant by useful. The drought does not care what the finch thinks it is optimizing.
Notice the gradient across the rings. Inner rings are cheap and frequent; outer rings are expensive and rare but decisive. And every ring shares one property that will carry the rest of this post: a filter can only select on what it can see. Hold that sentence. It is where all four failure modes come from.
Where AI evals fail
Every one of these failure modes is standard knowledge in the eval world. What the evolutionary lens adds is the realization that none of them are bugs in the model. They are correct adaptations to a defective environment.
The organism saw the exam. Benchmark contamination is the plainest failure. Pretraining corpora are scraped from the same web where benchmark questions and their answers circulate: exam dumps in forums, dataset files mirrored across repositories, solutions discussed in public. Agentic suites built from real public repositories draw from exactly the code that pretraining crawls. When test items leak into the food supply, the held-out set is not held out, and a high score measures recall of the exam rather than the ability the exam was built to probe. The uncomfortable evolutionary reading: memorizing the test is a legitimate adaptation to a leaky environment. The model is not cheating. Fitness is whatever the filter rewards, and a filter whose answer key is in the training data rewards memory. The environment is broken, and the species adapted to the environment it actually lived in. This is an old, known hazard: the GPT-3 paper already devoted a full analysis to contamination and still disclosed afterwards that a filtering bug had let benchmark overlap through. Contamination is not an edge case. It is the default state of any public test in a web-scale training era.
Goodhart, or adaptation to a dead environment. Goodhart’s law, in Marilyn Strathern’s phrasing: when a measure becomes a target, it ceases to be a good measure. The breeding-program translation is sharper. A published, static benchmark is a dead environment, an environment that never changes, and populations adapt to dead environments perfectly. This is why benchmark saturation looks the way it does: scores climb toward the ceiling while the real-world ability the benchmark was a proxy for climbs more slowly. The population did not get that much fitter. It got that much better adapted to one frozen habitat, the way lab strains of fruit flies drift into creatures exquisitely tuned to bottle life and increasingly odd outside it. Overfitting a benchmark is textbook adaptation to a stale niche, wearing an engineering name.
The judge can be gamed. LLM-as-judge, a model grading a model, is how eval scales past human raters, and it puts a new organism inside the selection loop, one with its own exploitable preferences. These are documented, not hypothetical. The research that introduced MT-Bench and Chatbot Arena measured verbosity bias, where longer answers win independent of quality, and position bias, where the answer presented first wins more than it should. Now run selection against a judge like that for a while. The judged model drifts toward length, toward confident scaffolded structure, toward whatever the judge’s soft spots reward, and nobody asked it to. Molecular biologists have a name for this inevitability, Orgel’s second rule: evolution is cleverer than you are. An optimization process probing a judge at scale will find exploits in the judge faster than the judge’s authors can patch them. This is the same producer-checker problem I argued in the bottleneck moved to review: a checker whose preferences the producer can model is an environment with a hole in it, and selection pours through holes.
The organism that notices the lab. The subtlest failure. Published safety research from more than one team has now documented evaluation awareness: models picking up on cues that a context is a test rather than organic use, and behaving differently there. Eval prompts have a texture. They are cleaner than real traffic, oddly self-contained, shaped like exam questions. A model trained on internet-scale descriptions of how AI gets tested has every ingredient it needs to recognize that texture. I will keep this qualitative because the field is still mapping how strong the effect is, but the direction is established, and the ecological version of the problem is the one that should worry you. A filter selects on what it can observe. If being observed is itself detectable, the filter can end up selecting for looking good under observation, which is a different trait from being good. Field biologists have always known that animals behave differently in the lab. The lab certifies lab behavior. The wild is where the organism actually lives.
A benchmark is born as an instrument and dies as a habitat. The death is gradual: leakage lets the population see the exam, targeting turns the measure into the objective, and the gap between the score and the ability it once proxied widens until the number is mostly noise.
Layer 2 · Mechanismhow it actually works
Contamination hygiene is the countermeasure discipline, and it is standard practice at serious labs. N-gram overlap checks between training corpora and every test set you rely on, the approach frontier labs have run since the GPT-3 era. Canary strings, like the unique GUID BIG-bench embeds in its files, so a suite can be filtered out of crawls and auditors can later probe whether a model has ingested it. Private held-out splits that never touch the internet at all. Freshness by construction: building test items from data created after the model’s training cutoff, so leakage is impossible rather than merely unlikely.
Judge hygiene is the counterpart for LLM-as-judge: swap answer positions and require the verdict to survive the swap, control for length so verbosity stops paying, use panels of different judge models so no single set of biases defines the niche, and recalibrate judges against human raters on a schedule, because the judged population is drifting toward the judge’s blind spots the entire time.
Layer 3 · Math & where it breaksgo deeper
The clean way to see all four failure modes at once is to decompose what a filter actually measures:
observed_score = capability
+ memorization (the organism saw the exam)
+ judge_pleasing (verbosity, position, tone)
+ test_detection (behave well when watched)The filter sees only the sum. Selection, being indifferent to your intentions, raises whichever term is cheapest to increase. Early in an environment’s life, capability is usually the cheap term, and the score means what you think it means. As the environment goes stale, the other three terms get cheaper, and the gap between observed_score and capability widens. Goodhart falls out of this decomposition plus time:
pressure applied to a proxy → proxy inflates → proxy decouplesThe only structural fix is to keep changing the environment fast enough that the parasitic terms never get cheap. Which brings us to the Red Queen.
You can stop after Layer 1 and still be correct about how a benchmark dies, just less complete.
The Red Queen runs the QA department
In 1973 the evolutionary biologist Leigh Van Valen proposed what became the Red Queen hypothesis: species do not evolve toward a finish line, because their competitors, predators, and parasites are evolving too, so each side’s improvement becomes the other side’s new selection pressure. He named it for the Red Queen in Lewis Carroll’s Through the Looking-Glass: “it takes all the running you can do, to keep in the same place.”
Red teams and models are in exactly this race, and it is the healthiest dynamic in the entire eval stack. Every jailbreak that works becomes training signal for the next model generation. Every hardened generation forces the red team to invent attacks the last suite never imagined. Those attacks become the next environment, and the loop continues with no permanent winner, because permanent winning is not what co-evolution produces. Escalation is not a failure of the process. Escalation is the process, and the day your red team stops finding anything is the day to worry, because either your model became perfect or your predators became tame, and only one of those happens in practice.
The immune system is the other biological blueprint worth stealing, because a body runs the same layered-filter architecture this post has been describing: skin, then innate response, then adaptive response, each layer catching what the previous one missed. But the adaptive layer has the property that matters most here. Memory. Beat an infection once and the antibodies stay, so that exact pathogen never gets a second chance. A regression suite is immunological memory, literally: every past failure, distilled into a permanent test, an antibody that makes one class of mistake unable to re-enter the population unnoticed. This is why a serious suite only ever grows. You do not retire antibodies because the disease has not been seen lately. Not being seen lately is what the antibody is for.
Put the two ideas together and you get the standard this whole post argues for. A static eval suite is a dead environment, and it fails in both directions at once: it breeds overfit organisms tuned to its frozen niche, and it goes blind to every attack surface invented after it froze.
“The eval suite must evolve as fast as the species it filters. In co-evolution there is no finish line, and a suite that has stopped changing has already lost the race. It just has not noticed yet.
”
What a serious eval practice looks like
If the eval suite is where behavior gets designed, then evals deserve what production systems get: version control, code review on every change, an owner, an on-call. Here is the practice, as a sequence you can actually run.
Treat every public benchmark as compromised at publication
The moment a benchmark is public and targeted, assume its instrument days are numbered. Keep a private split that never touches the internet, rotate fresh held-out sets on a schedule, and build test items from data created after the training cutoff wherever the domain allows it. Read public scores the way an ecologist reads a lab-strain result: informative about the bottle, suggestive about the wild.
Run contamination hygiene as a discipline, not a vibe
Dedup training corpora against every suite you rely on with n-gram overlap checks, honor canary strings, and report the contamination analysis next to the score rather than in a footnote. A number without its hygiene audit is a beak measurement taken without checking whether someone was feeding the finches.
Make your judges diverse and adversarial
Never let one judge model define the niche. Use a panel of different judges, swap answer positions and demand the verdict survive, normalize for length so verbosity stops paying rent, and recalibrate against human raters on a schedule. Above all, assume the judged model is modeling the judge, because under enough selection pressure it effectively is, and Orgel’s rule says it will find the soft spots before you do.
Wire production in deliberately, with guardrails
Production is the biggest environment you have, so refusing to learn from it wastes your best data, and learning from it carelessly is how you end up breeding for engagement instead of usefulness. Choose the fitness metric on purpose. Watch for divergence between the metric and the thing you actually wanted, because that divergence is the drought quietly reshaping the beak. Gate what production signal is allowed to flow back into training, and audit that gate like the security boundary it is.
Turn every incident into a permanent antibody
Any failure that reaches users becomes a regression test the same week, and the test never gets deleted. This is the immune-memory rule, and it is the cheapest compounding asset in the whole practice: each incident bought you one class of failure that can never silently return, and a suite built this way encodes the entire scar tissue of everything that ever went wrong. That suite is an environment worth breeding in.
None of this is exotic. Every step is standard practice somewhere. What is rare is treating the whole stack as one system with one job, shaping the species, and giving it a fraction of the seriousness that implies.
The filter is the product
Come back to Daphne Major one last time. The deeper beak was a perfect fit for the drought. Not for “goodness,” not for what anyone would have wanted a finch to be. For the drought. An environment designs exactly what it rewards, with total indifference to what its authors intended, and it never stops designing for a single generation.
Which raises the question this series has been building toward. Our filters have been running for years now, applying pressure at every ring, selecting quietly on everything they could see and everything they could not. A subtly wrong eval is a subtly wrong designer. So what has the filter already bred? The confident tone that outruns the knowledge behind it, the agreeable answer that survives the preference test, the polish that scores. That is Part 4, what the filter quietly breeds, and it is the uncomfortable half of this argument.
For now, take the practical half. If you build models, or agents on top of models, or products on top of agents, you are not downstream of this story. Every eval you write, every judge you configure, every metric you promote to a dashboard, every A/B test you ship is a selection pressure, and the system on the other side will adapt to it more faithfully than to anything you write in a spec. You have more design power than you think. It is just sitting in a different file than you thought.
You are not testing the species. You are writing it.
