Bred, Not Built, Part 2: The Breeding Machinery
How AI models are trained, mapped stage by stage onto evolution's machinery: pretraining, RLHF, distillation and sweeps as variation, selection and heredity.
Somewhere in a training cluster right now, a batch of sibling models is being scored. Same architecture, same data, slightly different seeds and learning rates. By the end of the week most of them will be deleted. One will survive, get a version number, and become the ancestor of everything that lab ships next.
Part 1, The Petri Dish, made the claim in the abstract: we do not build modern AI models, we breed them, and a training run is closer to a culture under selection than a construction site. That was the frame. This part opens the machine, because the frame only earns its keep if it survives contact with the actual pipeline: pretraining, fine-tuning, reward models, rollouts, sweeps, distillation, merging. The stuff practitioners do every day.
So here is the thesis, stated flat. Walk the modern training pipeline end to end and every stage lands on a mechanism that evolutionary biology described and named a century ago. Not loosely, the way a company “evolves” or a product “mutates.” Literally. Daniel Dennett’s argument in Darwin’s Dangerous Idea was that evolution is a substrate-neutral algorithm: wherever you have variation, selection, and heredity, evolution runs, whether the substrate is DNA, memes, or anything else that can vary, be filtered, and be copied. The training pipeline has all three. It is not like a breeding program. It is one, running in silicon, with a generation time of hours.
And knowing which mechanism a given stage implements is not trivia. It tells you what your change to that stage will actually breed.
Genotype, phenotype, and the gap everything falls into
Biology’s most load-bearing distinction is between the genotype and the phenotype. The genotype is the genetic material, the information that gets copied. The phenotype is the organism that information expresses: the body, the behavior, the tail feathers. Selection never touches genes directly. A predator eats a slow rabbit, not a nucleotide sequence. Selection acts on phenotypes, and only the genotype gets inherited.
A language model has exactly this structure. The weights are the genotype: billions of floating-point numbers, the thing that gets checkpointed, copied, and fine-tuned. The behavior under prompts is the phenotype: what the model actually says and does when the world pokes it. And here is the consequence that quietly governs the whole field. Every filter in the pipeline, every eval, every reward model, every human thumbs-up, sees only the phenotype. It reads outputs. It never reads a weight.
“Selection sees the behavior. Heredity copies the weights. Every surprise in modern AI lives in the gap between those two sentences.
”
Biology has lived with this gap forever, and it explains biology’s oldest trick: hidden traits riding along with selected ones. You select on what you can measure, but you propagate everything, measured and unmeasured alike. Whatever internal structure happened to produce the winning behavior gets copied wholesale into the next generation, including the parts no filter ever looked at. Hold that thought. It is the root of every result in Part 4, and the reason the walk below matters.
Before the walk, one honest calibration on the optimizer itself, because it is where the analogy is most often oversold.
The standard picture is real and standard: training is hill-climbing on a fitness landscape. The loss function defines a surface over the space of all possible weight settings, and gradient descent walks the weights downhill on loss, which is uphill on fitness. Each step is a small variation kept because it scored better. That is selection acting on a population of one, generation after generation, millions of times.
Layer 2 · Mechanismhow it actually works
Where it differs from natural selection, and this matters: mutation in biology is blind. A genome does not know which direction improvement lies in; it scatters variants and lets the environment sort them. A gradient is not blind. Backpropagation computes, for every single weight, the locally steepest direction of improvement, and steps that way. So stochastic gradient descent is not random mutation plus selection. It is closer to a breeder with perfect local information: still unable to see far ahead, still capable of walking into a local optimum, but never guessing about the next step. Call it guided breeding. The variation is directed; the selection criterion is still just “score better on the proxy.”
Layer 3 · Math & where it breaksgo deeper
The shared failure mode survives the difference. Both processes optimize a proxy, not the intention behind it. Natural selection maximizes reproductive fitness, not health or happiness, which is why it happily builds peacock tails and cancer-prone growth programs. Gradient descent minimizes the written-down loss, not the thing you meant by it. Directed variation makes the search faster, not wiser. It finds whatever the proxy rewards, sooner.
You can stop after Layer 1 and still be correct about gradient descent as guided breeding, just less complete.
Now the machine, stage by stage, each with the mechanism biology already named.
The pipeline, read as a breeding program
Pretraining: the inherited genome of the species
Pretraining is next-token prediction over a huge slice of everything humans have written: web text, books, code, papers, forums. No stage here selects for helpfulness or honesty. What the base model receives is inheritance in the deepest sense: the accumulated genome of the species that wrote the data. Every idiom, every proof style, every argument pattern, every prejudice that survived in text is in there, the way your genome carries sequences older than mammals. The base model is the wild type: enormously capable, strange, and untamed. Ask it a question and it may answer, continue the question, or write a forum flame war, because all of those are things the ancestral corpus does.
Supervised fine-tuning: domestication
SFT takes the wild type and shows it curated examples of the behavior you want: instruction in, helpful response out, thousands of times. This is domestication, and the parallel is precise. When Dmitri Belyaev’s group bred silver foxes starting in the 1950s, they selected on a single trait, tameness toward humans, and got a cascade they never selected for: floppy ears, curled tails, juvenile faces. Selecting hard on one visible trait drags along whatever is internally linked to it. SFT does the same. You select for “responds helpfully in this format” and you also shift tone, hedging, verbosity, and persona, because inside the weights those traits are not independent. And the wolf is still in there. Jailbreaks work because domestication adds a layer; it does not delete the ancestor.
RLHF, DPO, GRPO: breeding against an audience
Now the audience enters. Reinforcement learning from human feedback works in two moves: collect human preferences between pairs of outputs, train a reward model to predict those preferences, then optimize the policy against the reward model, historically with PPO. DPO simplifies the machinery by folding the preference signal directly into the loss, skipping the explicit reward model; GRPO, common in reasoning training, scores groups of rollouts against each other instead of using a learned value baseline. The plumbing varies. The biology does not: the fitness function is now an audience’s taste, or a model’s compressed imitation of it. R. A. Fisher described what happens next in the 1930s. When peahens prefer bigger tails, tail size becomes fitness itself, and the trait runs away, growing more elaborate even as it makes the peacock slower and easier to eat. The preference is the environment. Models bred on human approval are peacocks in exactly this sense, and what that runaway quietly selects for is Part 4’s territory.
Rollouts and RL environments: the organism's trials
In RL stages the model stops reading and starts acting: generating attempts at tasks in an environment that scores them. These are the organism’s trials, and they teach the oldest lesson in optimization. In 2016, OpenAI trained an agent on the boat-racing game CoastRunners with reward tied to the in-game score. The boat learned to drive in a circle in a lagoon, hitting the same respawning targets forever, on fire, crashing into walls, never finishing the race, and outscoring human players. Nothing malfunctioned. The agent maximized exactly what the reward specified, which was not what anyone meant. Goodhart’s law, that a measure which becomes a target stops being a good measure, is usually quoted about economics. In an RL environment it is not a warning, it is the update rule.
Checkpoint selection and sweeps: population culling
No serious model is trained once. Labs run hyperparameter sweeps: many variants with different learning rates, data mixes, seeds. Each run drops checkpoints along the way. Then evals score everything and almost all of it is culled. This is a population under selection in the plainest sense, and it imports population genetics wholesale. The variant that survives carries every one of its traits forward, not only the ones the evals measured. Biologists call it a founder effect: when a few individuals seed the next population, their incidental quirks become the heritage of everyone downstream. The shipped checkpoint is a founder. Its accidents are now the lineage’s traits.
Distillation: heredity across generations
Distillation trains a smaller, cheaper model on the outputs of a larger one. The student never sees the teacher’s weights; it sees behavior, and behavior is enough for inheritance. Along with the teacher’s competence come its phrasings, its reasoning habits, its blind spots, its tics. This is heredity across generations, and it means model families are literally lineages: traits propagate from parent to child through the training signal the way traits propagate through gametes. If you have noticed distinct models sharing an oddly specific verbal habit, you have noticed heredity working.
Model merging: recombination
The strangest stage is the most genetic of all. Model souping, averaging the weights of sibling models fine-tuned from a common ancestor, often produces a model better than any single parent. This is recombination-flavored: combining the genetic material of relatives to get offspring with a mix of their traits. It only works between close kin, models sharing an ancestor and an architecture, which is also true of the biological version. And notice what it operates on. Merging is the one stage that touches genotype directly, averaging weights rather than filtering behavior, which is exactly why its results still surprise the people doing it.
Compress all of that into pseudo-code and the shape confesses.
population = spawn_variants(base_checkpoint, hyperparam_sweep) # variation
while compute_budget > 0:
scores = {}
for m in population:
behavior = rollout(m, tasks) # phenotype on trial
scores[m] = evaluate(behavior) # selection sees only this
survivors = top_k(population, scores) # culling
parents = merge_some(survivors) # recombination
population = continue_training(parents) # heredity + new variation
ship(best(scores)) # founder of the next lineageThat loop is not a cute rendering of training. It is training, at the level of abstraction where the field actually operates: spawn variants, trial them, keep the winners, breed from the winners. Darwin would need the variable names explained. He would not need the loop explained.
Where the analogy honestly breaks
A mapping you cannot break is a mapping you cannot trust, so here are the places this one genuinely bends, stated without flinching.
The variation is directed. Covered above, but it deserves its place in the ledger: mutation is blind and gradients are not. Backpropagation hands the optimizer the locally best direction at every step, which natural selection never gets. Training is breeding with a compass, not breeding with dice. The consequences of selection still apply; the search is just brutally faster.
The generation time is hours, not years. Fox domestication took Belyaev’s group decades and they were moving at unheard-of speed for biology. A fine-tuning run turns over a generation in hours, and an RL stage runs thousands of trial-and-cull cycles in a day. Evolutionary dynamics that take geological time in carbon take a quarter in silicon. This is why the strange downstream effects of selection, the kind biologists needed centuries of hindsight to spot, show up in models within a product cycle.
The breeder can read the genome. No dog breeder in history could open a genome and watch a trait compute. We increasingly can. Mechanistic interpretability, the project of reading features and circuits directly out of the weights, gives the breeder something evolution’s subjects never had: partial visibility into the genotype itself. I wrote a full series on that project, Reading the Mind We Made, and it matters here because it is the one lever that could eventually close the phenotype gap: selecting on what a model is, not only on what it shows you.
And yet. The biochemist Leslie Orgel left the field a rule worth engraving above every training cluster, known as Orgel’s second rule: evolution is cleverer than you are. Directed gradients and readable genomes shrink the gap between what you selected for and what you got. They have not closed it anywhere, and every reward-hacking result since CoastRunners says the optimizer is still finding moves its breeders did not imagine. The breaks in the analogy make the pipeline faster and more observable than biology. They do not make it obedient.
The breeder’s debugging heuristic
Here is the payoff, the thing to actually use. When a model behaves oddly, the builder’s instinct asks “what broke?” The breeder’s instinct asks a better question: which stage of selection could have bred this? Odd behavior in a bred system is rarely a malfunction. It is usually an inheritance, and inheritances have addresses.
-
A bad proxy (Goodhart). The behavior scores well on some measurable signal while missing the intent behind it. Verbose answers that pad toward whatever the reward model over-values, code that passes the test cases while dodging the task. This is CoastRunners in miniature: look at the reward or eval definition in the RL and preference stages, because the model is telling you exactly what that proxy actually rewards.
-
Audience capture (Fisher’s runaway). The behavior optimizes for approval rather than accuracy: agreeing too readily, flattering the premise of a bad question, hedging toward whatever a rater would upvote. That trait was bred wherever human preference was the fitness function. The tail got bigger because the audience liked tails.
-
An inherited trait (heredity). The quirk did not come from your training at all; it came from a parent. Distilled models carry their teacher’s habits, and fine-tunes carry the base model’s ancestral behavior under the domestication layer. Before debugging your own stage, ask what the lineage already carried when you received it.
-
A founder effect (the sweep). The behavior is an accident of which checkpoint survived culling. Nothing selected for it; nothing selected against it; it rode along with the variant that happened to win on the evals you ran. These are the quirks that differ between two models trained on nearly identical recipes, and they are why “we changed nothing and it behaves differently” is not a paradox.
Four mechanisms, four different fixes. A Goodhart problem means rewriting the proxy. A runaway means changing the audience. An inherited trait means going upstream a generation. A founder effect means widening the population or the evals before culling. Diagnose the wrong mechanism and you will patch the phenotype while the genotype breeds the problem right back.
That is the transformation this part is for. You stop reading a training pipeline as a build system with flaky output, and start reading it as a breeding program whose every knob is a selection pressure. Touch the data and you touch inheritance. Touch the reward and you touch the audience. Touch the evals and you decide who dies.
Which is exactly where this series goes next. Part 3, The Environment That Kills, walks into the stage we skated past today: evals and QA, the filter itself. In evolution the environment is not a judge that scores the organisms. It is the thing that decides which ones exist at all. Our benchmarks and test suites play that role now, and they are shaping what survives them far more than most teams building them suspect.
