Graphs and State Machines for Agents, Part 5: Running the Fleet on Frontier and Open-Weight Models

You have the machine now. Four parts of work went into it. The job has a shape and the shape is a graph. The graph is a finite state machine, feature for feature. The transition function is data you can print and check. And an agent that plans, calls tools, retries, and asks a human is that machine whether you draw it or not, so the safety-critical routes live in a table instead of a sentence. It is reliable. One of it.

Now the bill arrives. A reliable agent that classifies one ticket is a nice demo. The real job is ten thousand tickets before lunch, or a security sweep across three hundred files, or a research pass over four hundred sources. Run your careful single machine against that and it does the one thing Part 1 spent its whole length warning you about. It queues. It stands in its own line, a machine at a time, and charges you for every wait.

This is where the series pays off, and it is the part nobody plans for on purpose.

This is Part 5 of five. Part 1, “From Lines to Graphs,” drew the shape. Part 2, “Every Graph Is a State Machine,” named it. Part 3, “Building the Machine,” made it runnable. Part 4, “Agents as State Machines,” made it reliable by construction. Every one of those was setup for a single move: taking that reliable machine and running a thousand of them at once, cheaply, on a model mix you control node by node. Here is how.

Depth was never the lever. Breadth was.

There are two directions you can grow an agent, and only one of them scales.

You can grow it deeper. Add another step to the line. Another tool call, another “and then,” another careful instruction in the system prompt. This is the prompter’s move, and it has a ceiling that arrives fast, because a line pays the sum of its steps and dies if one of them stalls. Ten steps deep is ten sequential waits stacked on top of each other.

You can grow it wider. Add another node to a layer. Run the independent work at the same time instead of one after another. This is the architect’s move, and it scales in a direction the line cannot, because a layer pays the cost of its slowest node, not the sum of all of them. Ten nodes wide is one wait, paid once.

The whole discipline of running a fleet is this: find the work that has no real dependency between its parts, and run all of it at once. Fan-out is the primitive. It takes N independent nodes and fires them in parallel, and it is the one line of code that buys you the most in agent engineering.

// Fan out N independent nodes at once. A node that throws becomes null
// instead of sinking the whole batch, then the holes get filtered out.
async function fanOut(items, node) {
  const results = await Promise.all(
    items.map(item => node(item).catch(() => null))
  );
  return results.filter(r => r !== null);
}

That .catch(() => null) is not defensive decoration. It is the difference between a fleet and a chain. In a chain, one failure stops everything downstream. In a fleet, one failure is a hole you route around: eight good branches return while the ninth quietly drops out. Isolation by construction, one clause of code.

Fan-out has a partner you reach for far less often. Fan-in is a barrier: a point where every parallel branch converges and one node finally sees the whole set together. You need it only when a stage genuinely requires everything at once. Deduping across the full union of results needs it. Ranking the whole set needs it. Deciding to early-exit because everything came back empty needs it. Most intermediate stages do not, and reaching for a barrier by reflex is how you rebuild the queue you just escaped, because a barrier makes everyone wait on the slowest branch.

Put a fan-out and a fan-in back to back and you get the shape worth burning into muscle memory: the diamond. Split to gather breadth, work in parallel, merge to one answer. Its canonical form has three beats, and the middle one is where the money hides.

// The diamond: fan out (breadth), reduce (plain code, zero tokens),
// synthesize (one agent, once).
async function diamond(sources, query) {
  const raw     = await fanOut(sources, s => searchNode(s, query)); // split
  const merged  = dedupe(raw.flat()).filter(r => r.score > 0.4);    // reduce
  if (merged.length === 0) return "No support found.";              // early exit
  return synthesizeNode(query, merged);                             // synthesize
}

Look at the reduce step. It flattens, dedupes, filters, and it spends zero tokens doing it, because it is plain code. You do not pay a model to concatenate two lists. A startling share of real agent work is exactly this diamond: research, review, retrieval, triage. Fan out cheap, compress in code, spend one expensive call on the answer. Once you see it you cannot unsee it.

Let the model judge the node. Let code pick the edge.

Before the fleet gets interesting, one piece of wiring keeps it honest. Real work branches, and you have to decide who does the branching. The answer is not the model.

A router is a node that classifies, followed by ordinary control flow that chooses the edge. The model produces a validated field. Code reads that field and picks which downstream node fires.

async function route(ticket) {
  const { severity } = await classify(ticket); // judgment lives at the node
  if (severity === "high")   return pageOnCall(ticket); // code picks the edge
  if (severity === "medium") return fileBug(ticket);
  return autoReply(ticket);
}

The split matters more than it looks. The model’s judgment sits exactly where you want judgment, at the node, filling in a value under ambiguity. The control flow sits exactly where you want reliability, at the edge, in an if you can read, test, and version. There is no emergent surprise where the agent decided to skip the audit step, because the agent never decides the topology. The graph does. This is the Part 4 lesson applied to a fleet: keep the dangerous route in the table, not in the prompt.

Model tiering is where frontier meets open-weight

Here is the section the whole series was walking toward. Once the work is a graph of bounded nodes, the model is no longer one decision you make for the whole system. It is a decision you make per node, and that changes what a fleet costs.

Not every node needs the same brain. A node that extracts a field or classifies a ticket is bounded and repetitive. A node that synthesizes the final report or adjudicates a contested finding is judgment under real ambiguity. Running both on your most expensive model is hiring a principal engineer to alphabetize files. So you tier: cheap model on the bounded high-volume nodes, expensive model on the few judgment nodes. And the seam between those tiers is exactly where the frontier-versus-open-weight decision lives, node by node instead of all-or-nothing.

Frontier models (Claude, GPT, Gemini, reached over an API) give you the strongest per-node judgment, and one capability that is easy to miss. With Claude Code’s dynamic workflows, a frontier model can write the orchestration script itself. It emits a plain JavaScript coordinator that spawns a fleet of subagents, where the coordination between them is code rather than a conversation. That distinction is the cost story: a coordinator written as code runs the fan-out, the reduce, and the routing for zero model tokens. The frontier model is spent on the nodes that need judgment, not on narrating the plumbing between them.

// Illustrative Claude Code dynamic-workflow shape. A frontier model writes
// this coordinator; the coordination itself is plain code, so it costs no
// model tokens. Only the nodes that need judgment call a model.
async function reviewPR(files) {
  const findings = await fanOut(files, spawnReviewer); // subagents, in parallel
  const merged   = dedupe(findings.flat());            // plain code, zero tokens
  if (merged.length === 0) return "No issues found.";
  return spawnSynthesizer(merged);                     // one judgment call
}

Open-weight models (Llama, Qwen, DeepSeek, Mistral, gpt-oss, GLM), self-hosted through vLLM or Ollama, let you own the fleet outright. Run the high-volume bounded nodes at near-zero marginal cost, keep the data in-house, and stop paying a per-token API bill on precisely the nodes that fire the most. The graph does not care which company’s weights sit behind an endpoint. LangGraph and Jerry Liu’s LlamaIndex Workflows express the same node-and-edge graph over these backends, so the model becomes a config value on a node, not an architecture decision for the system.

The move that beats either purebred approach is the hybrid graph. Open-weight models on the wide fan-out of bounded extraction and classification. A frontier model reserved for the narrow synthesis and judge nodes at the merge. A diamond where the split is cheap and local and the point of the merge is expensive and smart.

The economics are the whole reason to bother. Ten thousand extraction calls a day on a self-hosted open-weight model cost you the electricity and a GPU you already rent. The same ten thousand on a frontier API is a real invoice that grows every day the product does. Reserve the frontier tokens for the handful of merges where judgment actually decides the outcome, and the same graph gets cheaper by a large multiple without getting dumber where it counts.

# Illustrative. vLLM serves an OpenAI-compatible API on localhost.
from openai import OpenAI
local = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

def extract(chunk: str) -> str:            # bounded, high volume -> open weight
    r = local.chat.completions.create(
        model="qwen2.5-7b-instruct",
        messages=[{"role": "user", "content": f"Extract fields as JSON:\n{chunk}"}],
    )
    return r.choices[0].message.content

import anthropic                            # judgment, once -> frontier
frontier = anthropic.Anthropic()

def synthesize(findings: list) -> str:
    r = frontier.messages.create(
        model="claude-opus-latest",         # use the current frontier model id
        max_tokens=2000,
        messages=[{"role": "user", "content": build_prompt(findings)}],
    )
    return r.content[0].text

# Illustrative LangGraph shape.
from langgraph.graph import StateGraph, END

g = StateGraph(dict)
g.add_node("extract", extract_fanned)       # open-weight, mapped per chunk
g.add_node("synthesize", synth_node)        # frontier, once at the merge
g.set_entry_point("extract")
g.add_edge("extract", "synthesize")
g.add_edge("synthesize", END)
app = g.compile()

Topology is cost, and pipeline beats barrier by default

There is a second lever hiding in the shape of time itself, and most teams never touch it because they only know one topology. A batch of items can move through your stages two very different ways.

A parallel graph puts a barrier between stages. Everything in stage one finishes before anything enters stage two. It is simple to reason about, and it wastes the slowest node’s shadow: the entire batch idles waiting on the one item that lagged.

A pipeline streams each item through all stages with no barrier. Item A can be in stage three while item B is still in stage one. The stages stay busy instead of standing idle behind a barrier, and throughput climbs. Default to the pipeline. Reach for a barrier only when a stage truly needs every prior result at once, which is ranking and final synthesis and not much else.

Isolation, so one bad node cannot poison the fleet

A fleet fails differently from a chain, and if you built the fan-out above you already have the cheap version of the fix. A node that throws resolves to null, you filter the nulls, and you design every fan-in to tolerate missing inputs. One malformed source cannot sink the batch. Eight good branches return; the ninth drops.

There is a heavier isolation you need for exactly one topology. When nodes write files in parallel, in place, they stomp on each other. That is when git-worktree isolation earns its keep: each node gets its own working tree, does its edits, and you merge the good ones afterward. Reach for it only when nodes actually mutate a shared workspace at the same time. It is the seatbelt for that one crash, not a default tax on every graph. Most nodes only read and return, and they need nothing heavier than the null-on-throw rule.

Self-routing: let the model draw the graph, then keep the good ones

The frontier and the graph meet at their most interesting point in dynamic workflows. Instead of hand-drawing the topology, you describe the objective and let the model write the graph for this run: which nodes, which fan-outs, which merges. Then you do the thing that turns a one-off into an asset. When a generated run works well, you save it as a version-controlled workflow. The model explores the space of graphs; you keep the ones that earned their place. Over time your repo becomes a library of shapes that solved real problems, which is a far better artifact than a folder of clever prompts. A prompt is a lottery ticket you buy again every run. A saved workflow is a machine you own.

When not to build a machine at all

The honest close to five parts of arguing for graphs and state machines is knowing when to reach for neither. Overbuilt structure is its own failure. Three cases genuinely do not want a machine.

• Genuinely sequential logic belongs in a plain function. If the work is A then B then C with no waiting, no events, and no failure branches, a graph adds ceremony and buys nothing. Write the three lines. The state machine earns its keep when there is a decision about what happens next, not when the next thing is simply the next line.
• Truly continuous state gains nothing from discrete states. A physics simulation, a numerical solver, an audio filter: their state is a vector of real numbers evolving every tick, not a handful of named modes. Forcing “idle, running, done” onto a differential equation is a costume, not a model.
• A two-state toggle needs a boolean, not a framework. If the entire state space is on or off, let open = false is the correct data structure. Wrapping it in a transition table is the kind of over-engineering that makes people distrust the whole idea.

So when have you actually grown a machine and should stop pretending a boolean is enough? There is a reliable trigger, and it is worth watching for. The moment a second boolean appears that interacts with the first, you have a state machine whether you name it or not, because two booleans can express four states and at least one of them is usually illegal. The other tell is textual: the moment a comment shows up in your code that reads “but only if we are currently …”, you are describing a transition that depends on the current state, which is the definition of the thing. That comment is a state machine asking to be written down.

Close the series

Go back to the line the whole series turned on: a prompter asks a question; an architect draws a graph. Everything in these five parts was one long argument that the second person wins, and wins in a way that compounds. The prompter improves an agent by adding steps to a line and hoping the model holds it together. The architect improves it by drawing the states, the events, and the transitions, then running that drawing wide across a fleet on whatever models the budget wants. One is negotiating with a nondeterministic component. The other made the structure carry the load, so the model is free to be exactly what it is good at: judgment at a node, never the wiring between them.

The graph is the thing you own. The models are just where you rent the thinking, and the mix is a dial, not a vow.

Something will arrive after graphs. It always does. There will be a new abstraction with a new name and a launch post that says the old way is dead. When it comes, you already have the only test that matters, and it is the same test Part 2 handed you. Ask it to name its states, its events, and its transitions. If it can, it is a machine you can reason about, run wide, and check. If it cannot, it is scattered control flow with better marketing, and you have spent five parts learning to tell the difference.