AI Is Becoming Society's Operating System. Nobody Voted for That.

A few months ago I was helping a friend navigate a visa rejection. He’s a skilled engineer, been employed continuously for eight years, no criminal record, strong financials. He got rejected anyway — a terse form letter, no explanation, no appeal pathway. When we dug into it, the decision had been scored by an algorithmic system. Nobody we could talk to understood how it worked. Nobody with authority over the outcome could explain which data point had sunk him.

This is not an edge case anymore. It’s the texture of daily life.

Your mortgage gets scored. Your resume gets ranked before a human reads it. Your credit card limit gets adjusted by a model that flagged a spending pattern shift during your vacation. Your health insurance company’s algorithm flags your claim for review. In every one of these cases, there is an AI system functioning as an invisible gatekeeper between you and something you need, and the accountability structure around it is roughly analogous to a locked room.

Shannon Vallor — philosopher of technology at the Edinburgh Futures Institute and former AI ethicist at Google — published The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking in 2024. Her central argument is elegant and somewhat devastating. She uses the myth of Narcissus to describe our relationship with AI: we are captivated by a reflection that we mistake for something other than ourselves. AI systems are built from “oceans of our data,” she writes, and they are mathematically optimized to reproduce patterns in that data. Which means they don’t open new futures. They reproduce the past — including its errors, its biases, and what she calls its “failures of wisdom.”

Vallor’s book is primarily a philosophical intervention about self-understanding. She’s worried about what it does to us — cognitively and morally — to outsource judgment and creativity to systems that only know how to recombine what already existed. That is a genuinely important argument and I think she’s right about it.

But I want to extend the analysis in a direction she gestures toward but doesn’t fully develop: the infrastructure problem.

From Tool to OS

There’s a meaningful distinction between a technology that’s a tool and a technology that’s an operating system. A tool does something when you pick it up. An OS structures what’s possible — it decides which programs can run, what resources they can access, how they communicate with each other. You interact with tools. You live inside an OS.

AI is completing a transition from the former to the latter. This isn’t hyperbole. Consider what’s happened in the last two or three years. Foundation models are now the computational substrate beneath everything from customer service to legal research to hiring to content moderation to medical imaging analysis. The companies that train and serve those models — currently a very short list — are in a position analogous to what Microsoft occupied in the 1990s, except the surface area is not just software but a significant portion of high-stakes human decision-making.

Only a handful of states and companies have the compute and talent to train frontier models. The cloud infrastructure needed to serve them at scale is controlled by an even shorter list of providers. The result is something that researchers have started calling a “critical infrastructure” problem: AI is becoming so embedded in essential services that participation in modern society increasingly means interacting with AI systems, almost none of which are publicly accountable in the way that, say, a court is, or even a utility.

Vallor’s mirror metaphor is useful here in a way she may not have fully intended. A mirror shows you what’s already there. So does algorithmic infrastructure that’s optimized on historical data: it systematically reproduces whoever already had access, whatever patterns already correlated with creditworthiness or hireability or visa approval. In a world where access to capital, employment, education, and mobility are already unequally distributed, that optimization pressure doesn’t just preserve inequality — it hardens it. It gives inequality the appearance of objective, technical inevitability.

The person who gets denied a loan isn’t told “we think people from your background are bad credit risks.” They’re told the model scored them below the threshold. The subjectivity is laundered through math.

The Access Layer

Here’s what I think about a lot, coming from where I come from. I grew up in Kerala. I’ve lived and worked in Bangalore, Dubai, Hanoi. I’ve watched what happens when a new technology wave crests — who gets to ride it and who gets restructured by it.

The internet was supposed to be the great equalizer. In some ways it was. In other ways, the primary economic benefit was captured by a few cities, a few companies, and a few demographics with the right combination of connectivity, capital, and credentials to build for it or work within it.

AI is following the same structural logic, but faster and with higher stakes. The productivity gains are real. The compression of what previously required expensive expertise is real — I’ve seen it firsthand in what I’m building at Zero. A student in rural Kerala with a good phone and internet connection can now access explanations and feedback that previously required a private tutor or a well-resourced school. That matters enormously.

But the same technology that democratizes access to knowledge is simultaneously being deployed to sort people — and the sorting criteria are opaque, the appeals process is nonexistent, and the concentration of control over those systems is extreme. The “country of geniuses in a datacenter” that Amodei describes in a different context applies here too: a small number of systems are making vast numbers of decisions about vast numbers of people, with essentially no oversight that scales.

This is what Vallor means when she warns about AI reproducing “unsustainable patterns of the past.” Not just culturally. Structurally. The patterns of who has access and who doesn’t — baked into decades of historical data — get treated as natural features of the world rather than contingent outcomes of prior decisions. The mirror doesn’t know the difference.

What an OS-Level Problem Requires

If AI is operating-system-level infrastructure, then tool-level governance is not adequate. Asking individual companies to self-regulate is like asking individual apps to manage kernel-level security. The layer doesn’t match the problem.

This is not an argument against AI. I build with AI. I think the legitimate uses vastly outnumber the harms. But I do think it requires a different conversation than the one most of the industry is having.

The question of who controls the infrastructure that sorts and gates access to opportunity in a society is a political question, not just a technical one. It’s the same question we asked about electricity, telecommunications, and the internet. In each of those cases, society eventually decided that pure market dynamics didn’t produce outcomes we were willing to accept — not because markets are bad but because some infrastructure is too foundational to leave entirely in private hands without accountability structures.

We’re nowhere near that conversation with AI. We’re still mostly at the level of “guidelines” and “principles” — voluntary documents that cost nothing to sign and less to ignore.

Vallor ends The AI Mirror with an argument for what she calls “reclaiming our humanity” — being intentional about which judgments we let AI make versus the ones we insist on making ourselves, with the full weight of human moral responsibility. That’s right as far as it goes, and it’s good advice for individuals and organizations.

But I’d add a layer. At the infrastructure level, the question isn’t just what do I let AI decide. It’s what does we let AI decide, and who counts as “we,” and who has standing to challenge a system when it’s wrong. Those are governance questions. Design questions. Political questions.

They’re also urgent ones. The infrastructure is being built right now. The patterns being embedded in it right now are the ones that will shape access and opportunity for the next two decades. The decisions being made in the next few years about accountability, transparency, and public oversight of AI decision-making are not preliminary or provisional. They’re foundational.

I don’t have a clean policy prescription here. I’m a builder, not a regulator. But I’ve spent enough time building systems to know that architecture decisions made in year one are almost impossible to undo in year five. The time to design for accountability is before the infrastructure is deeply embedded — not after.

We’re in year one. Maybe year two, if we’re lucky.

---