Prologue
This is the most difficult blog that I’ve ever undertaken to write, not conceptually of course, or technically although I’ve made little attempt to smooth the jargon. It’s difficult because I believe that there’s little short-term reward to saying that the emperor has no clothes. In the euphoric hype of the roaring 20’s, no-one was popular if they showed scepticism towards stocks. In the lead up to 2007, no-one was cool if they weren’t in housing and investing in Mortgage backed securities. Nonetheless, I believe my prediction isn’t something that I should keep to myself, if it’s in anyway beneficial towards others. The silence and carelessness which which corporations, especially the mid-market players, are attempting to replace their critical platforms and system of records with some Ai vibed solutions is concerning. Vibe coding, ai tools, are arguably the most powerful tools ever created. The disruption is undeniable, the impacts for both good and bad will be immense.
The Trusting Trust Attack
In August 1984, Ken Thompson stood up to accept the Turing Award, the closest thing computing has to a Nobel, and used the occasion to tell the assembled discipline that its foundations were rotten.
Thompson famous for co-creating Unix, arguably the most durable operating system ever conceived, was, in every meaningful sense, the highest authority in the world of software systems, and what he described has become known as the trusting trust attack.
In short, Thompson modified a compiler so that it would insert a backdoor into any login program it compiled. That alone would be detectable; a programmer reading the compiler’s source would find it. So Thompson went further. He taught the compiler to recognise when it was compiling itself, and to reinsert the entire backdoor into the new version. He could then delete the offending source code entirely. The compiler was clean. The source was clean. Every inspection would pass. And every program it built would be compromised, forever, with no evidence remaining anywhere in the visible system.
His conclusion was that you cannot trust code you did not create yourself, and that no amount of source-level scrutiny will protect you.
That was forty-two years ago. The industry applauded, published the lecture, made it required reading, and then proceeded to build an entire global economy on package registries and pipelines assembled out of code that essentially nobody has read. We did not disagree with Thompson. We simply found his advice inconvenient, difficult, filed it under ‘theoretical’, and got on with ‘ctrl-c’-ing & ‘ctrl-v’-ing our way to production over perfection.
And now, we’re putting this problem on steriods.
“Show me the incentive and I will show you the outcome.”
— Charlie Munger
The men who were paid to say yes
Past readers will know I have a particular hobby horse about the link between risk and responsibility, and what happens to a system when that link is severed. It is, I think, the single most useful lens for understanding why things that everybody could see coming nevertheless arrive.
Consider the credit ratings agencies in the run-up to 2008. Moody’s, Standard & Poor’s and Fitch stamped AAA, the same grade carried by sovereign debt, on tranches of securities built out of mortgages written to people with no income and no prospect of repayment. The subsequent inquiries produced a great deal of language about incompetence and about greed… like that was different to any other day on Wall Street. But the mechanism was simpler and considerably more depressing than either.
The agencies were paid by the issuers. Not by the investors relying on the rating. And because ratings were sold in a competitive market, any agency that applied genuine rigour would find the issuer walking down the street to a competitor who would not. The rational strategy for each individual agency, given the strategies available to the others, was to be generous. This is a textbook, worst case, Nash equilibrium, and it is a spectacularly bad one. Nobody had to be corrupt. Everybody had to be rational and self-interested, and rationality combined with self-interest with a broken risk and responsibility link led to global financial armageddon.
Now consider what Australia discovered about its own buildings between 2014 and 2019. A fire ran up the outside of the Lacrosse tower in Docklands. Opal Tower was evacuated on Christmas Eve. Mascot Towers followed six months later. The cladding audits that followed found combustible material on buildings across every capital city, and behind almost all of it sat a private certifier who had signed the compliance paperwork.
Those certifiers were, overwhelmingly, competent and credentialled people. They were also engaged by the developer, paid by the developer, and dependent on the developer for their next engagement, while being asked to form an independent judgement about the developer’s product. The Shergold–Weir review handed to the Building Ministers’ Forum in 2018 said as much in the politest possible terms. New South Wales eventually appointed a Building Commissioner with actual powers, roughly a decade after the powers would have been useful.
Two industries, two decades apart, in entirely different fields, producing an identical failure. In both cases the certification was technically performed. Reports were written. Boxes were ticked. Fees were paid. And in both cases the assurance was worth precisely nothing, because the party providing it had no exposure whatsoever to being wrong.
What we actually buy when we buy software
I have argued before that the great analytical error of our age is confusing the nominal with the real — mistaking the figure on the page for the productive fact underneath it. Nowhere is this error more expensive, or more thoroughly institutionalised, than in enterprise technology procurement.
The conventional wisdom about agentic AI runs roughly as follows. For twenty years, software-as-a-service sold you an interface, a data model and a set of integrations, billed monthly per seat. Fast-forward to August 2026, and agents now speak to APIs directly, so the interface ceases to be a moat (or a liability in the case of Salesforce) and becomes an overhead. What survives is the system of record. The data gravity, the retention obligations, the audit trail. Everything above that layer gets generated on demand, in-house, by a machine, in an afternoon, usually by a vibe coder with next to zero understanding of coding let alone the code that his Claude agent is generating.
But stop here for a second. While I’d bet that most CTOs and IT professionals would generally agree with the above analysis as broadly correct, it’s missing something critical, because it treats an enterprise software contract as a technical solution, an interface, a database structure. It is, but it’s also much more.
What a corporation actually purchases in a SaaS agreement is a very large quantity of transferred risk. You are renting the vendor’s security team. Their patching cadence. Their penetration testing cycle. Their SOC 2 report, their certification, their breach notification obligations and their professional indemnity insurance.
And here is the part that nobody sees: the cost of producing that assurance is amortised across the vendor’s entire customer base. One control environment. One audit. One insurer. Ten thousand customers. It is an enormous economy of scale, and it is invisible precisely because it arrives on the invoice as a single line beside the seat count. No CFO has ever seen it itemised. Which is exactly why no CFO will notice when it goes.
So when an internal generates a working micro-application over a long lunch, and the company congratulates everyone on the licence fee they didn’t pay, understand what has actually taken place. The licence has not been saved. The liability has been repatriated, stripped of its amortisation, and booked nowhere. The saving is immediate, legible, and celebrated in the EBIDTA improvement. The cost arrives eighteen months later, on a different line, wearing a different name.
The nominal saving is enormous. The real position has deteriorated. And as ever, the fundamentals are patient.
What is trust, actually?
Let me make a philosophical detour here, because I think the technical discussion of AI safety has been conducted almost entirely in the wrong vocabulary, and the wrong vocabulary is why serious people keep proposing solutions that are divorced from reality.
We talk about trusting software as though trust were a property of the thing trusted. It is not. Trust is a relationship, and the load-bearing component of that relationship has never been based on the competence of the trusted party. It has always been their exposure.
Consider what you actually mean when you say you trust a surgeon. You do not mean that you have inspected her technique. You mean she holds a licence that can be revoked, carries insurance that can be claimed against, belongs to a college that can strike her off, and has a career that ends if she is careless. Her competence matters enormously — but her competence is not what you are relying on, because you have no capacity whatsoever to assess it. What you are relying on is that being wrong would cost her something, or worse some heuristic based on reported past performance.
The same is true of the auditor who signs the accounts, the engineer who stamps the drawings, the ratings analyst who grades the paper and the certifier who approves the building. In every case, the value of the signature is not the diligence behind it. It is the fact that there is a named party who can be pursued, and who knows it. Diligence is what exposure produces. It is downstream, not upstream.
Which is why the failures I described above are all the same failure. The ratings agencies were competent. The certifiers were competent. What they lacked was skin in the game, and the moment that was gone the diligence evaporated, not through wickedness but through the ordinary operation of incentive on rational actors over time.
And this is where the issue of the the myopic hedonist appears, because the reason we tolerate hollow assurance is the same reason we tolerate everything else that is cheap now and ruinous later. Why do we accept a tick-box audit? Because a real one is slow and expensive and this quarter has targets. Why do we sign off the vendor questionnaire we did not read? Because reading it delays the launch. Why do we buy the certificate rather than the security? Because the certificate is available today and the breach is hypothetical and, with luck, somebody else’s problem by the time it lands. The mob is not irrational. The mob has simply never been able to see past the end of the financial year, or worse, the next earning report or worse the next bonus.
We are not, as a species, particularly good at paying now for a disaster that may not arrive. We are, however, superb at buying a piece of paper that says the disaster has been handled.
The machine that cannot be sued
So: to the actual question, which is whether artificial intelligence can certify artificial intelligence.
The proposal is seductive and I hear it constantly from people who ought to know better. If agents generate the code, have another agent review it. The review scales at the same rate as the generation, the marginal cost rounds to zero, and it has the pleasing shape of a problem that AI ought to be good at. Every vendor in the market is building this. Much of it is genuinely useful.
It cannot be the last line, and it fails for three reasons which are worth separating, because they are not the same reason and only one of them is technical.
First, false independent review. Two models trained on overlapping corpora, with broadly similar post-training regimes, share broadly similar blind spots. The same weights. The same plagiarised stack overflow training data. Pointing one at the other is not a second opinion. It is the same auditor, asked twice, in a marginally different accent and a different harness. There is real diversity value in the exercise and it should be harvested, model review catches genuine defects and belongs in any serious pipeline. Firms that are systematising this process are doign well, and they should be. But it is a layer, not an endpoint.
Second, instruction sanitation. In a language model, instructions and data travel down the same pipe. There is no architectural boundary (and I might be wrong on this point as it’s pushing the limits of my knowledge in how models work) between content I am reading and instructions I am obeying; this is not an implementation defect to be patched but a consequence of how the things work. A reviewing agent does not merely read the diff. It reads the comments, the commit message, the dependency README, the linked issue, the build log. Every one of those is untrusted text arriving through the same aperture as its orders parsed through the same blackbox.
Simon Willison has the cleanest statement of the exposure (Full article here and it’s worth a read), which he calls the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externallly. Any two are survivable. All three together is the problem. A code review agent with repository access has all three by construction, and it has them because that is precisely what makes it useful .
And stacking a supervising agent above it does not help. It adds a second instance of the identical vulnerability. You may build that tower as high as you like and you will not reach the ground. Thompson’s problem is back, wearing a very expensive new coat.
Third, corporations can’t sue the machine. Return to what trust actually is. An agent cannot hold a licence that can be revoked. It cannot carry indemnity insurance. It cannot be struck off, sued, fined, jailed, or made to appear before a Senate committee looking uncomfortable in a suit. It has no capital at risk, no reputation that survives the session, and no future it is trying to protect. It is, in the most literal and precise sense available, judgement-proof. The main problem that Tesla will face with their driverless cab, won’t be technical capability but rather legal culpability.
We spent 2008 learning what happens when the party providing assurance has no stake in being right. We are now proposing to hand assurance to a party that is structurally incapable of having a stake in anything at all. The ratings agencies at least had a brand to lose despite the fact that some would argue their reputation is closer to teflon than a pinboard.
The issue of speed
And the reason why this guaranteed to become a global issue, is the same reason why Ai safety departments are being sidelined and underfunded as quickly as Nvidia’s stock is going ballistic, it’s the argument taht if we don’t do it first, then our competitors will beat us to the punch. Your competitor ships in a fortnight because their agents write, review and merge without a human in the loop. Your product managers read about it. And the internal argument shifts from is this wise to why are we slower than them, which is an argument the security function loses every single time, because the cost of the control is visible this quarter and the cost of the failure is not.
What the fundamental’s revenge
The money, eventually… maybe… does not go where most of the venture capital is currently pointed. It does not go into a shiny new category of AI-powered security software, because that category is attempting to solve a problem of exposure with an application of technology. It goes to two considerably less glamorous places.
The first is deterministic tooling. Hermetic builds, signed artefacts, build environments with no network egress. All of it old, all of it unfashionable, and all of it valuable for exactly the reason it is unfashionable: it fails differently. A type error remains a type error no matter how persuasive the commit message is or how sophisticated the prompt-injection is. This is not a prediction that raises a Series B, but the greybeards of program analysis might not be on the bench forever. I’m not making investment advice, and the young blood ai security firms are doing well in the short run and they should given the immense progress they’re making. But my long-term bet, is to watch out for the security firms that take a blended approach.
The second is attestation rather than detection, and this is where the real industry lives. Trying to detect prompt injection is adversarial classification against an opponent that adapts faster than you can re-scan, with no stable ground truth to classify against. I don’t think companies will win this game. Constraining what a compromised agent is permitted to do is winnable: narrow credentials, no egress during build, mandatory human approval on anything irreversible, blast radius bounded by design rather than by hope.
The other reading
In saying all this, I have to admit that the counter-case is also strong.
The same conditions could just as easily drive consolidation rather than fragmentation. As machine-assisted attacks scale and the threat environment deteriorates, a risk-averse board has every reason to retreat into fewer, larger, heavily certified vendors, precisely because those vendors can produce a documented control environment and will accept contractual liability for it. Under that reading, agentic AI entrenches the incumbents and the security spend flows to the platforms rather than away from them.
My honest expectation is that both occur simultaneously. The systems of record consolidate and become more valuable, because liability absorption was what they were genuinely selling all along and everyone is about to find that out. Everything above that layer fragments into generated software that no vendor supports, no register lists and no one owns.
SMEs, especially small business will likely make the error of abandoning a tried and tested System of record entirely in the name of saving a few bucks and instead opt for a vibe coded alternative…
The ledger
- Enterprise software was never priced as an interface. It was priced as a risk transfer with the assurance costs amortised across thousands of customers and itemised on nobody’s invoice.
- Generating your own applications does not save the licence fee. It repatriates the liability, strips out the amortisation, and books the saving and the cost in different years.
- Trust has never been a property of the trusted thing. It is a relationship whose load-bearing element is exposure to loss. Competence is what exposure produces, not what it replaces.
- An agent therefore cannot certify an agent – not primarily because of correlated blind spots or because instructions and data share a channel, though both are true, but because a model is judgement-proof. It has nothing to lose and can therefore guarantee nothing.
- Competitive pressure will push firms into agent self-certification anyway, by exactly the mechanism that pushed the ratings agencies into AAA. Nobody needs to be corrupt. Everybody needs only to be rational.
- The equilibrium breaks from outside, and outside means the insurers.
Ken Thompson told us in 1984 that we could not trust code we had not written ourselves. We nodded, and then spent four decades building a civilisation on exactly that trust, because the alternative was slow and the risk was somebody else’s and the fundamentals, as always, seemed a very long way off.
We are now proposing to close the loop entirely: machines writing the code, machines reviewing the code, machines certifying the review, and a stack of paper at the end of it attesting that everything is fine. It will be cheaper than the alternative. It will be very much faster. Every incentive in the market points directly at it, and the firms that hesitate will watch the firms that don’t take their customers.
And it will hold, as these things always hold, right up until the morning it doesn’t — at which point somebody will go looking for the party who signed off, and discover that the signature belongs to a process that terminated eleven months ago, in a data centre in Virginia, leaving no capital, no licence, no insurer and no forwarding address.
Buy the boring companies. Buy the ones with professional indemnity and a partner whose name is on the door. Because when this one goes, and it will go, the fundamentals are going to come looking for someone to hold responsible — and there will be nobody left to sue.
References
- Ken Thompson, Reflections on Trusting Trust, Turing Award Lecture, Communications of the ACM, Vol. 27 No. 8, August 1984.
- Financial Crisis Inquiry Commission, Final Report of the National Commission on the Causes of the Financial and Economic Crisis in the United States, January 2011 — on the issuer-pays model and ratings failures.
- Peter Shergold & Bronwyn Weir, Building Confidence: Improving the effectiveness of compliance and enforcement systems for the building and construction industry across Australia, Building Ministers’ Forum, February 2018.
- Simon Willison, The Lethal Trifecta for AI Agents, simonwillison.net, June 2025.
- NSW Government, Office of the Building Commissioner — establishment and remediation programme, 2019 onward.
