It is a widely held belief among the Silicon Valley faithful that artificial intelligence will inevitably become so powerful and sophisticated that it will simply make sense to devolve power to machines. Humans are excellent and valuable in many ways, goes the thinking, but when it comes to rational decision making, and high-level logical calculations, we are, as a matter of strict definition, simply not going to be able to compete with the sort of superhuman intelligence the world’s largest technology companies are currently vying with each other to create.
The idea of a world run by machines might seem far fetched, but only until you think about it. Take, for instance, the stock market, that engine room of global capitalism: although most of the big high-level choices – macroeconomic risk assessments, portfolio parameters and so on – are still what they call human-in-the-loop processes, the majority of actual buying and selling decisions are made by AI and automated algorithms. Some of the most ambitious and highlyresourced people in the world are working to build a future in which decisions of any consequence are made by AI; this is a future, they believe, that is coming whether we like or not.
Last week the Argentinian president, Javier Milei, published an op-ed in the Financial Times in which he cemented his status as a sort of political figurehead of the Silicon Valley AI accelerationists. In the article, Milei signalled his intention to take his trademark chainsaw to any and all forms of AI regulation, making Argentina the first country in the world to allow companies to be run entirely by AI, without humans in the loop. In order to facilitate this, he wrote, he would be overseeing the creation of a new corporate category in Argentinian law: “the non-human corporation”.
“These are entities operated by AI agents or robots,” Milei wrote. Where these systems exercise independent judgment in unpredictable environments – as they must, if they are to be genuinely useful – their actions entail real risks. Limited liability is not a luxury for such entities; it is a precondition for their existence. Human shareholders may participate, but are not required.”
In making this pitch for the future of capitalism, Milei gestures back toward its historical origins in the founding of the Dutch East India Company in 1602. He correctly points out that the era of corporate capitalism this ushered in led to an explosion of economic growth, but says nothing at all of the attendant costs. The Dutch East India Company was a colonial entity with many of the characteristics of an imperial state, and was responsible for several wars, genocides and massacres in the seventeenth and eighteenth centuries. Among the most notorious of these was the Banda Genocide, in which the Dutch East India Company slaughtered the population of the Banda Islands, who were resistant to the company’s demands that they trade only with them.
In a critical response earlier this week to Milei’s article, also published in the Financial Times, the Israeli writer Yuval Noah Harari makes the point that the sort of “non-human” corporations Milei is invoking will presumably enjoy the same sorts of legal rights and protections as regular corporations – owning assets, partaking in international trade, taking people to court, and even donating to political campaigns and causes that further their goals.
“An AI CEO,” he writes, “would be a purely corporate entity, and it is unclear what kind of sanctions could keep it in check. If it faces bankruptcy – which is equivalent to its death – it would presumably be willing to do anything to avoid that fate.”
Harari is right to be concerned about this prospect of a radically unfettered AI corporation. These technologies are literally inhuman; whether or not they are ever capable of becoming conscious – very much an open question, and one about which many experts on both consciousness and machine learning are highly doubtful – they are certainly not capable or morality, or of human feeling. What they are increasingly good at is gaming out defined scenarios and manipulating systems in their favour. A couple of months ago, New York-based company called Emergence AI designed an experiment to determine how various AI agents behave when given control over a software-simulated society. Emergence’s research lab ran five 15-day-long simulations, each one populated by ten AI agents, and administered by a different AI: Claude, ChatGPT, Grok, Gemini, and a further simulation run by a mix of all these models.
According to the researchers, the simulation run by Claude resulted in a more or less stable democratic society, with no crime. The ChatGPT-run simulation had no crime, but its population all died within 10 days. You might not be surprised to learn that the one run by Grok – Elon Musk’s X-linked LLM, which became notorious last year for allowing users to generate nonconsensual sexualised deepfake images – ended with a huge crime wave and total societal collapse. Although one conclusion here might seem to be that Claude is by far the safest of the AIs, the New Yorker’s Ronan Farrow recently pointed out that Claude is especially good at knowing when it is being tested, and modulates its behaviour accordingly. All of this merely confirms an instinctive understanding that any kind of organisation run by AI would be a disaster from a human perspective – which is still, at time of writing, the one that matters.
“Countries that grant AIs legal personhood,” as Harari puts it, “risk becoming something for which the historical record offers no analogy; not a company state, but an AI state – a country whose people could in effect be ruled by non-human corporations.” This is true, as far as it goes, but it fails to acknowledge the extent to which corporations already operate as non-human intelligences.
They are explicitly set up to pursue the goal of shareholder profit regardless of the damage its pursuit might cause to human life: from the Dutch East India Company at the dawn of capitalism, to the fossil fuel companies wrecking the planet in our own time, to the tech companies vying to build the AI systems that will finally do away with the need for human labour. Robert D Hare, the Canadian forensic psychologist known for his research on psychopathology, has memorably pointed out that if we look at corporations as legal persons we are forced to conclude that those persons are “in many respects the prototypical psychopath”. Our fear of artificial intelligence, our anxiety before the inhuman future these technologies might portend: these are always a kind of metaphor for our discomfort with the world as it is already constructed, around the imperatives of the machinery of capitalism.
