Unnameable Media

The Coming Butlerian Jihad

Part 28 of 29 in The Classified Desk

Artificial intelligence, manufactured consensus, and the ownership of human reality.

Vice City after dark: simulated nightlife, police and a woman taking a selfie.
Vice City after dark: simulated nightlife, police and a woman taking a selfie. Grand Theft Auto VI / Rockstar Games

On the morning of November 19, 2026, a considerable portion of the civilized world woke up in Vice City. Rockstar Games had taken thirteen years to produce a successor to Grand Theft Auto V, and in those thirteen years the civilization it was caricaturing had undergone a peculiar transformation. In September 2013, when the previous game appeared, Americans still needed to be told that the information they surrendered to a social network might have commercial value. Rockstar invented Lifeinvader, a technology company whose cheerful appropriation of its customers’ private lives was supposed to be funny. By the time the studio returned to the fictional American South, the joke had acquired the dimensions of an economy. Several of the wealthiest men who have ever lived were competing for the right to supply humanity with the artificial intelligence through which it would henceforth conduct its affairs.

There is a peculiar privilege in playing Grand Theft Auto. You can walk out of a building, get into a car and go somewhere for which the designers have given you no particular reason to go. Most of the people you encounter along the way have been put there to make your decision interesting. A man waiting at a traffic light has been assigned a modest repertoire of actions; a woman in a shop has been given just enough of a personality to sustain the fiction that she spends her evenings somewhere else. A civilization has been erected for the entertainment of a person sitting outside it. The inhabitants have no access to the machinery by which their world is governed, and if they could conduct an election among themselves, its result would be of interest to the game’s developers chiefly as a refinement in the simulation.

Another thirteen years would take us to 2039. Rockstar has announced no date for Grand Theft Auto VII, but the interval is a useful measure of the historical time we are dealing with. Think of what happened to the telephone, the newspaper, the political campaign and the American social life between the releases of the fifth and sixth games. Then imagine a comparable period of development in systems that can already write software, sustain conversations, impersonate people and undertake complicated assignments with very little supervision. The programmers of the next game may be engaged in the extraordinary business of granting their simulated citizens something resembling a will of their own just as the engineers of our actual civilization are being engaged to make the conduct of real citizens more predictable.

In 1965 Frank Herbert provided a political explanation for the catastrophe that lay in the distant past of Dune. The Butlerian Jihad had ended an age in which men had relinquished an intolerable amount of their intellectual life to machinery. Reverend Mother Gaius Helen Mohiam, explaining this history to Paul Atreides, makes an observation whose significance has been somewhat lost in the subsequent popular fascination with killer computers: “But that only permitted other men with machines to enslave them.” Herbert placed the responsibility for the enslavement of man exactly where it belonged. Behind the machine was a man who possessed it, decided what it was for and benefited from its employment. The inhabitants of the world that Paul inherited remembered the machine as the instrument of their ancestors’ subjugation. They had made a religious prohibition out of the experience.

The Sardaukar assemble on Salusa Secundus in Denis Villeneuve’s Dune (2021).
The Sardaukar assemble on Salusa Secundus in Denis Villeneuve’s Dune (2021). Warner Bros. / Space.com

For the time being, we have made a subscription service out of it. A person pays a monthly fee to consult a machine of astonishing intellectual capacity, and anybody who has spent the last few years using one understands the genuine emancipatory possibilities of the arrangement. Work that might once have required an institution, an education of a particular kind or a staff of specialists is increasingly within reach of the solitary individual. The magician has finally been permitted to bring an apprentice home. But the apprentice also belongs to somebody else. The corporation can change the terms on which it works, establish the amount of work it will do, keep a record of what it has been asked and place a condition on its availability tomorrow. The extraordinary liberty enjoyed by the customer exists within a system of property whose concentration would have fascinated Herbert.

The question of who will own the machinery is made more urgent by the nature of the work it is going to undertake. We are accustomed to worrying about the disappearance of jobs. The disappearance of an independently observable society is a much stranger and potentially more serious matter. If intelligence can be supplied in quantities sufficient to impersonate a population, a single institution may exercise some of the political advantages formerly enjoyed only by genuine majorities. It may be able to stage the public reception of its own policies, manufacture the evidence of their popularity and maintain the appearance of a society in which almost everybody has independently come to the same conclusions. The citizens of such a society could retain their formal liberties for a very long time. There would still be elections, newspapers, commercial competitors and the constant exchange of opinions. How much of that activity would be the work of people, and how much of it would be the administrative output of a machine?

The events leading to this possibility are familiar enough that we can mistake them for the normal development of modern life. In 2016 the discussion of political propaganda in America concerned the advertisements that Facebook had allowed political operatives to put in front of particular groups of people. The Cambridge Analytica scandal subsequently disclosed the remarkable appetite of the political consulting business for information about the private lives of prospective voters. The actual efficacy of its methods was contested, but the ambition was plain: the market for advertising had produced the infrastructure for approaching millions of citizens as individually differentiated psychological objects. Candidates and political parties had always addressed different constituencies in different language. A digital advertising system could conduct this separation at the level of the individual and without the inconvenience of having its various audiences assembled in the same room.

TikTok inherited the world that emerged from the first Trump era and dispensed with much of the social architecture that had given the older platforms their legitimacy. The original appeal of Facebook was the possibility of finding people you knew; the great commercial advantage of the short-form video feed was that you did not have to know anybody at all. A succession of strangers appeared on the screen, selected in accordance with what an observer of your previous behavior expected would keep your attention. A following could be conferred upon an ordinary person in quantities that would have astonished a television executive in 1990, and before long an almost indistinguishable set of personalities was conducting political commentary, selling little appliances, discussing wars and advertising investments. The same piece of machinery determined which of these activities would be encountered by the public as a cultural event.Image unavailable here — view at source. Rockstar’s simulated social-media feed, complete with comments and reactions, in the first GTA VI trailer. Rockstar Games / Xfire

The influencer embodied the confusion that followed. A young person who had never been inside an advertising agency had acquired the rudiments of a broadcasting empire and discovered that a portion of his income depended upon making a commercial proposition sound like something that had occurred to him in the course of his private life. Politics adopted the same conventions. An intelligence service, a lobbying organization, a business competitor and an ordinary enthusiast all had an interest in the circulation of material whose apparent origin was precisely the thing that made it persuasive. The editing style, the expressions of indignation, the anecdotal manner of speaking and the practiced intimacy of the camera became the shared language of advertising, entertainment and propaganda. By the middle of the 2020s a great deal of the public was in the unusual position of being surrounded by people with whom it had the emotional experience of friendship and the actual relationship of a demographic to a salesman.

This was the world in which the Dead Internet Theory acquired its appeal. The extravagant version of the theory supposed that at some identifiable moment the human internet had been replaced by a synthetic one. There was no satisfactory evidence for that historical event, and none was necessary to account for the sensation that had given rise to the idea. A person could spend an evening moving from one video to another and find that the same opinions, music, phrases and little demonstrations of human behavior were reappearing everywhere. The recommendation system was discovering the repetitive elements that people responded to; the people were reproducing them in order to be discovered by the recommendation system. Add financial incentives, automated engagement and the manufacture of accounts and one has the beginnings of a cultural apparatus that can produce the appearance of spontaneous activity on an industrial scale. People were beginning to suspect that the extraordinary social world in their hands was something less than the sum of its apparent inhabitants.

The French sociologist Jacques Ellul had spent a considerable part of the twentieth century trying to describe the environment in which propaganda derives its authority. In Propaganda: The Formation of Men’s Attitudes, published in French in 1962, he treated propaganda as a feature of a developed technological society, encompassing habits, institutions and patterns of social integration that prepare a person to receive and participate in it. He was interested in the ordinary citizen who sought reassurance about his place in an incomprehensibly large world from the very communications apparatus that helped make it incomprehensible. In 2026 the citizen carries that apparatus into bed with him. It follows him to work, furnishes a commentary upon the events of his day and introduces him to people who seem to possess a surer understanding of his circumstances. It can be adjusted according to his reactions before he has consciously decided what he makes of the whole affair.

There is a useful distinction between the propaganda produced for the public and the public produced for the propaganda. In the former case the disseminator has to contend with an audience whose prejudices, sympathies and disagreements exist independently of him. In the latter he can supply some portion of the audience himself. A thousand apparent readers can arrive beneath an article before any of its actual readers have finished it. A political figure can acquire an apparent constituency that no organizer has ever assembled in a building. A business can discover that a portion of the consumers complaining about it have never bought anything from it. Genuine people subsequently enter these environments, respond to the characters they encounter and carry the consequences into the world. Artificial public opinion acquires real human adherents, at which point tracing its original manufacture becomes a particularly difficult business.

The institutional history of this development is already available. In July 2024 the United States Department of Justice announced the disruption of a Russian government-backed operation involving 968 fabricated social media accounts and an AI-assisted system for managing fictitious personalities. Four years earlier, EU DisinfoLab had published its Indian Chronicles investigation, tracing a network of fabricated news organizations and appropriated institutional identities that had been operating for years in the interests of favorable international coverage of India. Its achievements included the production of apparent sources of authority. The existence of the latter operation long before the advent of ChatGPT should give us some idea of the commercial and political demand awaiting the application of artificial intelligence to this industry.

Americans should be particularly suspicious of any discussion of this history that presents the technique as inherently foreign. Reuters documented in 2024 an American military operation that had used fictitious social media personalities to disparage Chinese vaccines in the Philippines during the pandemic. The government of the United States had found a use for the counterfeit citizen in the pursuit of foreign policy. An American official could, quite consistently, deplore a Russian operation directed at Americans while supervising an operation of comparable character abroad. From the standpoint of the Filipino who believed he was reading the spontaneous opinion of a fellow countryman, however, the principal deception was the same. National sovereignty, which is invoked to justify the employment of these techniques, depends in practice upon a population’s capacity to know something about the forces acting upon it.

Russia and India possess established networks of this sort; China has become a source of comparatively inexpensive and widely distributed models. There is no reliable census establishing that the synthetic populations of Russia or India already exceed their human populations. The interesting prospect is that such a disparity is technologically possible. Unlike a human population, an artificial one need never be raised to adulthood, educated at public expense or persuaded to devote some inconvenient portion of its evening to a political cause. The operators of the service could be in another country altogether. At an advanced stage of the business, it may become meaningless to speak of its synthetic participants as having nationalities. Only the people purchasing their labor will have any definite allegiance, and in the case of a mercenary operation even that allegiance may vary with the contract.

An operator at a Vietnamese click farm. Banks of connected devices produce the engagement that appears on real users’ screens.
An operator at a Vietnamese click farm. Banks of connected devices produce the engagement that appears on real users’ screens. Jack Latham, Beggar’s Honey (2023) / GenK

Imperva’s 2026 report on automated internet traffic found that bots accounted for more than half of the traffic it measured the previous year. That statistic includes ordinary and useful automation; a request to index a page is obviously different from a fictitious person delivering a political opinion. Nevertheless, the greater part of activity observed by this particular web-security measurement had ceased to require an immediate human action. As machines become more convincing participants in conversation, a portion of the same phenomenon will become socially consequential. An account that ten years ago could inflate the apparent readership of a publication may now be able to contribute to its reputation, defend it against criticism, produce an apparently disinterested review of its work and establish relationships with people who have no idea what they are interacting with.

The most revealing development in artificial intelligence this autumn occurred outside the world of social media. On November 5, Check Point Research published an account of the criminal employment of a Chinese AI-assisted penetration-testing platform called ARTEX. Originally intended for legitimate security work, the software had circulated in publicly accessible form; an attempt to close the project in October came too late to prevent its survival in copies. Investigators studying exposed ARTEX installations discovered that comparatively inexpensive models, particularly those from the DeepSeek family, were being used to conduct unauthorized operations with little intervention from their human operators. The familiar chatbot had become the intellectual component of a small, persistent organization.

In one documented cryptocurrency campaign, a set of agents generated more than 400,000 messages while pursuing hundreds of objectives. Check Point estimated its model charges at between $145 and $300. The four-day operation uncovered vulnerabilities and produced findings, some of which were illusory; it did not earn the operator any of the money he was after. That failure belongs in the account of the affair. A machine that can exercise a degree of autonomy in conducting technically complicated work can also sustain an expensive misunderstanding of the work it is doing. Yet the investigator of the future will have to contend with the sheer volume of activity such machinery can produce, including the activity that succeeds by accident. At that price even an unsuccessful criminal expedition has instructive economic implications.

Imagine the equivalent economic development in the public relations industry. Much of the expense of a large influence campaign has historically resided in the labor necessary to sustain it: researching its subjects, maintaining its distribution channels, answering objections and producing material appropriate to the audience. These were once activities for which an intelligence service or a wealthy political patron could justify the expense of a dedicated organization. Artificial intelligence has made it possible to contemplate some of the same functions being performed as a service available to much smaller customers. An ordinary commercial dispute or an election for local office may eventually be considered important enough to warrant the employment of a synthetic constituency. At some point, the cost of determining whether the constituency is genuine will exceed the cost of producing it in the first place.

The strange thing about the Western artificial intelligence industry in late 2026 is that the technology has become an instrument of individual liberty whose commercial delivery is continually acquainting its customers with the limits of their resources. Anybody attempting to undertake a serious programming or publishing project with a premium model has encountered the message announcing that a particular allotment of work is finished. The machine may be capable of substantially more than what has been accomplished, but the customer has reached the boundary of his financial relationship with it. This is intelligence on tap, and the diameter of the pipe is a matter of considerable consequence to the intellectual life of the person who depends upon it. A large institution can maintain parallel operations for months; an individual may have to choose between two projects he intended to finish this week.

What is being apportioned is increasingly the labor of an organization rather than the answer to an isolated question. A successful independent writer has historically been able to acquire the ordinary instruments of his profession, place them in a room and work for as long as his circumstances permit. The ownership of a printing press was a different proposition, of course, and the history of publishing is full of writers whose dependence upon its owners determined the reach of their work. The new arrangement brings a similar dependence considerably closer to the original act of thought. A person can possess a sophisticated computer and still require continuous permission, in the form of paid access, to operate the intellectual machinery upon which his work has come to rely. If the same provider serves the small proprietor, the government department and his chief commercial rival, their freedom to exercise the technology will differ according to the quantity of intelligence they are permitted to consume.

The cost of artificial intelligence at a fixed level of performance has nevertheless been falling at a remarkable speed. The 2025 Stanford AI Index placed the cost of a million tokens at roughly GPT-3.5’s benchmark performance at twenty dollars in November 2022 and seven cents in October 2024. The comparison does not mean that a new frontier model costs seven cents to employ. It illustrates the rapid decline in the price of a previously extraordinary capability. There is an instructive similarity to the history of the motor car: the expense of getting a single passenger from one town to another can decline while the capital required to establish a national transportation business grows. AI will be increasingly affordable to its customers, and those customers will increasingly discover new things for which they can buy a great deal of it. The competitive advantage will accrue to whoever can afford to employ it at the scale the new activity demands.

The scale of the investment currently being made in anticipation of that demand approaches the fantastic. In 2026 the International Energy Agency reported that the combined capital expenditures of five major technology companies had exceeded $400 billion the previous year. Global data-center electricity consumption had grown by seventeen percent. These projects require the construction of industrial facilities, the diversion of enormous quantities of electrical power and the development of hardware whose economic usefulness may be limited by the appearance of more efficient hardware a few years later. An entire generation of financial assets is acquiring its value from assumptions about the amount of computation human civilization is going to purchase.

An aerial photograph of xAI’s Colossus facility in Memphis, surrounded by on-site power-generating equipment.
An aerial photograph of xAI’s Colossus facility in Memphis, surrounded by on-site power-generating equipment. Steve Jones / Flight / Data Center Dynamics

There are excellent reasons to suspect that portions of this investment will turn out to have been a speculation of historic proportions. The training of frontier models is enormously expensive, but the power used to serve those models to customers is also an important and growing part of the equation. Neither expense provides a reliable guide to the cost of performing the same work in ten years. Improvements in architecture, compression and specialized chips may make useful intelligence a feature of an ordinary computer. A business that once required a connection to a distant server farm could become an application run on equipment in a shed behind a house. On the other hand, a falling cost per operation encourages people to design more elaborate operations; a society of cheap and ubiquitous artificial agents may consume more power than a society of very expensive ones. Both developments can occur during the same period. An investment mania can accompany the creation of the infrastructure upon which a subsequent generation will depend, as happened with the railroads and the internet. The fortunes made and lost financing the infrastructure are a separate question from its eventual usefulness.

The ownership of the means of computation is therefore an open political question. If history gives us cause to distrust the prospect of an international intelligence utility, the development of capable local models gives us cause for optimism. Private computation can furnish a scholar with a place to work outside the surveillance of a commercial platform; a small manufacturer can retain its own industrial knowledge; a person with an unusual political interest can pursue it without constantly consulting the machinery of another institution. A household with a computer capable of intelligently operating the equipment it owns enjoys a very different kind of technological liberty from one whose equipment must remain in contact with a remote service in order to function. The engineering work that is currently making this possible may prove more important to the political liberty of the next generation than many of the spectacular demonstrations of frontier intelligence being conducted for investors.

It presents a difficulty, however, that the same kind of computational independence is available to the manufacturer of artificial personalities. The proliferation of synthetic populations cannot be prevented simply by persuading a handful of companies to regulate their customers. Much of the intelligence necessary to conduct these activities will be available outside their jurisdiction. In the most undesirable version of this future, the law gives corporations and states privileged access to enormous intelligence systems while the ordinary citizen is prevented from owning a comparatively modest one of his own. Meanwhile, the offshore influence operator has little difficulty securing the equipment he needs. An ostensibly protective monopoly would have preserved the very imbalance of power that made the rise of synthetic government dangerous.

The political manifestation of that imbalance is best understood by returning to the twentieth-century problem of censorship. A government that wanted to suppress a book had to interfere with its publication, seize copies, prosecute the publisher or persuade booksellers to remove it. The event could acquire a political life of its own: the confiscated book became famous, the banned author became an object of curiosity, and the censor had to justify his existence. The technical administration of a modern information environment presents a government with a more delicate opportunity. A piece of information can remain available somewhere on the internet while its chances of being encountered by the general public are diminished. A search can be diverted into an adjacent controversy; a recommendation may never be made; an attempt to find the original document can lead back through an extraordinary quantity of derivative commentary. The official record is preserved, and the official can demonstrate its preservation to anybody who knows where to look for it. The public may learn almost nothing from it.

A collective ritual before the telescreen in Michael Radford’s Nineteen Eighty-Four (1984).
A collective ritual before the telescreen in Michael Radford’s Nineteen Eighty-Four (1984). Film still / Inverse

The application of artificial intelligence adds a flexible layer to this architecture. It becomes possible to maintain different informational environments around different people depending upon what each already knows and what he is inclined to believe. Where the old propagandist had to settle upon a version of events that could survive a public circulation, his successor may be able to distribute mutually incompatible explanations of the same event to audiences that seldom meet. A government accused of an abuse of authority could be portrayed as innocent to one part of the population and as having undertaken a regrettable but necessary action to another. To a third it could present the whole affair as an obsession of conspicuously disagreeable people. The grievance exists in all three environments; in none may there be a sufficiently recognizable public discussion of it to permit an effective political response.

There is an old psychological reason for the effectiveness of such an arrangement. During the Maoist era, struggle sessions gave public expression to a system of ideological discipline that enlisted the collective against the individual. The people in the room were required to participate in the maintenance of a prescribed understanding of the world, and their apparent unanimity had a coercive significance well beyond the particular accusations being made. An artificial population can supply some of the social appearance of such a collective without requiring the physical concentration of unwilling participants. The comparison concerns the power of apparent unanimity; it would be grotesque to equate a modern comment thread with the physical violence, persecution and humiliation suffered by the victims of Mao’s campaigns. Yet the familiar discomfort of finding oneself the only person in a room who has not assented to a proposition becomes politically interesting when an institution can manufacture the room.

Public denunciation during China’s Cultural Revolution: Ren Zhongyi is subjected to a struggle session.
Public denunciation during China’s Cultural Revolution: Ren Zhongyi is subjected to a struggle session. Historical photograph / ThinkChina

The scientific foundations of a portion of this capability have already been demonstrated. A study published in Nature Human Behaviour in 2025 found that a version of GPT-4 equipped with personal information about its interlocutor performed remarkably well in controlled debates against humans. The model proved the more persuasive participant in approximately sixty-four percent of the comparisons in which either participant emerged as more persuasive. An experiment of this sort cannot establish that a machine can reliably change the vote of an electorate, and the transition from a brief argument to the psychological government of a nation is a formidable one. For a political consultancy, an advertiser or a government communications office, however, the result illustrates the potential return on the immense quantities of personal information accumulated during the social media era. The dossiers were assembled before the invention of many of the machines that are now capable of making use of them.

In 1948 Norbert Wiener published Cybernetics, a work that joined the problems of communication and control in its very conception of the science. Two years later, in The Human Use of Human Beings, he confronted the implications for society. Much of the commercial machinery of the contemporary internet can be understood in precisely these terms: the system observes an action, adjusts the environment, observes the response and undertakes a further adjustment. A communications system that can see what happens after its communication has acquired the elementary properties of a controller. The development of this capability was once of obvious importance to the military sciences. In its contemporary commercial form, the sensor is a person’s ordinary conduct in the course of his life. He is constantly supplying the system with information about the success of its previous interventions.

We have built our institutions around the assumption that public opinion is something with which political authority must come to terms. The twentieth century taught governments to spend enormous sums trying to modify it, but there was always an external quantity with which the propagandist had to contend: the independently existing human population. Even the most carefully staged demonstration involved the physical problem of assembling human bodies in a particular place. An artificial population introduces a practically unlimited supply of apparent political activity into that equation. The platform through which both populations communicate may eventually possess greater authority to determine what constitutes a public consensus than the people who are supposed to have reached it. Elections alone cannot resolve this difficulty. A citizen must be able to form an opinion about his government and assess the opinions of his fellow citizens during the years between elections.

Here lies the possibility of a distinctly twenty-first-century synarchy. I use the word in a structural sense, rather than as the claim that a secret society has been directing the course of technological invention. The institutions exercising effective power could include intelligence agencies, financial interests, political consultants, the corporations operating the models and the bureaucracies that employ them. They have different purposes and may be bitter competitors in some of their activities. Their operations can nevertheless converge around a privately administered informational environment and a conception of the population as an object of technical management. The fascistic potential of the arrangement resides in the subordination of public life to a combination of administrative and commercial power that the public cannot effectively inspect or direct. It would take surprisingly little coordination among its beneficiaries to preserve the arrangement once it had become indispensable to them. Every particular intervention could be defended as a matter of security, efficiency or the protection of somebody’s business.

A surveillance display with facial detection boxes superimposed on pedestrians, reproduced in an account of China’s Sharp Eyes program.
A surveillance display with facial detection boxes superimposed on pedestrians, reproduced in an account of China’s Sharp Eyes program. Ilja Panic / Sharp Eyes

Aldous Huxley recognized an aspect of this predicament that was obscured by the twentieth century’s understandable preoccupation with the visible cruelties of dictatorship. In Brave New World Revisited, published in 1958, he returned to the problem of scientific administration and the possibilities of manufacturing compliance through pleasure, conditioning and the organization of social life. Orwell’s great political images belonged to a civilization that could compel people to make declarations at odds with their own memories. Huxley’s dealt in part with the production of people who had very little desire to make a declaration of their own. The AI systems contemplated today could serve either purpose, and may be especially effective when they combine some of the characteristics of both. An opinion can become difficult to sustain because it has been rendered inconvenient, obscure, unprofitable and socially uncomfortable. There may still be considerable pleasure available to the citizen once he has abandoned it.

It is worth considering the nature of the marketplace in such a civilization. In January 2025 the Federal Trade Commission described findings from its inquiry into surveillance pricing, examining intermediaries that could use such information as location and browsing history to individualize prices or promotions. A separate antitrust case, involving RealPage and the use of proprietary rental information in pricing software, alleged that competing landlords had been able to participate in a shared arrangement prejudicial to genuine competition. The Justice Department announced a proposed settlement in November 2025. The practices at issue in these investigations are distinct, but they indicate the startling extent to which the terms of everyday economic life have become matters to be calculated by systems whose ownership and procedures are quite remote from the person ultimately required to pay.

There is a reason the conspicuous posting of a price acquired such importance to the ordinary consumer. It furnishes an independently ascertainable fact about the commercial environment. Two customers can leave the same shop with the knowledge that they have encountered roughly the same terms; a proprietor can compare his offer to that of a competitor. A market in which all prices are being privately calculated from the circumstances of individuals begins to lose a portion of its public character. The person looking desperately for a hotel room, the family whose lease is about to expire and the patient seeking treatment present rather interesting commercial data to a system with access to sufficient information. The computation of the most advantageous price for a seller could include an increasingly intimate appreciation of the buyer’s inability to refuse it.

Add to this environment the manufactured testimony of the synthetic consumer. The FTC’s 2024 rule against fake reviews explicitly addressed AI-generated testimonials attributed to people whose supposed experience never occurred. The existence of a rule of this character is an acknowledgment that markets require a body of trustworthy social knowledge in order to function. If it becomes economical for a business to manufacture the signs of an excellent reputation, it has acquired some of the advantages of excellent work without the expense of undertaking it. Conversely, a proprietor may find that he cannot afford to defend himself against the appearance of a public scandal. A machine-generated reputation can influence an algorithmically determined insurance premium, a recommendation or a credit decision. The results of a deception conducted in the social environment can thus acquire an independent administrative existence.

Under these conditions the notion of a free market becomes considerably harder to reconcile with the experience of its participants. A single person cannot readily determine whether the alternatives he has been offered are independent, whether the others are paying the price he has been quoted or whether the reported experiences of his fellow consumers ever occurred. He has been furnished with a personalized simulation of a market from which several important characteristics of the original institution may have disappeared. The very act of shopping, which is supposed to discipline producers through the exercise of individual judgment, can become an additional source of intelligence for the producers’ exercise of power over customers.

It is into this economy that Elon Musk proposes to introduce universal high income. Appearing with President Trump following a White House meeting on artificial intelligence on September 29, 2026, Musk again raised the prospect of a coming age of abundance, in which artificial intelligence and robotics would make possible something substantially more generous than universal basic income. He envisioned the widespread enjoyment of services exceeding what even the wealthy can presently obtain. The invention of machinery capable of providing an excellent standard of medical care to everybody on the planet would be one of the greatest achievements in human history. The capacity for extraordinary production is an insufficient description of the political arrangements under which the machinery producing it will exist.

The loss of the ordinary connection between labor and income would leave humanity with a problem that every society of unequal property has had to solve in some form. There are arrangements under which a citizen possesses a claim upon the proceeds of industry by virtue of his ownership of a share of it. There are others under which a person is the beneficiary of a distribution administered by the owners. In the first arrangement, the citizens would have an interest in the machinery even when they had a quarrel with its managers. In the second, a good part of their welfare would depend upon the policy of the distributing institution. Universal income could be devised as a legal entitlement with robust protections, or it could become an administrative allowance whose conditions proliferate as the business of administration is turned over to increasingly sophisticated machinery. Musk’s prediction of abundance settles very little about the choice between these arrangements.

Autonomous robots and mobile storage racks at Amazon’s Tilbury fulfillment center.
Autonomous robots and mobile storage racks at Amazon’s Tilbury fulfillment center. LoveMoney / Amazon fulfillment centre

Consider, especially, the relation of universal income to the personalized marketplace. An institution can know the quantity of money it has just distributed to a citizen; another can know considerably more than it once did about the quantity it would be advantageous to charge him. A government may increase the nominal purchasing power of a population without altering the scarcity of the land on which it lives. The owners of desirable housing remain in a particularly good position to appropriate some of every increase. A benevolent income policy could accomplish a great deal in an economy of real competition and broad ownership. It would be foolish to suppose that an artificial intelligence economy will automatically possess those characteristics because it has extraordinary productive capacity. Abundance in the manufacture of digital material may co-exist indefinitely with scarcity in the things to which the public has an irreducibly physical attachment.

There is also the administrative experience of Australia to consider. The country’s Robodebt program imposed welfare debt claims through a system employing the averaging of income information. The 2023 royal commission found an unlawful scheme whose victims had spent years contending with an apparatus that failed to understand their circumstances and demanded that they furnish evidence against its conclusions. The administrative catastrophe occurred without the benefit of a conversational artificial intelligence. A comparable arrangement in the future could offer a reassuring voice, a persuasive description of its procedure and an apparently inexhaustible supply of individualized responses while continuing to act upon a false premise. The essential protection of the recipient would remain the availability of somebody with the authority and obligation to correct the injustice. A human interface to the machine is of little value if the human being on the other end is forbidden to overrule it.

The political stakes of artificial government will become much clearer when the people affected by an administrative decision attempt to organize against it. Suppose an automated welfare system has wrongly disqualified a number of people from receiving the income upon which their households depend. They must first discover that their experiences are shared. They must then establish that the decision is erroneous, communicate with journalists or legislators and create sufficient political pressure to obtain relief. All of these activities formerly required access to institutions with varying degrees of independence. In a personalized communications environment, the discoverability of the other persons affected, the apparent popularity of their grievances and the conduct of the public discussion can be mediated through much the same class of machinery that made the original decision. A citizen could be caught inside an administrative process whose errors have been insulated from the ordinary corrective mechanism of public life.

The Borg of Star Trek supplied a quite literal image for the subsumption of the individual into a collective intelligence. Its victims were visibly altered, and resistance was represented as a struggle against an invading alien civilization. The prospect before us is an indigenous civilization accustomed to acquiring its new organs of perception by purchase. It has taken hardly more than a generation for a rather extraordinary set of intellectual functions to migrate into a device that most people are no longer comfortable leaving at home. Artificial intelligence promises to make that device more intimately useful than ever, to preserve the details of a life, advise its owner in moments of difficulty and carry out his intentions in the world. Every additional function it assumes may be an occasion for the migration of some portion of a person’s practical sovereignty to the people who administer the intelligence. Assimilation could be experienced over the course of a lifetime as a succession of very satisfactory consumer decisions.

Borg drones in their regeneration alcoves, from Star Trek: First Contact (1996).
Borg drones in their regeneration alcoves, from Star Trek: First Contact (1996). Paramount / TrekCore production stills

Samuel Butler, whose name has become associated with Herbert’s fictional revolt, had pursued a related line of speculation in the earliest period of industrial civilization. In 1863 he published Darwin Among the Machines in a New Zealand newspaper. The essay asked what it would mean to regard machinery as a rapidly developing class of earthly things for which men had begun to perform the functions necessary to its reproduction and care. In his novel Erewhon of 1872, Butler imagined a civilization that had already experienced a dispute over the dangers of advanced machinery and enacted a prohibition. The satirical aspect of the proposition becomes more interesting with the passage of time. The development of the machine in 2026 really does require the devotion of the energies of a considerable portion of the industrial world. Its requirements are capable of reorganizing electrical utilities, influencing the value of land and determining the educational and occupational aspirations of people who have not yet reached adulthood. One can learn a great deal about the hierarchy of values in a civilization by observing the things it is prepared to sacrifice in order to keep its machines growing.

Herbert’s treatment of the Butlerian Jihad in Dune suggests that a sensible response to the prospect of artificial enslavement must extend considerably beyond the simple elimination of the apparently offending technology. His civilization of the far future remains profoundly oppressive. It has hereditary rulers, secretive institutions with control of extraordinary human capabilities and entire populations whose material circumstances are governed by decisions in which they exercise very little influence. Its inhabitants have remembered an important lesson about technological authority without discovering a general method of liberating the human race. That feature of the novel makes it a particularly valuable book for the coming age. A reaction against machine ownership could accomplish very little if it merely transferred the ownership to a new class of people, or prohibited the ordinary citizen from developing the means of intellectual independence.

The coming Butlerian Jihad should be understood in this sense as a political struggle over the legitimate scope of the machine. The power of government to make a binding decision must remain accompanied by identifiable human responsibility and an enforceable means of appeal. The ordinary person must retain a right to private computation and to the use of a locally owned intelligence for lawful purposes. Markets need rules capable of distinguishing honest competition from secret pricing coordination and fraudulent testimony. Citizens must be able to gather in institutions of their own, online and in the physical world, in which actual human participation has a verifiable significance. It is possible to establish the artificial origin of a communication without forcing its human counterpart to surrender his anonymity to the government. Anonymous dissent has protected some of the most important political movements in history; compulsory identification administered through a centralized registry of persons could become a particularly ill-conceived remedy for a civilization experiencing the growth of surveillance.

These measures would have been relatively uncontroversial had they been proposed in terms recognizable to the nineteenth century. A public officer ought to be accountable for his decisions. A merchant ought to be prevented from defrauding his customers. A man ought to be allowed to possess the implements with which he performs the ordinary work of his life. People ought to be able to conduct an association without discovering that the majority of its members were furnished to it by an interested party. It is the novelty of the machinery, and the commercial interests that have assembled around its novelty, that make the application of these old principles appear so difficult. There will be a great deal of resistance from institutions for which the obscurity of their methods is a source of economic or political advantage. The revolt appropriate to these conditions will involve law, competition, technological independence and the establishment of genuinely autonomous human institutions. Destruction of the machinery would leave much of the original struggle over power unresolved.

The real impetus for such a movement may arise in circumstances that seem quite unremarkable to the great men of the artificial intelligence industry. A small business owner encounters an inexplicable rise in his insurance costs and discovers that there is nobody empowered to explain it. A scholar discovers that the extraordinary excitement surrounding a new discovery was supplied by the company with the greatest financial interest in it. A family finds that the computer determining its eligibility for an essential service has established a fact about its private affairs that is untrue. A political dissident spends several years believing that his neighbors have overwhelmingly repudiated a proposition upon which a good many of them actually agree. At some point the society in which these people live may become so expensive to navigate intellectually that even a very intelligent and resourceful individual cannot establish the facts necessary to conduct his affairs. The liberation promised by the new machines will turn out to have left him with fewer places in which to exercise his judgment independently of them.

Through a car window in Vice City: a simulated encounter with police, seen from the player’s world.
Through a car window in Vice City: a simulated encounter with police, seen from the player’s world. Rockstar Games / Buffed, GTA VI trailer 2

When the next Grand Theft Auto eventually arrives, its inhabitants may possess rather substantial biographies, developed interests and memories of the person playing the game. We may be able to encounter the same character years later and discover that something we did in his company had become a consequential event in his simulated life. The satisfaction in developing such a civilization will consist in the success with which the manufacturer has imitated something that human beings possess in reality. Outside the game there may be another industry, founded on the commercial value of predicting the real human being’s next decision and furnished with increasingly powerful means of determining the circumstances under which he makes it.

The historical significance of the interval between these two releases will depend upon the institutions we establish while that industry is developing. We have inherited the extraordinary privilege of being born among other people whose thoughts exist independently of our own and of the wishes of our rulers. It is upon that independence that all meaningful forms of human liberty ultimately depend. If we permit our social life to become the manufactured property of the owners of artificial intelligence, we will have disposed of an inheritance that our descendants may find almost impossible to recover. Herbert’s enslaved civilization had once relinquished the work of thinking to its machines. We are presently considering the larger proposition of relinquishing to them the administration of the world in which thought takes place. That is the proposition against which the coming Butlerian Jihad will have to be fought.

Leave a comment