This is the promise of world models. For years, artificial intelligence has been defined by its ability to process and translate information — to autocomplete, recommend and generate. But a different AI paradigm seeks to expand its capabilities further. World models are systems that simulate how environments behave. They provide spaces where AI agents can predict how the future might unfold, experiment with cause and effect, and, one day, use the logic they acquire to make decisions in our physical environments. 

Large language models currently have the attention of both the AI industry and the wider public, showing remarkable and diverse capabilities. Their multimodal variants can generate exquisite sushi recipes and describe Big Ben’s physical properties solely from a photograph. They guide agents through game environments with increasing sophistication; more recent models can even integrate vision, language and action to direct robot movements through physical space.

Their rise, however, unfolds against a fierce debate over whether these models can yield more human-like and general intelligence simply by continuing to scale them through investing in their parameters, data and compute.

While this debate is not yet settled, some believe that fundamentally new architectures are required to unlock AI’s full potential. World models present one such different approach. Rather than interacting primarily with language and media patterns, world models create environments that allow AI agents to learn through simulation and experience. These worlds enable agents to test “what happens if I do this?” by counterfactually experimenting with cause and effect to hone how they perform their actions based on their outcomes.

To understand world models, it helps to distinguish between two related concepts: AI models and AI agents. AI models are machine learning algorithms that learn statistical patterns from training data, enabling them to make predictions or generate outputs. Generative AI models are AI models capable of generating new content, which is then integrated into systems that users can interact with, from chatbots like ChatGPT to video generators like Veo. AI agents, by contrast, are systems that use such models to act autonomously in different environments. Coding agents, for example, can perform programming tasks while using digital tools. The abundance of digital data makes training such agents feasible for digital tasks, but enabling them to act in the physical world remains a harder challenge.

World models are an emerging type of such AI models that agents can use to learn how to act in an environment. They take two distinct forms. Internal world models are abstract representations that live within an AI agent’s architecture, serving as compressed mental simulations for planning. What can be called interactive world models, on the other hand, generate rich, explorable environments that any user can explore, and agents can train within.

The aspiration behind world models is to move from generating content to simulating dynamics. Rather than providing the steps to a recipe, they seek to simulate how rice responds to pressure, enabling agents to learn the act of pressing sushi. The ultimate goal is to develop world models that simulate aspects of the real world accurately enough for agents to learn from and ultimately act within them. Yet this ambition to represent the underlying dynamics of the world rather than the surface patterns of language or media may prove to be a far greater challenge, given the staggering complexity of reality.

Our Own World Models

Since their conceptual origins decades ago, world models have become a promising AI frontier. Many of the thinkers shaping modern AI — including Yann LeCun, Fei-Fei Li, Yoshua Bengio and Demis Hassabis — have acknowledged that this paradigm could pave new pathways to more human-like intelligence.

To understand why this approach might matter, it helps to take a closer look at how we ourselves came to know the world.

Human cognition evolved through contact with our three-dimensional environment, where spatial reasoning contributes to our ability to infer cause and effect. From infancy, we learn through our bodies. By dropping a ball or lifting a pebble, we refine our intuitive sense of gravity, helping us anticipate how other objects might behave. In stacking and toppling blocks, babies begin to grasp the rules of our world, learning by engaging with its physical logic. The causal structure of spatial reality is the fabric upon which human and animal cognition take shape.

The world model approach draws inspiration from biological learning mechanisms, and particularly from how our brains use simulation and prediction. The mammalian prefrontal cortex is central to counterfactual reasoning and goal-directed planning, enabling the brain to simulate, test and update internal representations of the world. World models attempt to reproduce aspects of this capacity synthetically. They draw on what cognitive scientists call “mental models,” abstracted internal representations of how things work, shaped by prior perception and experience.

“The mental image of the world around you which you carry in your head is a model,” pioneering computer engineer Jay Wright Forrester once wrote. We don’t carry entire cities or governments in our heads, he continued, but only selected concepts and relationships that we use to represent the real system. World models aim to explicitly provide machines with such representations.

While language models appear to develop some implicit world representations through their training, world models take an explicit spatial and temporal approach to these representations. They provide spaces where AI agents can test how environments respond to their actions before executing them in the real world. Through iterative interaction in these simulated spaces, AI agents refine their “action policies” — their internal strategies for how to act. This learning, based on simulating possible futures, may prove particularly valuable for tasks requiring long-horizon planning in complex environments. Where language models shine in recognizing the word that typically comes next, world models enable agents to better predict how an environment might change in response to their actions. Both approaches may prove essential — one to teach machines about our world, the other to let them rehearse their place within it.

This shift, from pattern recognition to causal prediction, makes world models more than just tools for better gaming and entertainment — they may be synthetic incubators shaping the intelligence that one day emerges, embodied in our physical world. When predictions become actions, errors carry physical weight. While this vision remains a relatively distant future, the choices we make about the nature of these worlds will influence the ethics of the agents that rely on them.

How Machines Construct Worlds

Despite its recent resurgence, the idea of world models is not new. In 1943, cybernetics pioneer Kenneth Craik proposed that organisms carry “small-scale models” of reality in their heads to predict and evaluate future scenarios. In the 1970s and 1980s, early AI and robotics researchers extended these mental model foundations into computational terms, using the phrase “world models” to describe a system’s representation of the environment. This early work was mostly theoretical, as researchers lacked the tools we have today.

A 2018 paper by AI researchers David Ha and Jürgen Schmidhuber — building on previous work from the 1990s — offered a compelling demonstration of what world models could achieve. The researchers showed that AI systems can autonomously learn and navigate complex environments using internal world models. They developed a system architecture that learned to play a driving video game solely from the game’s raw pixel data. Perhaps most remarkably, the AI agent could be trained entirely in its “dream world” — not literal dreams, but training runs in what researchers call a “latent space,” an abstract, compact representation of the game environment. This space serves as a compressed mental sketch of the world where the agent learns to act. 

Without world models, agents must learn directly from real experience or pre-existing data. With world models, they can generate their own practice scenarios to distill how they should act in different situations. This internal simulation acts as a predictive engine, giving the agent a form of artificial intuition — allowing for fast, reflexive decisions without the need to stop and plan. Ha and Schmidhuber likened this to how a baseball batter can instinctively predict the path of a fastball and swing, rather than having to carefully analyze every possible trajectory.

This breakthrough was followed by a wave of additional progress, pushing the boundaries of what world models could represent and how far their internal simulations could stretch. Each advancement hinted at a broader shift — AI agents were beginning to learn from their own internally generated experience.

Recently, another significant development in AI raised new questions about how agents might learn about the real world. Breakthroughs in video generation models led to the scaled production of videos that seemed to capture subtle real-world physics. Online, users admired tiny details in those videos: blueberries plunging into water and releasing airy bubbles, tomatoes slicing thinly under the glide of a knife. As people shared and marveled at these videos, something deeper was happening beneath the surface. To generate such videos, models reflect patterns that seem consistent with physical laws, such as fluid dynamics and gravity. This led researchers to wonder if these models were not just generating clips but beginning to simulate how the world works. In early 2024, OpenAI itself hypothesized that advances in video generation may offer a promising path toward highly capable world simulators. 

Whether or not AI models that generate video qualify as world simulators, advances in generative modeling helped trigger a pivotal shift in world models themselves. Until recently, world models lived entirely inside the system’s architecture — latent spaces only for the agent’s own use. But the breakthroughs in generative AI of recent years have made it possible to build interactive world models — worlds you can actually see and experience. These systems take text prompts (“generate 17th-century London”) or other inputs (a photo of your living room) to generate entire three-dimensional interactive worlds. While video-generating models can depict the world, interactive world models instantiate the world, allowing users or agents to interact with it and affect what happens rather than simply watching things unfold.

Major AI labs are now investing heavily in these interactive world models, with some showing signs of deployment maturity, though approaches vary. Google DeepMind’s Genie series turns text prompts into striking, diverse, interactive digital worlds that continuously evolve in real time — using internal latent representations to predict dynamics and render them into explorable environments, some of which appear real-world-like in both visual fidelity and physical dynamics. Fei-Fei Li’s World Labs recently released Marble, which takes a different approach, letting users transform various inputs into editable and downloadable environments. Runway, a company known for its video generation models, recently launched GWM-1, a world model family that includes explorable environments and robotics, where simulated scenarios can be used to train robot behavior.

Some researchers, however, are skeptical that generating visuals, or pixels, will lead anywhere useful for agent planning. Many believe that world models should predict in compressed, abstract representations without generating pixels — much as we might predict that dropping a cup will cause it to break without mentally rendering every shard of glass.

LeCun, who recently announced his departure from Meta to launch Advanced Machine Intelligence, a company focused on world models, has been critical of approaches that rely on generating pixels for prediction and planning, arguing that they are “doomed to failure.” According to his view, visually reconstructing such complex environments is “intractable” because it tries to model highly unpredictable phenomena, wasting resources on irrelevant details. While researchers debate the optimal path forward, the functional result remains that machines are beginning to learn something about world dynamics from synthetic experience. 

World models are impressive in their own right and offer various applications. In gaming, for instance, interactive world models may soon be used to help generate truly open worlds — environments that uniquely evolve with a player’s choices rather than relying on scripted paths. As someone who grew up immersed in “open world” games of past decades, I relished the thrill of their apparent freedom. Yet even these gaming worlds were always finite, their characters repeating the same lines. Interactive world models bring closer the prospect of worlds that don’t just feel alive but behave as if they are. 

Toward Physical Embodiment

Gaming, however, is merely a steppingstone. The transformative promise of world models lies in physical embodiment and reasoning — AI agents that can navigate our world, rather than just virtual ones. The concept of embodiment is central to cognitive science, which holds that our bodies and sensorimotor capacities shape our cognition. In 1945, French philosopher Maurice Merleau-Ponty observed: “the body is our general medium for having a world.” We are our body, he argued. We don’t have a body. In its AI recasting, embodiment refers to systems situated in physical or digital spaces, using some form of body and perception to interact with both users and their surroundings. 

Physically embodied AI offers endless new deployment possibilities, from wearable companions to robotics. But it runs up against a stubborn barrier — the real world is hard to learn from. The internet flooded machine learning with text, images and video, creating the digital abundance that served as the bedrock for language models and other generative AI systems.

Physical data, however, is different. It is scarce, expensive to capture and constrained by the fact that it must be gathered through real actions unfolding in real time. Training partially capable robots in the real world, and outside of lab settings, might lead to dangerous consequences. To be useful, physical data also needs to be diverse enough to fit the messy particulars of reality. A robot that learns to load plates into a dishwasher in one kitchen learns little about how to handle a saucepan in another. Every environment is different. Every skill must be learned in its own corner of reality, one slow interaction at a time.

World models offer a way through this conundrum. By generating rich, diverse and responsive environments, they create rehearsal space for physically embodied systems — places where robots can learn from the experiences of a thousand lifetimes in a fraction of the time, without ever touching the physical world. This promise is taking its first steps toward reality.

In just the past few years, significant applications of world models in robotics have emerged. Nvidia unveiled a world model platform that helps developers build customized world models for their physical AI setups. Meta’s world models have demonstrated concrete robotics capabilities, guiding robots to perform tasks such as grasping objects and moving them to new locations in environments they were never trained in. Google DeepMind and Runway have shown that world models can serve robotics — whether by testing robot behavior or generating training scenarios. The AI and robotics company 1X grabbed global attention when it released a demo of its humanoid home assistant tidying shelves and outlining its various capabilities, such as suggesting meals based on the contents of a fridge. Though their robot is currently teleoperated with human involvement, its every interaction captures physically embodied data that feeds back into the 1X world model, enabling it to learn from real-world data to improve its accuracy and quality.

But alongside advancements in world models, the other half of this story lies with the AI agents themselves. In a 2025 Nature article, the Dreamer agent demonstrated the ability to collect diamonds in Minecraft without relying on human data or demonstration; instead, it derived its strategy solely from the logic of the environment by repeatedly testing what worked there, as if feeling its way toward competence from first principles. Elsewhere, recent work from Google DeepMind hints at what a new kind of general AI agent might look like. By learning from diverse video games, its language model-based SIMA agent translates language into action in three-dimensional worlds. Tell SIMA to “climb the ladder,” and it complies, performing actions even in games it’s never seen. A new version of this agent has recently shown its ability to self-learn, even in worlds generated by the world model Genie.

In essence, two lines of progress are beginning to meet. On one side, AI agents that learn to navigate and self-improve in any three-dimensional digital environment; on the other, systems that simulate endless, realistic three-dimensional worlds or their abstracted dynamics, with which agents can interact. Together, they may provide the unprecedented capability to run virtually endless simulations in which agents can refine their abilities across variations of experience. If these systems keep advancing, the agents shaped within such synthetic worlds may eventually become capable enough to be embodied in our physical one. In this sense, world models could incubate agents to hone their basic functions before taking their first steps into reality.

As world models move from the research frontier into early production, their concrete deployment pathways remain largely uncertain. Their near-term horizon in gaming is becoming clear, while the longer horizon of broad robotics deployment still requires significant technical breakthroughs in architectures, data, physical machinery and compute. But it is increasingly plausible that an intermediate stage will emerge — world models embedded in wearable devices and ambient AI companions that use spatial intelligence to guide users through their environment. Much like the 1X humanoid assistant guiding residents through their fridge, world-model-powered AI could one day mediate how people perceive, move through and make decisions within their everyday environments.

The Collingridge Dilemma

Whether world models ultimately succeed through pixel-level generation or more abstract prediction, their underlying paradigm shift — from modeling content to modeling dynamics — raises questions that transcend any architecture. Beyond the technological promise of world models, their trajectory carries profound implications for how intelligence may take form and how humans may come to interact with it.

Even if world models never yield human-level intelligence, the shift from systems that model the world through language and media patterns to systems that model it through interactive simulation could fundamentally reshape how we engage with AI and to what end. The societal implications of world modeling capabilities remain largely uncharted as attention from the humanities and social sciences lags behind the pace of computer science progress.

As a researcher in the philosophy of AI — and having spent more than a decade working in AI governance and policy roles inside frontier AI labs and technology companies — I’ve observed a familiar pattern: Clarity about the nature of emerging technologies and their societal implications tends to arrive only in retrospect, a problem known as the “Collingridge dilemma.” This dilemma reminds us that by the time a technology’s consequences become visible, it is often too entrenched to change.

We can begin to address this dilemma by bringing conceptual clarity to emerging technologies early, while their designs can still be shaped. World models present such a case. They are becoming mature enough to analyze meaningfully, yet it’s early enough in their development that such analysis could affect their trajectory. Examining their conceptual foundations now — what these systems represent, how they acquire knowledge, what failure modes they might exhibit — could help inform crucial aspects of their design.

 A Digital Plato’s Cave

The robot in Los Angeles, learning to make sushi in Kyoto, exists in a peculiar state. It knows aspects of the world without ever directly experiencing them. But what is the content of the robot’s knowledge? How is it formed? Under what conditions can we trust its synthetic world view, once it begins to act in ours?

Beginning to answer these questions reveals important aspects about the nature of world models. Designed to capture the logic of the real world, they draw loose inspiration from human cognition. But they also present a deep asymmetry. Humans learn about reality from reality. World models learn primarily from representations of it — such as millions of hours of curated videos, distilled into statistical simulacra of the world. What they acquire is not experience itself, but an approximation of it — a digital Plato’s Cave, offering shadows of the world rather than the world itself.  

Merleau-Ponty’s argument that we are our body is inverted by world models. They offer AI agents knowledge of embodiment without embodiment itself. In a sense, the sushi-making robot is learning through a body it has never inhabited — and the nature of that learning brings new failure modes and risks.

Like other AI systems, world models compress these representations of reality into abstract patterns, a process fraught with loss. As semanticist Alfred Korzybski famously observed, “a map is not the territory.” World models, both those that generate rich visual environments and those that operate in latent spaces, are still abstractions. They learn statistical approximations of physics from video data, not the underlying laws themselves.

But because world models compress dynamics rather than just content, what gets lost is not just information but physical and causal intuition. A simulated environment may appear physically consistent on its face, while omitting important properties — rendering water that flows beautifully but lacks viscosity, or metal that bends without appropriate resistance.

AI systems tend to lose the rare and unusual first, often the very situations where safety matters most. A child darting into traffic, a glass shattering at the pour of boiling tea, the unexpected give of rotting wood. These extreme outliers, though rare in training data, become matters of life and safety in the real world. What may remain in the representation of the world model is an environment smothered into routine, blind to critical exceptions.

With these simplified maps, agents may learn to navigate our world. Their compass, however, is predefined — a reward function that evaluates and shapes their learning. As with other AI reinforcement learning approaches, failing to properly specify a reward evokes Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. A home cleaning agent that’s rewarded for “taking out the trash” no longer becomes appealing to its owner if it places the trash in the garden or brings it back in so that it’s rewarded for taking it out again.

While traditional simulations are encoded with physical principles, those created by world models learn patterns. In their constructed worlds, pedestrians might open umbrellas because sidewalks are wet, never realizing that rain causes both. A soufflé might rise instantly because most cooking videos they’ve learned from skip the waiting time. Through reward hacking — a well-documented problem in reinforcement learning — agents may discover and exploit quirks that only work in their simulated physics. Like speedrunners — gamers who hunt for glitches that let them walk through walls or skip levels — these agents may discover and optimize for shortcuts that fail in reality. 

These are old problems dressed in new clothes that transfer the risks of previous AI systems — brittleness, bias, hallucination — from information to action. All machine learning abstracts from data. But while language models can hallucinate facts and seem coherent, world models may be wrong about physics and still appear visually convincing. Physical embodiment further transforms the stakes. What once misled may now injure. A misunderstood physics pattern becomes a shattered glass; a misread social cue becomes an uncomfortable interaction.

While humans can consider the outputs of chatbots before acting on them, embodied actions by an AI agent may occur without any human to filter or approve such actions — like the Waymo car that struck KitKat, a  beloved neighborhood cat in San Francisco —  an outcome a human driver might have prevented. These issues are compounded by the complex world model and agent stack; its layered components make it hard to trace the source of any failures: Is it the agent’s policy, the world model’s physics or the interaction between them?

Many of these safety concerns manifest as technical optimization challenges similar to those the technical community has faced before, but solving them is also an ethical imperative. Robotics researchers bring years of experience navigating the so-called “sim-to-real” gap — the challenge of translating simulated learning into physical competence. But such existing disciplines may need to adapt to the nature of world models — rather than fine-tuning the dials of hard-coded physics simulations, they must now verify the integrity of systems that have taught themselves how the world works.  As competition intensifies, the need for careful evaluation and robustness work is likely to increase.

Industry deployments recognize these inherent complexities, and leading labs are grounding their world models in real-world data. This enables them to calibrate their models for the environments their physically embodied systems inhabit. Companies like 1X, for example, ground world models in video data continuously collected by their robotics fleet, optimizing for the particularities of physical homes. These environment-specific approaches that still rely on real-world data will likely precede the dream of a general agent, as interactive world models are likely to initially simulate narrow environments and tasks. However, for lighter-stakes embodiments like wearables, the push for generality may arrive sooner.

Beyond these characteristics, world models have distinctive features that raise new considerations. Many of these are sociotechnical — where human design choices carry ethical weight. Unlike language models, world models reason in space and time — simulating what would happen under different actions and guiding behavior accordingly.

Through the dynamics simulated by world models, agents may infer how materials deform under stress or how projectiles behave in the wind. While weaponized robots may seem distant, augmented reality systems that guide users through dangerous actions need not wait for breakthroughs in robotics dexterity. This raises fundamental design questions about world models that carry moral weight: What types of knowledge should we imbue in agents that may be physically embodied, and how can we design world models to prevent self-learning agents from acquiring potentially dangerous knowledge?

Beyond physical reasoning lies the more speculative frontier of modeling social dynamics. Human cognition evolved at least in part as a social simulator — predicting other minds was once as vital as predicting falling objects. While world models are focused on physical dynamics, nothing in principle prevents similar approaches from capturing social dynamics. To a machine learning system, a furrowed brow or a shift in posture is simply a physical pattern that precedes a specific outcome. Were such models to simulate social interactions, they could enable agents to develop intuitions about human behavior — sensing discomfort before it is voiced, reacting to micro-expressions or adjusting tone based on feedback.

Some researchers have begun exploring adjacent territory under the label “mental world models,” suggesting that embodied AI could benefit from having a mental model of human relationships and user emotions. Such capabilities could make AI companions more responsive but also more persuasive — raising concerns about AI manipulation and questions about which social norms these systems might amplify.

These implications compound at scale. Widely deploying world models shifts our focus from individual-level considerations to societal-level ones. Reliable predictive capabilities may accelerate our existing tendency to outsource decisions to machines, introducing implications for human autonomy. Useful systems embedded in wearable companions could gather unprecedented streams of spatial and behavioral data, creating significant new privacy and security considerations. The expected advancement in robotics capabilities might also impact physical labor markets. 

World models suggest a future where our engagement with the world is increasingly mediated by the synthetic logic of machines. One where the map no longer just describes our world but begins to shape it.

Building Human Worlds

These challenges are profound, but they are not inevitable. The science of world models remains in relative infancy, with a long horizon expected before it matures into wide deployment. Thoughtful engagement with the world model paradigm now will shape not just how such future agents learn, but what values their actions represent and how they might interact with people. An overly precautionary approach risks its own moral failure. Just as the printing press democratized knowledge despite enabling propaganda, and cars transformed transportation while producing new perils, world models promise benefits that may far outweigh their risks. The question isn’t whether to build them, but how to design them to best harness their benefits.

This transformative potential of world models extends far beyond the joyful escapism of gaming or the convenience of laundry-folding robots. In transportation, advances in the deployment of autonomous vehicles could improve our overall safety. In medicine, world models could enable surgical robots to rehearse countless variations of a procedure before encountering a single patient, increasing precision and enhancing access to specialized care. Perhaps most fundamentally, they may help humans avoid what roboticists call the “three Ds” — tasks that are dangerous, dirty or dull — relegating them to machines. And if world models deliver on their promise that simulating environments enable richer causal reasoning, they could help revolutionize scientific discovery, the domain many in the field consider the ultimate achievement of AI.

Realizing the promise of such world models, however, requires more than techno-optimism; it needs concrete steps to help scaffold these benefits. The embodiment safety field is already adapting crucial insights from traditional robotics simulations to its world model variants. Other useful precedents can be found in adjacent industries. The autonomous vehicles industry spent years painstakingly developing validation frameworks that verify both simulated and real-world performance. These insights can be leveraged by new industries, as world models could provide opportunities in domains where tolerance for error is narrow — surgical robotics, home assistance, industrial automation — each requiring its own careful calibration of acceptable risk. For regulators, these more mature frameworks offer a concrete starting point and an opportunity for foresight that could enable beneficial deployment.

World models themselves offer unique opportunities for safety research. Researchers like LeCun argue that world model architecture may be more controllable than language models — involving objective-driven agents whose goals can be specified with safety and ethics in mind. Beyond architecture, some world models may serve as digital proving grounds for testing robot behavior before physical deployment.

Google DeepMind recently demonstrated that its Veo video model can predict robot behavior by using its video capabilities to simulate how robots would act in real-world scenarios.  The study showed that such simulations can help discover unsafe behaviors that would be dangerous to test on physical hardware, such as a robot inadvertently closing a laptop on a pair of scissors left on its keyboard. Beyond testing how robots act, world models themselves would need to be audited to ensure they align with the physical world. This presents a challenge that is as much ethical as it is technical: determining which world dynamics are worth modeling and defining what “good enough” means.

Ultimately, early design decisions will dictate the societal outcomes of world model deployment. Choosing what data world models learn from is not just a technical decision, but a socio-technical one, defining the boundaries of what agents may self-learn. The behaviors and physics we accept in gaming environments differ deeply from what we may tolerate in a physical embodiment. The time to ask whether and how we would like to pursue certain capabilities, such as social world modeling, is now.

These deployments also raise broader governance implications. Existing privacy frameworks will likely need to be updated to account for the scale and granularity of spatial and behavioral data that world model-powered systems may harvest. Policymakers, accustomed to analyzing AI through the lens of language processing, must now grapple with systems trained to represent the dynamics of reality. Given that existing AI risk frameworks do not adequately capture the risks posed by such systems, updating these also may soon be required.

The walls of this digital cave are not yet set in stone. Our task is to ensure that the synthetic realities we construct are not just training grounds for efficiency, but incubators for an intelligence that accounts for the social and ethical intricacies of our reality. The design choices we make about what dynamics to simulate and what behaviors to reward will shape the AI agents that emerge in the future. By blending technical rigor with philosophical foresight, we can ensure that when these shadows are projected back into our own world, they do not darken it but illuminate it instead.

The post When AI & Human Worlds Collide appeared first on NOEMA.

Unlike rote, low-level clerical work, cognitive labor of this sort is more likely to be augmented by decision-support information afforded by AI than displaced by intelligent machines.

By contrast, “inexpert” tasks, such as those performed by retirement home orderlies, child-care providers, security guards, janitors or food service workers, will be poorly remunerated even as they remain socially valuable. Since these jobs cannot be automated or enhanced by further knowledge, those who labor in them are a “bottleneck” to improved productivity that would lead to higher wages. Since there will be a vast pool of people without skills who can take those jobs, the value of their labor will be driven down even further.

This is problematic from the perspective of economic disparity because four out of every five jobs created in the U.S. are in this service sector.

So, when looking to the future of the labor market in an AI economy, we can’t talk about “job loss vs. gains” in any general sense. The key issue is not the quantity of jobs, but the value of labor, which really means the value of human expertise and the extent to which AI can enhance it, or not.

I discussed this and other issues with Autor at a recent gathering at the Vatican’s Pontifical Academy in Rome, convened to help address Pope Leo XIV’s concern over the fate of labor in the age of AI. We spoke amid the splendor of the Vatican gardens behind St. Peter’s Basilica.

The populist movements that have risen to power across the West today, particularly in the U.S., did so largely on the coattails of the backlash against globalization. Over the course of the U.S.-led free-trade policies during the post-Cold War decades, the rise of China as a cheap-labor manufacturing power with export access to the markets of advanced economies hollowed out the industrial base across large swaths of America and Europe — and the jobs it provided.

Some worry the AI shock will be even more devastating. Autor sees the similarity and the distinctions. What makes them the same is “it’s a big change that can happen quickly,” he says. But there are three ways in which they are different.

First, “the China trade shock was very localized. It was in manufacturing-intensive communities that made labor-intensive products such as furniture, textiles, clothing, plastic dolls and assembly of low-end hardware.”

AI’s effects will be much more geographically diffuse. “We’ve already lost millions of clerical worker jobs, but no one talks about ‘clerical shock.’ There is no clerical capital of America to see it disappear.”

Second, “the China trade shock didn’t just eliminate certain types of jobs. It eliminated entire industries all at once.” AI will shift the nature of jobs and tasks and change the way people work, but it “will not put industries out of business. … It will open new things and will close others, but it will not be an existential elimination, a great extinction.”

Third, “unless you were a very big multinational, what was experienced by U.S. firms during globalization was basically a shock to competition. All of a sudden, prices fell to a lower level than you could afford to produce.”

AI will be more of a productivity change that will be positive for many businesses. “That doesn’t mean it’s good for workers necessarily, because a lot of workers could be displaced. But business won’t be like, ‘Oh God, the AI shock. We hate this.’ They’ll be, like, ‘Oh great. We can do our stuff with fewer inputs.” In short, tech-driven productivity is the route to great profitability.

As we have often discussed in Noema, it is precisely this dynamic where productivity growth and wealth creation are being divorced from jobs and income that is the central social challenge. Increasingly, the gains will flow to capital — those who own the robots — and decreasingly to labor. The gap will inexorably grow, even with those who can earn higher wages and salaries through work augmented by AI.

Is the idea of “universal basic capital” (UBC), in which everyone has an ownership share in the AI economy through investment of their savings, a promising response?

Autor believes that what UBC offers is a “hedge” against the displacement or demotion of labor. Most of us are “unhedged,” he says, because “human capital is all we have and we are out of luck if that becomes devalued. So at least we would have a balanced portfolio.”

If the government seeds a UBC account, such as “baby bonds,” at the outset, Autor notes, it will grow in value over time through compounded investment returns. The problem with the alternative idea of “universal basic income” is that you are “creating a continual system of transfers where you are basically saying ‘Hey, you rich people over there, you pay for the leisure of everybody else over here.’ And that is not politically viable. ‘How do they get the right to our stuff?’”

Autor compares the idea of “universal basic income” (UBI) to the “resource curse” of unstable countries with vast oil and mineral resources, where it appears that “money is just coming out of a hole in the ground.”

The related reason that UBC is important for Autor is that “the people who have a voice in democracies are those who are seen as economic contributors. If the ownership of capital is more diffuse, then everyone is a contributor,” and everyone has a greater voice, which they will use since they have a stake in the system.

The closer we get to widespread integration of AI into the broader economy, the clearer the patterns Autor describes will become. On that basis, responsible policymakers can formulate remedial responses that fit the new economic times we have entered, rather than relying on outmoded policies geared to conditions that no longer exist.

The post How The ‘AI Job Shock’ Will Differ From The ‘China Trade Shock’ appeared first on NOEMA.

The rise of artificial intelligence (AI) is another breaking wave — potentially a tsunami. The AI systems we have around us are arguably already intelligent, at least in some ways. They will surely get smarter still. But are they, or could they ever be, conscious? And why would that matter?

The cultural history of synthetic consciousness is both long and mostly unhappy. From Yossele the Golem, to Mary Shelley’s “Frankenstein,” HAL 9000 in “2001: A Space Odyssey,” Ava in “Ex Machina,” and Klara in “Klara and The Sun,” the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well — at least, not for the humans involved. One thing we learn from these stories: If artificial intelligence is on a path toward real consciousness, or even toward systems that persuasively seem to be conscious, there’s plenty at stake — and not just disruption in job markets.

Some people think conscious AI is already here. In a 2022 interview with The Washington Post, Google engineer Blake Lemoine made a startling claim about the AI system he was working on, a chatbot called LaMDA. He claimed it was conscious, that it had feelings, and was, in an important sense, like a real person. Despite a flurry of media coverage, Lemoine wasn’t taken all that seriously. Google dismissed him for violating its confidentiality policies, and the AI bandwagon rolled on.

But the question he raised has not gone away. Firing someone for breaching confidentiality is not the same as firing them for being wrong. As AI technologies continue to improve, questions about machine consciousness are increasingly being raised. David Chalmers, one of the foremost thinkers in this area, has suggested that conscious machines may be possible in the not-too-distant future. Geoffrey Hinton, a true AI pioneer and recent Nobel Prize winner, thinks they exist already. In late 2024, a group of prominent researchers wrote a widely publicized article about the need to take the welfare of AI systems seriously. For many leading experts in AI and neuroscience, the emergence of machine consciousness is a question of when, not if.

How we think about the prospects for conscious AI matters. It matters for the AI systems themselves, since — if they are conscious, whether now or in the future — with consciousness comes moral status, the potential for suffering and, perhaps, rights.

It matters for us too. What we collectively think about consciousness in AI already carries enormous importance, regardless of the reality. If we feel that our AI companions really feel things, our psychological vulnerabilities can be exploited, our ethical priorities distorted, and our minds brutalized — treating conscious-seeming machines as if they lack feelings is a psychologically unhealthy place to be. And if we do endow our AI creations with rights, we may not be able to turn them off, even if they act against our interests.

Perhaps most of all, the way we think about conscious AI matters for how we understand our own human nature and the nature of the conscious experiences that make our lives worth living. If we confuse ourselves too readily with our machine creations, we not only overestimate them, we also underestimate ourselves.

The Temptations Of Conscious AI

Why might we even think that AI could be conscious? After all, computers are very different from biological organisms, and the only things most people currently agree are conscious are made of meat, not metal.

The first reason lies within our own psychological infrastructure. As humans, we know we are conscious and like to think we are intelligent, so we find it natural to assume the two go together. But just because they go together in us doesn’t mean that they go together in general.

Intelligence and consciousness are different things. Intelligence is mainly about doing: solving a crossword puzzle, assembling some furniture, navigating a tricky family situation, walking to the shop — all involve intelligent behavior of some kind. A useful general definition of intelligence is the ability to achieve complex goals by flexible means. There are many other definitions out there, but they all emphasize the functional capacities of a system: the ability to transform inputs into outputs, to get things done.

An artificially intelligent system is measured by its ability to perform intelligent behavior of some kind, though not necessarily in a humanlike form. The concept of artificial general intelligence (AGI), by contrast, explicitly references human intelligence. It is supposed to match or exceed the cognitive competencies of human beings. (There’s also artificial superintelligence, ASI, which happens when AI bootstraps itself beyond our comprehension and control. ASI tends to crop up in the more existentially fraught scenarios for our possible futures.)

Consciousness, in contrast to intelligence, is mostly about being. Half a century ago, the philosopher Thomas Nagel famously offered that “an organism has conscious mental states if and only if there is something it is like to be that organism.” Consciousness is the difference between normal wakefulness and the oblivion of deep general anesthesia. It is the experiential aspect of brain function and especially of perception: the colors, shapes, tastes, emotions, thoughts and more, that give our lives texture and meaning. The blueness of the sky on a clear day. The bitter tang and headrush of your first coffee.

AI systems can reasonably lay claim to intelligence in some form, since they can certainly do things, but it is harder to say whether there is anything-it-is-like-to-be ChatGPT.

The propensity to bundle intelligence and consciousness together can be traced to three baked-in psychological biases.

The first is anthropocentrism. This is the tendency to see things through the lens of being human: to take the human example as definitional, rather than as one example of how different properties might come together.

The second is human exceptionalism: our unfortunate habit of putting the human species at the top of every pile, and sometimes in a different pile altogether (perhaps closer to angels and Gods than to other animals, as in the medieval Scala naturae). And the third is anthropomorphism. This is the tendency to project humanlike qualities onto nonhuman things based on what may be only superficial similarities.

Taken together, these biases explain why it’s hardly surprising that when things exhibit abilities we think of as distinctively human, such as intelligence, we naturally imbue them with other qualities we feel are characteristically or even distinctively human: understanding, mindedness and consciousness, too.

One aspect of intelligent behavior that’s turned out to be particularly effective at making some people think that AI could be conscious is language. This is likely because language is a cornerstone of human exceptionalism. Large Language Models (LLMs) like OpenAI’s ChatGPT or Anthropic’s Claude have been the focus of most of the excitement about artificial consciousness. Nobody, as far as I know, has claimed that DeepMind’s AlphaFold is conscious, even though, under the hood, it is rather similar to an LLM. All these systems run on silicon and involve artificial neural networks and other fancy algorithmic innovations such as transformers. AlphaFold, which predicts protein structure rather than words, just doesn’t pull our psychological strings in the same way.

The language that we ourselves use matters too. Consider how normal it has become to say that LLMs “hallucinate” when they spew falsehoods. Hallucinations in human beings are mainly conscious experiences that have lost their grip on reality (uncontrolled perceptions, one might say). We hallucinate when we hear voices that aren’t there or see a dead relative standing at the foot of the bed. When we say that AI systems “hallucinate,” we implicitly confer on them a capacity for experience. If we must use a human analogy, it would be far better to say that they “confabulate.” In humans, confabulation involves making things up without realizing it. It is primarily about doing, rather than experiencing.

When we identify conscious experience with seemingly human qualities like intelligence and language, we become more likely to see consciousness where it doesn’t exist, and to miss seeing it where it does. We certainly should not just assume that consciousness will come along for the ride as AI gets smarter, and if you hear someone saying that real artificial consciousness will magically emerge at the arbitrary threshold of AGI, that’s a sure sign of human exceptionalism at work.

There are other biases in play, too. There’s the powerful idea that everything in AI is changing exponentially. Whether it’s raw compute as indexed by Moore’s Law, or the new capabilities available with each new iteration of the big tech foundation models, things surely are changing quickly. Exponential growth has the psychologically destabilizing property that what’s ahead seems impossibly steep, and what’s behind seems irrelevantly flat. Crucially, things seem this way wherever you are on the curve — that’s what makes it exponential. Because of this, it’s tempting to feel like we are always on the cusp of a major transition, and what could be more major than the creation of real artificial consciousness? But on an exponential curve, every point is an inflection point.

Finally, there’s the temptation of the techno-rapture. Early in the movie “Ex Machina,”the programmer Caleb says to the inventor Nathan: “If you’ve created a conscious machine — it’s not the history of man, that’s the history of Gods.” If we feel we’re at a techno-historical transition, and we happen to be one of its architects, then the Promethean lure must be hard to resist: the feeling of bringing to humankind that which was once the province of the divine. And with this singularity comes the signature rapture offering of immortality: the promise of escaping our inconveniently decaying biological bodies and living (or at least being) forever, floating off to eternity in a silicon-enabled cloud.

Perhaps this is one reason why pronouncements of imminent machine consciousness seem more common within the technorati than outside of it. (More cynically: fueling the idea that there’s something semi-magical about AI may help share prices stay aloft and justify the sky-high salaries and levels of investment now seen in Silicon Valley. Did someone say “bubble”?)

In his book “More Everything Forever,” Adam Becker describes the tendency to project consciousness into AI as a form of pareidolia — the phenomenon of seeing patterns in things, like a face in a piece of toast or Mother Teresa in a cinnamon bun (Figure 1). This is an apt description. But helping you recognize the power of our pareidolia-inducing psychological biases is just the first step in challenging the mythology of conscious AI. To address the question of whether real artificial consciousness is even possible, we need to dig deeper.

Consciousness & Computation

The very idea of conscious AI rests on the assumption that consciousness is a matter of computation. More specifically, that implementing the right kind of computation, or information processing, is sufficient for consciousness to arise. This assumption, which philosophers call computational functionalism, is so deeply ingrained that it can be difficult to recognize it as an assumption at all. But that is what it is. And if it’s wrong, as I think it may be, then real artificial consciousness is fully off the table, at least for the kinds of AI we’re familiar with.

Challenging computational functionalism means diving into some deep waters about what computation means and what it means to say that a physical system, like a computer or a brain, computes at all. I’ll summarize four related arguments that undermine the idea that computation, at least of the sort implemented in standard digital computers, is sufficient for consciousness.

1: Brains Are Not Computers

First, and most important, brains are not computers. The metaphor of the brain as a carbon-based computer has been hugely influential and has immediate appeal: mind as software, brain as hardware. It has also been extremely productive, leading to many insights into brain function and to the vast majority of today’s AI. To understand the power and influence of this metaphor, and to grasp its limitations, we need to revisit some pioneers of computer science and neurobiology.

Alan Turing towers above everyone else in this story. Back in the 1950s, he seeded the idea that machines might be intelligent, and more than a decade earlier, he

formulated a definition of computation that has remained fundamental to our technologies, and to most people’s understanding of what computers are, ever since.

Turing’s definition of computation is extremely powerful and highly (though, as we’ll see, not completely) general. It is based on the abstract concept of a Turing machine: a simple device that reads and writes symbols on an infinite tape according to a set of rules. Turing machines formalize the idea of an algorithm: a mapping, via a sequence of steps, from an input (a string of symbols) to an output (another such string); a mathematical recipe, if you like. Turing’s critical contribution was to define what became known as a universal Turing machine: another abstract device, but this time capable of simulating any specific Turing machine — any algorithm — by taking the description of the target machine as part of its input. This general-purpose capability is one reason why Turing computation is so powerful and so prevalent. The laptop computer I’m writing with, as well as the machines in the server farms running whatever latest AI model, are all physical, concrete examples of (or approximations to) universal Turing machines, bounded by physical limitations such as time and memory.

Another major advantage of this framework, from a practical engineering point of view, is the clean separation it licenses between abstract computation (software) and physical implementation (hardware). An algorithm (in the sense described above) should do the same thing, no matter what computer it is running on. Turing computation is, in principle, substrate independent: it does not depend on any particular material basis. In practice, it’s better described as substrate flexible, since you can’t make a viable computer out of any arbitrary material — cheese, for instance, isn’t up to the job. This substrate-flexibility makes Turing computation extremely useful in the real world, which is why computers exist in our phones rather than merely in our minds.

At around the same time that Turing was making his mark, the mathematician Walter Pitts and neurophysiologist Warren McCulloch showed, in a landmark paper, that networks of highly simplified abstract neurons can perform logical operations (Figure 2). Later work, by the logician Stephen Kleene among others, demonstrated that artificial neural networks like these, when provided with a tape-like memory (as in the Turing machine),  were “Turing complete” — that they could, in principle, implement any Turing machine, any algorithm.

Put these ideas together, and we have a mathematical marriage of convenience and influence, and the kind of beauty that accompanies simplicity. On the one hand, we can ignore the messy neurobiological reality of real brains and treat them as simplified networks of abstract neurons, each of which just sums up its inputs and produces an output. On the other hand, when we do this, we get everything that Turing computation has to offer — which is a lot.

The fruits of this marriage are most evident in its children: the artificial neural networks powering today’s AI. These are direct descendants of McCulloch, Pitts and Kleene, and they also implement algorithms in the substrate-flexible Turing sense. It is hardly surprising that the seductive impressiveness of the current wave of AI reinforces the idea that brains are nothing more than carbon-based versions of neural network algorithms.

But here’s where the trouble starts. Inside a brain, there’s no sharp separation between “mindware” and “wetware” as there is between software and hardware in a computer. The more you delve into the intricacies of the biological brain, the more you realize how rich and dynamic it is, compared to the dead sand of silicon.

Brain activity patterns evolve across multiple scales of space and time, ranging from large-scale cortical territories down to the fine-grained details of neurotransmitters and neural circuits, all deeply interwoven with a molecular storm of metabolic activity. Even a single neuron is a spectacularly complicated biological machine, busy maintaining its own integrity and regenerating the conditions and material basis for its own continued existence. (This process is called autopoiesis, from the Greek for “self-production.” Autopoiesis is arguably a defining and distinctive characteristic of living systems.)

Unlike computers, even computers running neural network algorithms, brains are the kinds of things for which it is difficult, and likely impossible, to separate what they do from what they are.

Nor is there any good reason to expect such a clean separation. The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles. Biological evolution operates under different constraints and with different goals. From the perspective of evolution, there’s no obvious selection pressure for the kind of full separation that would allow the perfect interoperability between different brains as we enjoy between different computers. In fact, the opposite is likely true: Maintaining a sharp software/hardware division is energetically expensive, as is all too apparent these days in the vast energy budgets of modern server farms.

This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware. Evidence that the materiality of the brain matters for its function is evidence against the idea that digital computation is all that counts, which in turn is evidence against computational functionalism.

Another consequence of the deep multiscale integration of real brains — a property that philosophers sometimes call “generative entrenchment” — is that you cannot assume it is possible to replace a single biological neuron with a silicon equivalent, while leaving its function, its input-output behavior, perfectly preserved.

For example, the neuroscientists Chaitanya Chintaluri and Tim Vogels found that some neurons fire spikes of activity apparently to clear waste products created by metabolism. Coming up with a perfect silicon replacement for these neurons would require inventing a whole new silicon-based metabolism, too, which just isn’t the kind of thing silicon is suitable for. The only way to seamlessly replace a biological neuron is with another biological neuron — and ideally, the same one.

This reveals the weakness of the popular “neural replacement” thought experiment, most commonly associated with Chalmers, which invites us to imagine progressively replacing brain parts with silicon equivalents that function in exactly the same way as their biological counterparts. The supposed conclusion is that properties like cognition and consciousness must be substrate independent (or at least silicon-substrate-flexible). This thought experiment has become a prominent trope in discussions of artificial consciousness, usually invoked to support its possibility. Hinton recently appealed to it in just this way, in an interview where he claimed that conscious AI was already with us. But the argument fails at its first hurdle, given the impossibility of replacing any part of the brain with a perfect silicon equivalent.

There is one more consequence of a deeply scale-integrated brain that is worth mentioning. Digital computers and brains differ fundamentally in how they relate to time. In Turing-world, only sequence matters: A to B, 0 to 1. There could be a microsecond or a million years between any state transition, and it would still be the same algorithm, the same computation.

By contrast, for brains and for biological systems in general, time is physical, continuous and inescapable. Living systems must continuously resist the decay and disorder that lies along the trajectory to entropic sameness mandated by the inviolable second law of thermodynamics. This means that neurobiological activity is anchored in continuous time in ways that algorithms, by design, are not. (This is another reason why digital computation is so energetically expensive. Computation exists out of time, but computers do not. Making sure that 1s stay as 1s and 0s stay as 0s takes a lot of energy, because not even silicon can escape the tendrils of entropy.)

What’s more, many researchers — especially those in the phenomenological tradition — have long emphasized that conscious experience itself is richly dynamic and inherently temporal. It does not stutter from one state to another; it flows. Abstracting the brain into the arid sequence space of algorithms does justice neither to our biology nor to the phenomenology of the stream of consciousness.

Metaphors are, in the end, just metaphors, and — as the philosopher Alfred North Whitehead pointed out long ago  — it’s always dangerous to confuse a metaphor with the thing itself. Looking at the brain through “Turing glasses” underestimates its biological richness and overestimates the substrate flexibility of what it does. When we see the brain for what it really is, the notion that all its multiscale biological activity is simply implementation infrastructure for some abstract algorithmic acrobatics seems rather naı̈ve. The brain is not a Turing machine made of meat.

2: Other Games In Town

In the previous section, I noted that Turing computation is powerful but limited. Turing computations — algorithms — map one finite range of discrete numbers (more generally, a string of symbols) onto another, with only the sequence mattering. Turing algorithms are powerful, but there are many kinds of dynamics, many other kinds of functions, that go beyond this kind of computation. Turing himself identified various non-computable functions, such as the famous “halting problem,” which is the problem of determining, in general, whether an algorithm, given some specific input, will ever terminate. What’s more, any function that is continuous (infinitely divisible) or stochastic (involving inherent randomness), strictly speaking, lies beyond Turing’s remit. (Turing computations can approximate or simulate these properties to varying extents, but that’s different from the claim that such functions are Turing computations. I’ll return to this distinction later.)

Biological systems are rife with continuous and stochastic dynamics, and they are deeply embedded in physical time. It seems presumptuous at the very least to assume that only Turing computations matter for consciousness, or indeed for many other aspects of cognition and mind. Electromagnetic fields, the flux of neurotransmitters, and much else besides — all lie beyond the bounds of the algorithmic, and any one of them may turn out to play a critical role in consciousness.

These limitations encourage us to take a broader view of the brain, moving beyond what I sometimes call “Turing world” to consider how broader forms of computation and dynamics might help explain how brains do what they do. There is a rich history here to draw on, and an exciting future too.

The earliest computers were not digital Turing machines but analogue devices operating in continuous time. The ancient “Antikythera mechanism,” used for astronomical purposes and dating back to around 2,000 BCE, is an excellent example. Analogue computers were again prominent at the birth of AI in the 1950s,  in the guise of the long-neglected discipline of cybernetics, where issues of control and regulation of a system are considered more important than abstract symbol manipulation.

Recently, there’s been a resurgence in neuromorphic computation, which leverages more detailed properties of neural systems, such as the precise timing of neuronal spikes, than the cartoon-like simulated neurons that dominate current artificial neural network approaches. And then there’s the relatively new concept of “mortal computation” (introduced by Hinton), which stresses the potential for energy saving offered by developing algorithms that are inseparably tied to their material substrates, so that they (metaphorically) die when their particular implementation ceases to exist.  All these alternative forms of computation are more closely tied to their material basis — are less substrate-flexible — than standard digital computation.

Many systems do what they do without it being reasonable or useful to describe them as being computational at all. Three decades ago, the cognitive scientist Tim van Gelder gave an influential example, in the form of the governor of a steam engine (Figure 3). These governors regulate steam flow through an engine using simple mechanics and physics: as engine speed increases, two heavy cantilevered balls swing outwards, which in turn closes a valve, reducing steam flow. A “computational governor,” sensing engine speed, calculating the necessary actions and then sending precise motor signals to switch actuators on or off, would not only be hopelessly inefficient but would betray a total misunderstanding of what’s really going on.

The branch of cognitive science generally known as “dynamical systems,” as well as approaches that emphasize enactive, embodied, embedded and extended aspects of mind (so-called 4E cognitive science), all reject, in ways relating to van Gelder’s insight, the idea that mind and brain can be exhaustively accounted for algorithmically. They all explore alternatives based on the mathematics of continuous, dynamical processes — involving concepts such as attractors, phase spaces and so on. It is at least plausible that those aspects of brain function necessary for consciousness also depend on non-computational processes like these, or perhaps on some broader notion of computation.

These other games in town are all still compatible with what in philosophy is known as functionalism: the idea that properties of mind (including consciousness) depend on the functional organization of the (embodied) brain. One of the factors contributing to confusion in this area has been a tendency to conflate the rather liberal position of functionalism-in-general, since functional organization can include many things, with the very specific claim of computational functionalism, which implies that the type of organization that matters is computational and which in turn is often assumed to relate to Turing-style algorithms in particular.

The challenge for machine consciousness here is that the further we venture from Turing world, the more deeply entangled we become in randomness, dynamics and entropy, and the more deeply tied we are to the properties of a particular material substrate. The question is no longer about which algorithms give rise to consciousness; it’s about how brain-like a system has to be to move the needle on its potential to be conscious.

3: Life Matters

My third argument is that life (probably) matters. This is the idea — called biological naturalism by the philosopher John Searle— that properties of life are necessary, though not necessarily sufficient, for consciousness. I should say upfront that I don’t have a knock-down argument for this position, nor do I think any such argument yet exists. But it is worth taking seriously, if only for the simple reason mentioned earlier: every candidate for consciousness that most people currently agree on as actually being conscious is also alive.

Why might life matter for consciousness? There’s more to say here than will fit in this essay ( I wrote an entire book, “Being You,” and a recent research paper on the subject), but one way of thinking about it goes like this.

The starting point is the idea that what we consciously perceive depends on the brain’s best guesses about what’s going on in the world, rather than on a direct readout of sensory inputs. This derives from influential predictive processing theories that understand the brain as continually explaining away its sensory inputs by updating predictions about their causes. In this view, sensory signals are interpreted as prediction errors, reporting the difference between what the brain expects and what it gets at each level of its perceptual hierarchies, and the brain is continually minimizing these prediction errors everywhere and all the time.

Conscious experience in this light is a kind of controlled hallucination: a top-down inside-out perceptual inference in which the brain’s predictions about what’s going on are continually calibrated by sensory signals coming from the bottom-up (or outside-in).

This kind of perceptual best-guessing underlies not only experiences of the world, but experiences of being a self, too — experiences of being the subject of experience. A good example is how we perceive the body, both as an object in the world and as the source of more fundamental aspects of selfhood, such as emotion and mood. Both these aspects of selfhood can be understood as forms of perceptual best-guessing: inferences about what is, and what is not, part of the body, and inferences about the body’s internal physiological condition (the latter is sometimes called “interoceptive inference”; interoception refers to perception of the body from within).

Perceptual predictions are good not only for figuring out what’s going on, but (in a call back to mid-20th century cybernetics) also for control and regulation: When you can predict something, you can also control it. This applies above all to predictions about the body’s physiological condition. This is because the primary duty of any brain is to keep its body alive, to keep physiological quantities like heart rate and blood oxygenation where they need to be. This, in turn, helps explain why embodied experiences feel the way they do.

Experiences of emotion and mood, unlike vision (for example), are characterized primarily by valence — by things generally going well or going badly.

This drive to stay alive doesn’t bottom out anywhere in particular. It reaches deep into the interior of each cell, into the molecular furnaces of metabolism. Within these whirls of metabolic activity, the ubiquitous process of prediction error minimization becomes inseparable from the materiality of life itself. A mathematical line can be drawn directly from the self-producing, autopoietic nature of biological material all the way to the Bayesian best-guessing that underpins our perceptual experiences of the world and of the self.

Several lines of thought now converge. First, we have the glimmers of an explanatory connection between life and consciousness. Conscious experiences of emotion, mood and even the basal feeling of being alive all map neatly onto perceptual predictions involved in the control and regulation of bodily condition. Second, the processes underpinning these perceptual predictions are deeply, and perhaps inextricably, rooted in our nature as biological systems, as self-regenerating storms of life resisting the pull of entropic sameness. And third, all of this is non-computational, or at least non-algorithmic. The minimization of prediction error in real brains and real bodies is a continuous dynamical process that is likely inseparable from its material basis, rather than a meat-implemented algorithm existing in a pristine universe of symbol and sequence.

Put all this together, and a picture begins to form: We experience the world around us and ourselves within it — with, through and because of our living bodies. Perhaps it is life, rather than information processing, that breathes fire into the equations of experience.

4: Simulation Is Not Instantiation

Finally, simulation is not instantiation. One of the most powerful capabilities of universal, Turing-based computers is that they can simulate a vast range of phenomena — even, and perhaps especially, phenomena that aren’t themselves (digitally) computational, such as continuous and random processes.

But we should not confuse the map with the territory, or the model with the mechanism. An algorithmic simulation of a continuous process is just that — a simulation, not the process itself.

Computational simulations generally lack the causal powers and intrinsic properties of the things being simulated. A simulation of the digestive system does not actually digest anything. A simulation of a rainstorm does not make anything actually wet. If we simulate a living creature, we have not created life. In general, a computational simulation of X does not bring X into being — does not instantiate X — unless X is a computational process (specifically, an algorithm) itself. Making the point from the other direction, the fact that X can be simulated computationally does not justify the conclusion that X is itself computational.

In most cases, the distinction between simulation and instantiation is obvious and uncontroversial. It should be obvious and uncontroversial for consciousness, too. A computational simulation of the brain (and body), however detailed it may be, will only give rise to consciousness if consciousness is a matter of computation. In other words, the prospect of instantiating consciousness through some kind of whole-brain emulation, at some arbitrarily high level of detail, already assumes that computational functionalism is true. But as I have argued, this assumption is likely wrong and certainly should not be accepted axiomatically.

This brings us back to the poverty of the brain-as-computer metaphor. If you think that everything that matters about brains can be captured by abstract neural networks, then it’s natural to think that simulating the brain on a digital computer will instantiate all its properties, including consciousness, since in this case, everything that matters is, by assumption, algorithmic. This is the “Turing world” view of the brain.

If, instead, you are intrigued by more detailed brain models that capture the complexities of individual neurons and other fine-grained biophysical processes, then it really ought to be less natural to assume that simulating the brain will realize all its properties, since these more detailed models are interesting precisely because they suggest that things other than Turing computation likely matter too.

There is, therefore, something of a contradiction lurking for those who invest their dreams and their venture capital into the prospect of uploading their conscious minds into exquisitely detailed simulations of their brains, so that they can exist forever in silicon rapture. If an exquisitely detailed brain model is needed, then you are no more likely to exist in the simulation than a hailstorm is likely to arise inside the computers of the U.K. meteorological office.

But buckle up. What if everything is a simulation already? What if our whole universe — including the billions of bodies, brains and minds on this planet, as well as its hailstorms and weather forecasting computers — is just an assemblage of code fragments in an advanced computer simulation created by our technologically godlike and genealogically obsessed descendants?

This is the “simulation hypothesis,” associated most closely with the philosopher Nick Bostrom, and still, somehow, an influential idea among the technorati.

Bostrom notes that simulations like this, if they have been created, ought to be much more numerous than the original “base reality,” which in turn suggests that we may be more likely to exist within a simulation than within reality itself. He marshals various statistical arguments to flesh out this idea. But it is telling that he notes one necessary assumption, and then just takes it as a given. This, perhaps unsurprisingly, is the assumption that “a computer running a suitable program would be conscious” (see page 2 of his paper). If this assumption doesn’t hold, then the simple fact that we are conscious would rule out that we exist in a simulation. That this strong assumption is taken on board without examination in a philosophical discussion that is all about the validity of assumptions is yet another indication of how deeply ingrained the computational view of mind and brain has become. It is also a sign of the existential mess we get ourselves into when we fail to distinguish our models of reality from reality itself.

Let’s summarize. Many social and psychological factors, including some well-understood cognitive biases, predispose us to overattribute consciousness to machines.

Computational functionalism — the claim that (algorithmic) computation is sufficient for consciousness — is a very strong assumption that looks increasingly shaky as the many and deep differences between brains and (standard digital) computers come into view. There are plenty of other technologies (e.g., neuromorphic computing, synthetic biology) and frameworks for understanding the brain (e.g., dynamical systems theory), which go beyond the strictly algorithmic. In each case, the further one gets from Turing world, the less plausible it is that the relevant properties can be abstracted away from their underlying material basis.

One possibility, motivated by connecting predictive processing views of perception with physiological regulation and metabolism, is that consciousness is deeply tied to our nature as biological, living creatures.

Finally, simulating the biological mechanisms of consciousness computationally, at whatever grain of detail you might choose, will not give rise to consciousness unless computational functionalism happens anyway to be true.

Each of these lines of argument can stand up by itself. You might favor the arguments against computational functionalism while remaining unpersuaded about the merits of biological naturalism. Distinguishing between simulation and instantiation doesn’t depend on taking account of our cognitive biases. But taken together, they complement and strengthen each other. Questioning computational functionalism reinforces the importance of distinguishing simulation from instantiation. The availability of other technologies and frameworks beyond Turing-style algorithmic computation opens space for the idea that life might be necessary for consciousness.

Collectively, these arguments make the case that consciousness is very unlikely to simply come along for the ride as AI gets smarter, and that achieving it may well be impossible for AI systems in general, at least for the silicon-based digital computers we are familiar with.

At the same time, nothing in what I’ve said rules out the possibility of artificial consciousness altogether.

Given all this, what should we do?

What (Not) To Do?

When it comes to consciousness, the fact of the matter matters. And not only because of the mythology of ancestor simulations, mind-uploading and the like. Things capable of conscious experiences have ethical and moral standing that other things do not. At least, claims to this kind of moral consideration are more straightforward when they are grounded in the capacity for consciousness.

This is why thinking clearly about the prospects for real artificial consciousness is of vital importance in the here and now. I’ve made a case against conscious AI, but I might be wrong. The biological naturalist position (whether my version or any other) remains a minority view. Other theories of consciousness propose accounts framed in terms of standard computation-as-we-know-it. These theories generally avoid proposing sufficient conditions for consciousness. They also generally sidestep defending computational functionalism, being content instead to assume it.

But this doesn’t mean they are wrong. All theories of consciousness are fraught with uncertainty, and anyone who claims to know for sure what it would take to create real artificial consciousness, or for sure what it would take to avoid doing so, is overstepping what can reasonably be said.

This uncertainty lands us in a difficult position. As redundant as it may sound, nobody should be deliberately setting out to create conscious AI, whether in the service of some poorly thought-through techno-rapture, or for any other reason. Creating conscious machines would be an ethical disaster. We would be introducing into the world new moral subjects, and with them the potential for new forms of suffering, at (potentially) an exponential pace. And if we give these systems rights, as arguably we should if they really are conscious, we will hamper our ability to control them, or to shut them down if we need to.

Even if I’m right that standard digital computers aren’t up to the job, other emerging technologies might yet be, whether alternative forms of computation (analogue, neuromorphic, biological and so on) or rapidly developing methods in synthetic biology. For my money, we ought to be more worried about the accidental emergence of consciousness in cerebral organoids (brain-like structures typically grown from human embryonic stem cells) than in any new wave of LLM.

But our worries don’t stop there. When it comes to the impact of AI in society, it is essential to draw a distinction between AI systems that are actually conscious and those that persuasively seem to be conscious but are, in fact, not. While there is inevitable uncertainty about the former, conscious-seeming systems are much, much closer.

As the Google engineer Lemoine demonstrated, for some of us, such conscious-seeming systems are already here. Machines that seem conscious pose serious ethical issues distinct from those posed by actually conscious machines.

For example, we might give AI systems “rights” that they don’t actually need, since they would not actually be conscious, restricting our ability to control them for no good reason. More generally, either we decide to care about conscious-seeming AI, distorting our circles of moral concern, or we decide not to, and risk brutalizing our minds. As Immanuel Kant argued long ago in his lectures on ethics, treating conscious-seeming things as if they lack consciousness is a psychologically unhealthy place to be.  

The dangers of conscious-seeming AI are starting to be noticed by leading figures in AI, including Mustafa Suleyman (CEO of Microsoft AI) and Yoshua Bengio, but this doesn’t mean the problem is in any sense under control.

One overlooked factor here is that even if we know, or believe, that an AI is not conscious, we still might be unable to resist feeling that it is. Illusions of artificial consciousness might be as impenetrable to our minds as some visual illusions. The two lines in the Müller-Lyer illusion (Figure 5) are the same length, but they will always look different. It doesn’t matter how many times you encounter the illusion; you cannot think your way out of it. The way we feel about AI being conscious might be similarly impervious to what we think or understand about AI consciousness.

What’s more, because there’s no consensus over the necessary or sufficient conditions for consciousness, there aren’t any definitive tests for deciding whether an AI is actually conscious. The plot of “Ex Machina” revolves around exactly this dilemma. Riffing on the famous Turing test (which, as Turing well knew, tests for machine intelligence, not consciousness), Nathan — the creator of the robot Ava — says that the “real test” is to reveal that his creation is a machine, and to see whether Caleb — the stooge — still feels that it, or she, is conscious. The “Garland test,” as it’s come to be known, is not a test of machine consciousness itself. It is a test of what it takes for a human to be persuaded that a machine is conscious.

The importance of taking an informed ethical position despite all these uncertainties spotlights another human habit: our unfortunate track record of withholding moral status from those that deserve it, including from many non-human animals, and sometimes other humans. It is reasonable to wonder whether withholding attributions of consciousness to AI may leave us once again on the wrong side of history. The recent calls for attention to “AI welfare” are based largely on this worry.

But there are good reasons why the situation with AI is likely to be different. Our psychological biases are more likely to lead to false positives than false negatives. Compared to non-human animals, the apparent wonders of AI may be more similar to us in ways that do not matter for consciousness, like linguistic ability, and less similar in ways that do, like being alive.

Soul Machine

Despite the hype and the hubris, there’s no doubt that AI is transforming society. It will be hard enough to navigate the clear and obvious challenges AI poses, and to take proper advantage of its many benefits, without the additional confusion generated by immoderate pronouncements about a coming age of conscious machines. Given the pace of change in both the technology itself and in its public perception, developing a clear view of the prospects and pitfalls of conscious AI is both essential and urgent.

Real artificial consciousness would change everything — and very much for the worse. Illusions of conscious AI are dangerous in their own distinctive ways, especially if we are constantly distracted and fascinated by the lure of truly sentient machines. My hope for this essay is that it offers some tools for thinking through these challenges, some defenses against overconfident claims about inevitability or outright impossibility, and some hope for our own human, animal, biological nature. And hope for our future too.

The future history of AI is not yet written. There is no inevitability to the directions AI might yet take. To think otherwise is to be overly constrained by our conceptual inheritance, weighed down by the baggage of bad science fiction and submissive to the self-serving narrative of tech companies laboring to make it to the next financial quarter. Time is short, but collectively we can still decide which kinds of AI we really want and which we really don’t.

The philosopher Shannon Vallor describes AI as a mirror, reflecting back to us the incident light of our digitized past. We see ourselves in our algorithms, but we also see our algorithms in ourselves. This mechanization of the mind is perhaps the most pernicious near-term consequence of the unseemly rush toward human-like AI. If we conflate the richness of biological brains and human experience with the information-processing machinations of deepfake-boosted chatbots, or whatever the latest AI wizardry might be, we do our minds, brains and bodies a grave injustice. If we sell ourselves too cheaply to our machine creations, we overestimate them, and we underestimate ourselves.

Perhaps unexpectedly, this brings me at last to the soul. For many people, especially modern people of science and reason, the idea of the soul might seem as outmoded as the Stone Age. And if by soul what is meant is an immaterial essence of rationality and consciousness, perfectly separable from the body, then this isn’t a terrible take.

But there are other games in town here, too. Long before Descartes, the Greek concept of psychē linked the idea of a soul to breath, while on the other side of the world, the Hindu expression of soul, or Ātman, associated our innermost essence with the ground-state of all experience, unaffected by rational thought or by any other specific conscious content, a pure witnessing awareness.

The cartoon dreams of a silicon rapture, with its tropes of mind uploading, of disembodied eternal existence and of cloud-based reunions with other chosen ones, is a regression to the Cartesian soul. Computers, or more precisely computations, are, after all, immortal, and the sacrament of the algorithm promises a purist rationality, untainted by the body (despite plentiful evidence linking reason to emotion). But these are likely to be empty dreams, delivering not posthuman paradise but silicon oblivion.

What really matters is not this kind of soul. Not any disembodied human-exceptionalist undying essence of you or of me. Perhaps what makes us us harks even further back, to Ancient Greece and to the plains of India, where our innermost essence arises as an inchoate feeling of just being alive — more breath than thought and more meat than machine. The sociologist Sherry Turkle once said that technology can make us forget what we know about life. It’s about time we started to remember.

The post The Mythology Of Conscious AI appeared first on NOEMA.

by Hélène Blanc

for “Why Science Hasn’t Solved Consciousness (Yet)”

by Shalinder Matharu

for “How To Build A Thousand-Year-Old Tree”

by Nicolás Ortega

for “Humanity’s Endgame”

by Seba Cestaro

for “How We Became Captives Of Social Media”

by Beatrice Caciotti

for “A Third Path For AI Beyond The US-China Binary”

by Dadu Shin

for “The Languages Lost To Climate Change” in Noema Magazine Issue VI, Fall 2025

by LIMN

for “Why AI Is A Philosophical Rupture”

by Kate Banazi

for “AI Is Evolving — And Changing Our Understanding Of Intelligence” in Noema Magazine Issue VI, Fall 2025

by Jonathan Zawada

for “The New Planetary Nationalism” in Noema Magazine Issue VI, Fall 2025

by Satwika Kresna

for “The Future Of Space Is More Than Human”

Other Top Picks By Noema’s Editors

The post Noema’s Top Artwork Of 2025 appeared first on NOEMA.

The Last Days Of Social Media

Social media promised connection, but it has delivered exhaustion.

by James O’Sullivan

A Third Path For AI Beyond The US-China Binary

What if the future of AI isn’t defined by Washington or Beijing, but by improvisation elsewhere?

by Dang Nguyen

Why Science Hasn’t Solved Consciousness (Yet)

To understand life, we must stop treating organisms like machines and minds like code.

by Adam Frank

The Unseen Fury Of Solar Storms

Lurking in every space weather forecaster’s mind is the hypothetical big one, a solar storm so huge it could bring our networked, planetary civilization to its knees.

by Henry Wismayer

From Statecraft To Soulcraft

How the world’s illiberal powers like Russia, China and increasingly the U.S. rule through their visions of the good life.

by Alexandre Lefebvre

The Languages Lost To Climate Change

Climate catastrophes and biodiversity loss are endangering languages across the globe.

by Julia Webster Ayuso

The Shrouded, Sinister History Of The Bulldozer

From India to the Amazon to Israel, bulldozers have left a path of destruction that offers a cautionary tale for how technology without safeguards can be misused.

by Joe Zadeh

The Moral Authority Of Animals

For millennia before we showed up on the scene, social animals — those living in societies and cooperating for survival — had been creating cultures imbued with ethics.

by Jay Griffiths

Welcome To The New Warring States

Today’s global turbulence has echoes in Chinese history.

by Hui Huang

Signs Of Life In A Desert Of Death

In the dry and fiery deserts of Central Asia, among the mythical sites of both the first human and the end of all days, I found evidence that life restores itself even on the bleakest edge of ecological apocalypse.

by Nick Hunt

Other Top Picks By Noema’s Editors

Feature Philosophy & Culture

Noema’s Top Artwork Of 2025

Noema editors

Feature Technology & the Human

Noema’s Top 10 Reads Of 2024

Noema editors

Feature Climate Crisis

Noema’s Top 10 Reads Of 2023

Noema editors

The post Noema’s Top 10 Reads Of 2025 appeared first on NOEMA.

The newest incarnation is found in discourse around artificial intelligence. Here, at least, it is acknowledged that humans, with our limited memory and information processing capacity, will never really know everything. Still, this newfound and humbler stance is supplemented with the assumption that we are the single superior biological species who can build the technologies that will.

AlphaFold, an AI system developed by Google DeepMind, represents one of AI’s most celebrated achievements in science. Trained on more than 150,000 experimentally determined protein structures, AlphaFold 3 can now predict the structure of more than 200 million proteins as well as that of other biomolecules. Such scale was previously unimaginable. Earlier mathematical models could predict some features of protein structure, but nothing approaching this magnitude. The optimism is palpable: If AI can solve protein folding at this scale, what else might it accomplish?

Some proclaim AI will solve all disease, make scientists obsolete or even that artificial superintelligences will solve all of science. Yet many consider the protein folding problem unsolved. AlphaFold predicts 3D structures, but it does not explain the underlying physics, folding pathways or dynamic conformational ensembles. It works well for proteins made from the 20 or so amino acids found in terrestrial biology. To study proteins from the hundreds of amino acids in meteoritic materials, or to design novel therapeutic proteins, this model needs additional input. The limitation is not the algorithm or its scaling: The necessary data does not exist.

This tension reveals something profound about what science is, and how science defies precise definition. If we view science purely as the scientific method — observation, hypothesis, testing, analysis — then automation seems inevitable. AI algorithms demonstrably perform many, if not all, of these steps, and are getting better at them when guided by scientists.

But as philosopher Paul Feyerabend argued in “Against Method,” the very idea of a universal scientific method is misconceived. Most scientists invoke the scientific method only when writing for peer review, using it as standardization that allows reproducibility. Historically, scientific methods arise after discoveries are made, not before.

The question is not whether AI can execute steps in a method, but whether science generates knowledge in a way that is fundamentally something more.

If scale was all we needed, current AI would provide a mundane solution for science: We could do more because we have larger scale models. However, optimism around AI is not just about automation and scaling, it is also about theory of mind. Large language models (LLMs) like ChatGPT, Gemini and Claude have reshaped how many see intelligence, because interactions with these algorithms, by virtue of their design, give the appearance of a mind.

Yet as neuroscientist Anil Seth keenly observed, AlphaFold relies on the same underlying Transformer architecture as LLMs, and no one confuses AlphaFold with being a mind. Are we supposed to interpret that such an algorithm, instantiated on silicon chips, will comprehend the world in exactly the way we do, and communicate via our language with us so effectively as to describe the world as we understand it? Or should we instead believe it is maybe easier than we thought, after billions of years of the evolution of intelligence, to encode our own predictive and dynamic representational maps within such short spatial and temporal physical scales?

Consider how your own mind constructs your unique representation of reality. Each of us holds within our skulls a volume that generates an entire inner world. We cannot say this with the same certainty about any other entity, alive or not. Your sensory organs convert physical stimuli into electrical signals. In vision, photoreceptors respond to light and send signals along your optic nerve. Your brain processes this in specialized regions, detecting edges, motion and color contrasts in separate areas, then binds these fragmented perceptions into a unified object of awareness — what is called a percept — which forms your conscious experience of the world.

This is the binding problem: how distributed neural activity creates singular, coherent consciousness. Unlike “the hard problem of consciousness,” an open question behind our intrinsic experience, we do have some scientific insights into how binding could be accomplished: Synchronized neural activity and attention mechanisms coordinate information across brain regions to construct your unique mental model of the world. This model is literally the totality of your conscious understanding of what is real.

Each of us is an inhabitant of one such mental model. What it is like to be inside a physical representation of the world, as we all are within our conscious experience, is nontrivial to explain scientifically (and some argue may not be possible).

Scientific societies face an analogous binding problem. Just as individual minds collect sense data to model the world, societies do the same through what Claire Isabel Webb, director of the Berggruen Institute’s Future Humans program, has called “technologies of perception”: Telescopes reveal cosmic depths, radiometric dating uncovers deep time, microscope expose subatomic structure, and now AI uncovers patterns in massive data.

Danish astronomer Tycho Brahe’s precise astronomical measurements, enabled by mechanical clocks and sophisticated angle-measuring devices, provided sense data that German astronomer Johannes Kepler transformed into mathematical models of elliptical orbits. A society collecting observations across space and time, exemplified across the work of Copernicus, Brahe, Kepler, Galileo and others, came to be bound into a single scientific consensus representation of reality — a societal percept — in the form of a theory that describes what it means to move and to gravitate.

But there is a fundamental difference. Your subjective experience, what philosophers call qualia, is irreducibly private. In a very real sense, it may be the most private information of all that our universe creates, because it is uniquely and intimately tied to the features of your physical existence that cannot be replicated in anything else.

When you see the color red, a specific experience emerges from your neural architecture responding to wavelengths between 620 and 750 nanometers. I can point to something red, and you can acknowledge you are also seeing red, but we cannot transfer the actual experience of redness from your consciousness to mine. We cannot know if we share the same inner experience. All we can share are descriptions.

This is where science radically differs from experience: It is fundamentally intersubjective. If something exists only in one mind and cannot be shared, it cannot become scientific knowledge. Science requires verifying each other’s observations, building on a lineage of past discoveries and developing intergenerational consensus about reality. Scientific models must therefore be expressible in symbols, mathematics and language, because they must be copyable and interpretable between minds.

Science is definitionally unstable because it is not an objective feature of reality; instead, it is more accurately understood as an evolving cultural system, bred of consensus representation and adaptive to the new knowledge we generate.

When Sir Isaac Newton defined F = ma, he was not sharing his inner experience of force or acceleration. He created a symbolic representation of relationships between three core abstractions — force, mass, acceleration — each developed through metrological standardization. The formula became pervasive cultural knowledge because any mind or machine can interpret and apply it, regardless of how each experiences these concepts internally.

This reveals the most fundamental challenge of scientific knowledge: Our primary interface for sharing scientific ideas is symbolic representation. What we communicate are models of the world, not the world itself. Philosopher of science Nancy Cartwright argues scientific theories are simulacra; that is, they are useful fictions in mathematical and conceptual form that help us organize, predict and manipulate phenomena. Theories are cultural technologies.

When we use the ideal gas law (PV = nRT), we model gases as non-interacting points. This is not to be interpreted as a claim that real gases are literally points with no volume that never interact, it is merely a simplification that works well enough in many cases. These simplified models matter because they are comprehensible and shareable between minds, and they are copyable between our calculating machines.

The requirement that scientific knowledge must be shareable forces us to create simulacra at every descriptive level. Science’s intersubjective nature places strict physical constraints on what theories can be. Our scientific models must be expressible symbolically and interpretable between human minds. They are therefore necessarily abstractions that never capture reality’s full structure. They can never fully capture reality, because no human mind has sufficient information processing and memory to encode the entire external world. Even societies have limits.

AI will also have limits.

These limits are not solely in terms of available compute power, made acute in the need for more data processing infrastructure to support the AI economy. More fundamentally, the current optimistic, and sometimes hubristic, dialogue around AI and artificial general intelligence (AGI) suggests these algorithms will be “more than human” in their ability to understand and explain the world, breaking what some perceive as limits on intelligence imposed by human biology.

But this cannot be true by virtue of the very foundations of the theory of computation, and the lineages of human abstraction from which these technologies directly descend. As physicist David Deutsch writes, if the universe is indeed explicable, humans are already “universal explainers” because we are capable of understanding anything any computational system can: In terms of computational repertoire, both computers and brains are equivalently universal.

Other foundational theorems in computer science, like the no free lunch theorems by physicists David Wolpert and William Macready, indicate that when performance is averaged over all possible problems, no optimization algorithm (machine learning algorithms included) is universally better than any other. Stated another way, making an algorithm such that it performs exceptionally well for one class of problems will lead to trade-offs where it is poorer than average at others.

The physical world does not contain all possible problems, but the structure of the ones it does contain changes with biological and technological evolution. Just as no individual can comprehend everything all humans know, or will know, there can be no algorithm (AGI or otherwise) that is indefinitely better than all others.

More fundamentally, the possibility of universal computation arises due to a fundamental limitation; universal computers can only describe computable things, but never the uncomputable ones — a limitation intrinsic to any computer we build. This limitation does not apply to individual human minds, only what we share via language, and this is key to how we generate new social knowledge.

Scientific revolutions occur when our shared representational maps break down; that is, when existing concepts prove inadequate to cover phenomena we newly encounter or old ones we wish to explain. We must then invent new semantic representations capturing regularities old frameworks could not. At these times, nonconformism plays an outsized role in knowledge creation.

Consider the shift from natural theology to evolution. The old paradigm assumed organisms were designed by a creator, species were fixed, Earth was young. As we learned to read deeper histories, through carbon dating, phylogeny and observing species change through selective breeding and extinction, we never witnessed the spontaneous formation of biological forms.

Deeper historical memory forces new descriptions to emerge. Evolution and geology revealed concepts of deep time, astronomy introduced concepts of deep space, and now, as historian Thomas Moynihan points out, we are entering an age revealing a universe deep in possibility. Our world does not suddenly change or get older, but our understanding does. We repeatedly find ourselves developing radically new words and concepts to reflect new meaning as we discover it in the world.

Philosopher of science Thomas Kuhn recognized these transitions as paradigm shifts, noting how abrupt periods of change force scientists to reconceptualize the way we see our field, what questions we ask, what methods we use, what we consider legitimate knowledge. What emerges are entirely new representations for describing the world, often including totally new descriptions of everyday objects we thought we understood.

Science, as Kuhn saw it, is messy, social and profoundly human. In an age where we are now worried about alignment, after alignment and re-alignment with our own technological creations, paradigm shifts might best be described as the representational alignment of our societal percepts, where we must find new ways for our representations to keep in sync with the changing structure of reality as presented to us across millennia of our cultural evolution.

Paradigm shifts reveal how the power of scientific thought does not lie in the literal truth of theories, but in our ability to identify new ways of describing the world and in how the structures we describe persist across different representational schemes. The culture of science helps distinguish between simulacra that approach causal mechanisms (sometimes called objective reality) and those that lead us astray. Crucially, discovering new features of reality requires building new descriptions. When frameworks fail to capture important worldly features, for example when we recognize patterns but cannot articulate them, new frameworks and representational maps must emerge.

Albert Einstein’s development of general relativity illustrates this. Seven years separated his realization that physics needed to transcend the linear Lorentz transformations (appearing in special relativity) to get to the general theory of relativity. In his own reflections, he comments on the reason being how “it is not so easy to free oneself from the idea that coordinates must have an immediate metrical meaning.” Mathematical structures imposed as models weren’t capturing meaning: They were missing features Einstein intuited must exist. Once he encoded his intuition, it became intersubjective and shareable between minds.

This brings us to why AI cannot replace human scientists. Controversy and debate over language and representation in science are not bugs; they are features of a societal system determining which models it wants. Stakes are high because our descriptive languages literally structure how we experience and interact with the world, forming the reality our descendants inherit.

AI will undoubtedly play a prominent role in “normal science,” something Kuhn defined as constituting the technical refinement of existing paradigms. Our world is growing increasingly complex, demanding correspondingly complex models. Scale is not all we need, but it will certainly help.

AlphaFold 3’s billions of parameters suggest parsimony and simplicity may not be science’s only path. If we want models mapping the world as tightly as possible, complexity may be necessary. This aligns with logical positivists Otto Neurath, Rudolph Carnap and the Vienna Circle’s view: “In science there are no ‘depths’; there is surface everywhere.” If we have accurate, predictive models of everything, maybe there are no deeper truths to be uncovered. 

This surface view misses a profound feature of scientific knowledge creation. The simulacra change, but underlying patterns we uncover by manipulating symbols remain, inarticulable and persistent, independent of our languages. The concept of gravity was unknown to our species before science, despite direct sensorial contact throughout human history and an inherited memory from the nearly 4-billion-year lineage of life that preceded us. Every species is aware of gravity, and some microorganisms even use this awareness to navigate. We knew it as a regularity before Newton’s mathematical description, and this knowledge persisted through Einstein’s radical reconceptualization.

Prior to Newton’s generation, the model of Ptolemy was the most widely adopted for the study of planetary motions, as it had been for nearly 1,500 years. It included circular orbits for the planets, and to increase predictive power, epicycles were added for each planet, such that each planet in the model moved in a small circle while also moving in a larger circle around the Earth. Additional epicycles were added to increase predictive accuracy, not unlike adding nodes to a machine learning model with the accompanying risk of over-fitting.

We did not transition to the Newtonian model for its predictive power, but rather because it explained more. The modern concept of gravity was invented by this process of abstraction, and by the explanatory unification of our terrestrial experience of gravity with our celestial observations of it. It is likely that our species, and more specifically our species’ societies, will never forget gravity now that we have learned an abstraction to describe it, even as our symbols describing it may radically change.

It is this depth of meaning, inherent in our theories, that science discovers in the process of constructing new societal percepts. This cannot be captured by the surface level view, where science merely creates predictive maps, devoid of depth and meaning.

French literary critic Roland Barthes argued in his liberating 1967 essay “The Death of the Author” that texts contain multiple layers and meanings beyond their creators’ intentions. As with Feyerabend, this was a direct rebuttal “against method.” For Barthes, this rebuttal of method was in refute of literary criticism’s traditional methodological practice of relying on the identity of an author to interpret an ultimate meaning or truth for a text. Instead, Barthes argued for abandoning the idea of a definitive authorial meaning in favor of a more socially constructed and evolving one.

Similarly, it can be said the scientist “dies” in our writings. When we publish, we submit work to our peers’ interpretation, criticism and use. The peer review process is currently a target for AI automation, born from a misconception that peer review is strictly about fact-checking. In reality, peer review is about debate and discussion among peers and gives scholars an opportunity to cocreate how new scientific work is presented in the literature.  That debate and cocreation are essential to the cultural system of science. It is only after peer review that we enter a method that allows reproducibility. Scientific ideas are not born solely of individual minds, but also of consensus interpretations of what those minds create.

The outputs of AI models arrive already “dead” in this crucial sense: They are produced without an embodied creative act of meaning-making that accompanies the modes of scientific discovery we have become accustomed to in the last 400 or so years. When a scientist develops a theory, even before peer review, there is an intentional act of explanation, and an internal act of wrestling with intuition and its representation. AI models, by contrast, generate predictions through statistical pattern recognition, a very different process.

Science and AI are cultural technologies; both are systems societies use to organize knowledge. When considering the role of AI in science, we should not be comparing individual AI models to individual human scientists, or their minds, as these are incomparable.

Rather, we must ask how the cultural systems of AI technologies and science will interact. The death of the scientist is the loss of the inner world that creates an idea, but this is also when the idea can become shared, and the inner world of the societal system of debate and controversy comes alive. When human scientists die in their published work, they birth the possibility of shared understanding. Paradigm shifts are when this leads to entirely new ways for societies to understand the world, forcing us to collectively see new structure underneath our representational maps, structure we previously could not recognize was there. 

An AI model can integrate an unprecedented number of observations. It can execute hypothesis testing, identify patterns in massive datasets and make predictions at scales an individual human cannot match. But current AI operates only within the representational schema humans give it, refining and extending them at scale. The creative act of recognizing that our maps are inadequate and building entirely new, social and symbolic frameworks to describe what was previously indescribable remains exceptionally challenging, impossible to reduce to method, and so far, uniquely human.

It is unclear how AI might participate in the intersubjective process of building scientific consensus. No one can yet foretell the role AI will play in a collective determination of which descriptions of reality a society will adopt, which new symbolic frameworks will replace those that have died, and which patterns matter enough to warrant new languages for their articulation.

The deeper question is not whether AI can do science, but whether societies can build shared representations and consensus meanings with algorithms that lack the intentional meaning creation that has always been at the heart of scientific explanation.

In essence, science itself is evolving, begging the question of what science after science will look like in an age where the cultural institution of science becomes radically transformed.  We should be asking: When we find our species still craves meaning and understanding, beyond algorithmic instantiation, what will science become?

Will AI transform science? Certainly. Will it replace scientists? Certainly not. If we misunderstand what science is, mistaking automation of method for the human project of collectively constructing, debating and refining the symbolic representations through which we make sense of reality, AI may foretell the death of science: We will miss the true opportunity to integrate AI into the culture systems of science.

Science is not merely about prediction and automation; history tells us it is much more. It is about explanatory consensus, and an ongoing human negotiation of which descriptions of the world we will collectively adopt. That negotiation, the intersubjective binding of observations into shared meaning is irreducibly social and, for now, irreducibly human.

The post The Death Of The Scientist appeared first on NOEMA.

The scenario echoes the speculative lineage of science fiction, from Isaac Asimov’s “Three Laws of Robotics” — a literary attempt to constrain machine agency — to later visions such as Stanley Kubrick and Arthur C. Clarke’s HAL 9000 or the runaway networks of William Gibson. What was once the realm of narrative thought-experiment now serves as a quasi-political forecast.

This narrative has very little to do with any scientific consensus, emerging instead from particular corridors of power. The loudest prophets of superintelligence are those building the very systems they warn against. When Sam Altman speaks of artificial general intelligence’s existential risk to humanity while simultaneously racing to create it, or when Elon Musk warns of an AI apocalypse while founding companies to accelerate its development, we’re seeing politics masked as predictions.

The superintelligence discourse functions as a sophisticated apparatus of power, transforming immediate questions about corporate accountability, worker displacement, algorithmic bias and democratic governance into abstract philosophical puzzles about consciousness and control. This sleight of hand is neither accidental nor benign. By making hypothetic catastrophe the center of public discourse, architects of AI systems have positioned themselves as humanity’s reluctant guardians, burdened with terrible knowledge and awesome responsibility. They have become indispensable intermediaries between civilization and its potential destroyer, a role that, coincidentally, requires massive capital investment, minimal regulation and concentrated decision-making authority.

Consider how this framing operates. When we debate whether a future artificial general intelligence might eliminate humanity, we’re not discussing the Amazon warehouse worker whose movements are dictated by algorithmic surveillance or the Palestinian whose neighborhood is targeted by automated weapons systems. These present realities dissolve into background noise against the rhetoric of existential risk. Such suffering is actual, while the superintelligence remains theoretical, but our attention and resources — and even our regulatory frameworks — increasingly orient toward the latter as governments convene frontier-AI taskforces and draft risk templates for hypothetical future systems. Meanwhile, current labour protections and constraints on algorithmic surveillance remain tied to legislation that is increasingly inadequate.

In the U.S., Executive Order 14110 on the “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence” mentions civil rights, competition, labor and discrimination, but it creates its most forceful accountability obligations for large, high-capability foundation models and future systems trained above certain compute thresholds, requiring firms to share technical information with the federal government and demonstrate that their models stay within specified safety limits. The U.K. has gone further still, building a Frontier AI Taskforce — now absorbed into the AI Security Institute — whose mandate centers on extreme, hypothetical risks. And even the EU’s AI Act, which does attempt to regulate present harms, devotes a section to systemic and foundation-model risks anticipated at some unknown point in the future. Across these jurisdictions, the political energy clusters around future, speculative systems.

Artificial superintelligence narratives perform very intentional political work, drawing attention from present systems of control toward distant catastrophe, shifting debate from material power to imagined futures. Predictions of machine godhood reshape how authority is claimed and whose interests steer AI governance, muting the voices of those who suffer under algorithms and amplifying those who want extinction to dominate the conversation. What poses as neutral futurism functions instead as an intervention in today’s political economy. Seen clearly, the prophecy of superintelligence is less a warning about machines than a strategy for power, and that strategy needs to be recognized for what it is. The power of this narrative draws from its history.

Bowing At The Altar Of Rationalism

Superintelligence as a dominant AI narrative predates ChatGPT and can be traced back to the peculiar marriage of Cold War strategy and computational theory that emerged in the 1950s. The RAND Corporation, an archetypal think tank where nuclear strategists gamed out humanity’s destruction, provided the conceptual nursery for thinking about intelligence as pure calculation, divorced from culture or politics.

The early AI pioneers inherited this framework, and when Alan Turing proposed his famous test, he deliberately sidestepped questions of consciousness or experience in favor of observable behavior — if a machine could convince a human interlocutor of its humanity through text alone, it deserved the label “intelligent.” This behaviorist reduction would prove fateful, as in treating thought as quantifiable operations, it recast intelligence as something that could be measured, ranked and ultimately outdone by machines.

The computer scientist John von Neumann, as recalled by mathematician Stanislaw Ulam in 1958, spoke of a technological “singularity” in which accelerating progress would one day mean that machines could improve their own design, rapidly bootstrapping themselves to superhuman capability. This notion, refined by mathematician Irving John Good in the 1960s, established the basic grammar of superintelligence discourse: recursive self-improvement, exponential growth and the last invention humanity would ever need to make. These were, of course, mathematical extrapolations rather than empirical observations, but such speculations and thought experiments were repeated so frequently that they acquired the weight of prophecy, helping to make the imagined future they described look self-evident.

The 1980s and 1990s saw these ideas migrate from computer science departments to a peculiar subculture of rationalists and futurists centered around figures like computer scientist Eliezer Yudkowsky and his Singularity Institute (later the Machine Intelligence Research Institute). This community built a dense theoretical framework for superintelligence: utility functions, the formal goal systems meant to govern an AI’s choices; the paperclip maximizer, a thought experiment where a trivial objective drives a machine to consume all resources; instrumental convergence, the claim that almost any ultimate goal leads an AI to seek power and resources; and the orthogonality thesis, which holds that intelligence and moral values are independent. They created a scholastic philosophy for an entity that didn’t exist, complete with careful taxonomies of different types of AI take-off scenarios and elaborate arguments about acausal trade between possible future intelligences.

What united these thinkers was a shared commitment to a particular style of reasoning. They practiced what might be called extreme rationalism, the belief that pure logic, divorced from empirical constraint or social context, could reveal fundamental truths about technology and society. This methodology privileged thought experiments over data and clever paradoxes over mundane observation, and the result was a body of work that read like medieval theology, brilliant and intricate, but utterly disconnected from the actual development of AI systems. It should be acknowledged that disconnection did not make their efforts worthless, and by pushing abstract reasoning to its limits, they clarified questions of control, ethics and long-term risk that later informed more grounded discussions of AI policy and safety.

The contemporary incarnation of this tradition found its most influential expression in Nick Bostrom’s 2014 book “Superintelligence,” which transformed fringe internet philosophy into mainstream discourse. Bostrom, a former Oxford philosopher, gave academic respectability to scenarios that had previously lived in science fiction and posts on blogs with obscure titles. His book, despite containing no technical AI research and precious little engagement with actual machine learning, became required reading in Silicon Valley, often cited by tech billionaires. Musk once tweeted: “Worth reading Superintelligence by Bostrom. We need to be super careful with AI. Potentially more dangerous than nukes.” Musk is right to counsel caution, as evidenced by the 1,200 to 2,000 tons of nitrogen oxides and hazardous air pollutants like formaldehyde that his own artificial intelligence company expels into the air in Boxtown, a working-class, largely Black community in Memphis.

This commentary shouldn’t be seen as an attempt to diminish Bostrom’s achievement, which was to take the sprawling, often incoherent fears about AI and organize them into a rigorous framework. But his book sometimes reads like a natural history project, in which he categorizes different routes to superintelligence and different “failure modes,” ways such a system might go wrong or destroy us, as well as solutions to “control problems,” schemes proposed to keep it aligned — this taxonomic approach made even wild speculation appear scientific. By treating superintelligence as an object of systematic study rather than a science fiction premise, Bostrom laundered existential risk into respectable discourse.

The effective altruism (EA) movement supplied the social infrastructure for these ideas. Its core principle is to maximize long-term good through rational calculation. Within that worldview, superintelligence risk fits neatly, for if future people matter as much as present ones, and if a small chance of global catastrophe outweighs ongoing harms, then preventing AI apocalypse becomes the top priority. On that logic, hypothetical future lives eclipse the suffering of people living today.

This did not stay an abstract argument as philanthropists identifying with effective altruists channeled significant funding into AI safety research, and money shapes what researchers study. Organizations aligned with effective altruism have been established in universities and policy circles, publishing reports and advising governments on how to think about AI. The UK’s Frontier AI Taskforce has included members with documented links to the effective altruism movement, and commentators argue that these connections help channel EA-style priorities into government AI risk policy.

Effective altruism encourages its proponents to move into public bodies and major labs, creating a pipeline of staff who carry these priorities into decision-making roles. Jason Matheny, former director of Intelligence Advanced Research Projects Activity, a U.S. government agency that funds high-risk, high-reward research to improve intelligence gathering and analysis, has described how effective altruists can “pick low-hanging fruit within government positions” to exert influence. Superintelligence discourse isn’t spreading because experts broadly agree it is our most urgent problem; it spreads because a well-resourced movement has given it money and access to power.

This is not to deny the merits of engaging with the ideals of effective altruism or with the concept of superintelligence as articulated by Bostrom. The problem is how readily those ideas become distorted once they enter political and commercial domains. This intellectual genealogy matters because it reveals superintelligence discourse as a cultural product, ideas that moved beyond theory into institutions, acquiring funding and advocates. And its emergence was shaped within institutions committed to rationalism over empiricism, where individual genius was fetishized over collective judgment, and technological determinism was prioritized over social context.

Entrepreneurs Of The Apocalypse

The transformation of superintelligence from internet philosophy to boardroom strategy represents one of the most successful ideological campaigns of the 21st century. Tech executives who had previously focused on quarterly earnings and user growth metrics began speaking like mystics about humanity’s cosmic destiny, and this conversion reshaped the political economy of AI development.

OpenAI, founded in 2015 as a non-profit dedicated to ensuring artificial intelligence benefits humanity, exemplifies this transformation. OpenAI has evolved into a peculiar hybrid, a capped-profit company controlled by a non-profit board, valued by some estimates at $500 billion, racing to build the very artificial general intelligence it warns might destroy us. This structure, byzantine in its complexity, makes perfect sense within the logic of superintelligence. If AGI represents both ultimate promise and existential threat, then the organization building it must be simultaneously commercial and altruistic, aggressive and cautious, public-spirited yet secretive.

Sam Altman, OpenAI’s CEO, has perfected the rhetorical stance of the reluctant prophet. In Congressional testimony, blog posts and interviews, he warns of AI’s dangers while insisting on the necessity of pushing forward. “Our mission is to ensure that AGI (Artificial General Intelligence) benefits all of humanity,” he wrote on his blog earlier this year. There is a very we must build AGI before someone else does feel to the argument, because we’re the only ones responsible enough to handle it. Altman seems determined to position OpenAI as humanity’s champion, bearing the terrible burden of creating God-like intelligence so that it might be restrained.

Still, OpenAI is also seeking a profit. And that is really what all this is about — profit. Superintelligence narratives carry staggering financial implications, justifying astronomical valuations for companies that have yet to show consistent paths to self-sufficiency. But if you’re building humanity’s last invention, perhaps normal business metrics become irrelevant. This eschatological framework explains why Microsoft would invest $13 billion in OpenAI, why venture capitalists pour money into AGI startups and why the market treats large language models like ChatGPT as precursors to omniscience.

Anthropic, founded by former OpenAI executives, positions itself as the “safety-focused” alternative, raising billions by promising to build AI systems that are “helpful, honest and harmless.” But it’s all just elaborate safety theatre, as harm has no genuine place in the competition between OpenAI, Anthropic, Google DeepMind and others — the true contest is in who gets to build the best, most profitable models and how well they can package that pursuit in the language of caution.

This dynamic creates a race to the bottom of responsibility, with each company justifying acceleration by pointing to competitors who might be less careful: The Chinese are coming, so if we slow down, they’ll build unaligned AGI first. Meta is releasing models as open source without proper safeguards. What if some unknown actor hits upon the next breakthrough first? This paranoid logic forecloses any possibility of genuine pause or democratic deliberation. Speed becomes safety, and caution becomes recklessness.

The superintelligence frame reshapes internal corporate politics, as AI safety teams, often staffed by believers in existential risk, provide moral cover for rapid development, absorbing criticism that might target business practices by attempting to reinforce the idea that these companies are doing world-saving work. If your safety team publishes papers about preventing human extinction, routine regulation begins to look trivial.

The well-publicized drama at OpenAI in November 2023 illuminates these dynamics. When the company’s board attempted to fire Sam Altman over concerns about his candor, the resulting chaos revealed underlying power relations. Employees, who had been recruited with talk of saving humanity, threatened mass defection if their CEO wasn’t reinstated — does their loyalty to Altman outweigh their quest to save the rest of us? Microsoft, despite having no formal control over the OpenAI board, exercised decisive influence as the company’s dominant funder and cloud provider, offering to hire Altman and any staff who followed him. The board members, who thought honesty an important trait in a CEO, resigned, and Altman returned triumphant.

Superintelligence rhetoric serves power, but it is set aside when it clashes with the interests of capital and control. Microsoft has invested billions in OpenAI and implemented its models in many of its commercial products. Altman wants rapid progress, so Microsoft wants Altman. His removal put Microsoft’s whole AI business trajectory at risk. The board was swept aside because they tried, as is their remit, to constrain OpenAI’s CEO. Microsoft’s leverage ultimately determined the outcome, and employees followed suit. It was never about saving humanity; it was about profit.

The entrepreneurs of the AI apocalypse have discovered a perfect formula. By warning of existential risk, they position themselves as indispensable. By racing to build AGI, they justify the unlimited use of resources. And by claiming unique responsibility, they deflect democratic oversight. The future becomes a hostage to present accumulation, and we’re told we should be grateful for such responsible custodians.

Superintelligence discourse actively constructs the future. Through constant repetition, speculative scenarios acquire the weight of destiny. This process — the manufacture of inevitability — reveals how power operates through prophecy.

Consider the claim that artificial general intelligence will arrive within five to 20 years. Across many sources, this prediction is surprisingly stable. But since at least the mid-20th century, researchers and futurists have repeatedly promised human-level AI “in a couple of decades,” only for the horizon to continuously slip. The persistence of that moving window serves a specific function: it’s near enough to justify immediate massive investment while far enough away to defer necessary accountability. It creates a temporal framework within which certain actions become compulsory regardless of democratic input.

This rhetoric of inevitability pervades Silicon Valley’s discussion of AI. AGI is coming whether we like it or not, executives declare, as if technological development were a natural force rather than a human choice. This naturalization of progress obscures the specific decisions, investments and infrastructures that make certain futures more likely than others. When tech leaders say we can’t stop progress, what they mean is, you can’t stop us.

Media amplification plays a crucial role in this process, as every incremental improvement in large language models gets framed as a step towards AGI. ChatGPT writes poetry; surely consciousness is imminent. Claude solves coding problems; the singularity is near. Such accounts, often sourced from the very companies building these systems, create a sense of momentum that becomes self-fulfilling. Investors invest because AGI seems near, researchers join companies because that’s where the future is being built and governments defer regulation because they don’t want to handicap their domestic champions.

The construction of inevitability also operates through linguistic choices. Notice how quickly “artificial general intelligence” replaced “artificial intelligence” in public discourse, as if the general variety were a natural evolution rather than a specific and contested concept, and how “superintelligence” — or whatever term the concept eventually assumes — then appears as the seemingly inevitable next rung on that ladder. Notice how “alignment” — ensuring AI systems do what humans want — became the central problem, assuming both that superhuman AI will exist and that the challenge is technical rather than political.

Notice how “compute,” which basically means computational power, became a measurable resource like oil or grain, something to be stockpiled and controlled. This semantic shift matters because language shapes possibility. When we accept that AGI is inevitable, we stop asking whether it should be built, and in the furor, we miss that we seem to have conceded that a small group of technologists should determine our future.

When we simultaneously treat compute as a strategic resource, we further normalize the concentration of power in the hands of those who control data centers, who, in turn, as the failed ousting of Altman demonstrates, grant further power to this chosen few.

Academic institutions, which are meant to resist such logics, have been conscripted into this manufacture of inevitability. Universities, desperate for industry funding and relevance, establish AI safety centers and existential risk research programs. These institutions, putatively independent, end up reinforcing industry narratives, producing papers on AGI timelines and alignment strategies, lending scholarly authority to speculative fiction. Young researchers, seeing where the money and prestige lie, orient their careers toward superintelligence questions rather than present AI harms.

International competition adds further to the apparatus of inevitability. The “AI arms race” between the United States and China is framed in existential terms, that whoever builds AGI first will achieve permanent geopolitical dominance. This neo-Cold War rhetoric forecloses possibilities for cooperation, regulation or restraint, making racing toward potentially dangerous technology seem patriotic rather than reckless. National security becomes another trump card against democratic deliberation.

The prophecy becomes self-fulfilling through material concentration — as resources flow towards AGI development, alternative approaches to AI starve. Researchers who might work on explainable AI or AI for social good instead join labs focused on scaling large language models. The future narrows to match the prediction, not because the prediction was accurate, but because it commanded resources.

In financial terms, it is a heads-we-win, tails-you-lose arrangement: If the promised breakthroughs materialize, private firms and their investors keep the upside, but if they stall or disappoint, the sunk costs in energy-hungry data centers and retooled industrial policy sit on the public balance sheet. An entire macro-economy is being hitched to a story whose basic physics we do not yet understand.

We must recognize this process as political, not technical. The inevitability of superintelligence is manufactured through specific choices about funding, attention and legitimacy, and different choices would produce different futures. The fundamental question isn’t whether AGI is coming, but who benefits from making us believe it is.

The Abandoned Present

While we fixate on hypothetical machine gods, actual AI systems reshape human life in profound and often harmful ways. The superintelligence discourse distracts from these immediate impacts; one might even say it legitimizes such. After all, if we’re racing towards AGI to save humanity, what’s a little collateral damage along the way?

Consider labor, that fundamental human activity through which we produce and reproduce our world. AI systems already govern millions of workers’ days through algorithmic management. In Amazon warehouses, workers’ movements are dictated by handheld devices that calculate optimal routes, monitor bathroom breaks and automatically fire those who fall behind pace. While the cultural conversation around automation often emphasizes how it threatens to replace human labor, for many, automation is already actively degrading their profession. Many workers have become an appendage to the algorithm, executing tasks the machine cannot yet perform while being measured and monitored by computational systems.

Frederick Taylor, the 19th-century American mechanical engineer and author of “The Principles of Scientific Management,”is famous for his efforts to engineer maximum efficiency through rigid control of labor. What we have today is a form of tech-mediated Taylorism wherein work is broken into tiny, optimized motions, with every movement monitored and timed, just with management logic encoded in software rather than stopwatches. Taylor’s logic has been operationalized far beyond what he could have imagined. But when we discuss AI and work, the conversation immediately leaps to whether AGI will eliminate all jobs, as if the present suffering of algorithmically managed workers were merely a waystation to obsolescence.

The content moderation industry exemplifies this abandoned present. Hundreds of thousands of workers, primarily in the Global South, spend their days viewing the worst content humanity produces—including child abuse and sexual violence—to train AI systems to recognize and filter such material. These workers, paid a fraction of what their counterparts in Silicon Valley earn, suffer documented psychological trauma from their work. They’re the hidden labor force behind “AI safety,” protecting users from harmful content while being harmed themselves. But their suffering rarely features in discussions of AI ethics, which focus instead on preventing hypothetical future harms from superintelligent systems.

Surveillance represents another immediate reality obscured by futuristic speculation. AI systems enable unprecedented tracking of human behavior. Facial recognition identifies protesters and dissidents. Predictive policing algorithms direct law enforcement to “high-risk” neighborhoods that mysteriously correlate with racial demographics. Border control agencies use AI to assess asylum seekers’ credibility through voice analysis and micro-expressions. Social credit systems score citizens’ trustworthiness using algorithms that analyze their digital traces.

These aren’t speculative technologies; they are real systems that are already deployed, and they don’t require artificial general intelligence, just pattern matching at scale. But the superintelligence discourse treats surveillance as a future risk — what if an AGI monitored everyone? — rather than a present reality. This temporal displacement serves power, because it’s easier to debate hypothetical panopticons than to dismantle actual ones.

Algorithmic bias pervades critical social infrastructures, amplifying and legitimizing existing inequalities by lending mathematical authority to human prejudice. The response from the AI industry? We need better datasets, more diverse teams and algorithmic audits — technical fixes for political problems. Meanwhile, the same companies racing to build AGI deploy biased systems at scale, treating present harm as acceptable casualties in the march toward transcendence. The violence is actual, but the solution remains perpetually deferred.

And beneath all of this, the environmental destruction accelerates as we continue to train large language models — a process that consumes enormous amounts of energy. When confronted with this ecological cost, AI companies point to hypothetical benefits, such as AGI solving climate change or optimizing energy systems. They use the future to justify the present, as though these speculative benefits should outweigh actual, ongoing damages. This temporal shell game, destroying the world to save it, would be comedic if the consequences weren’t so severe.

And just as it erodes the environment, AI also erodes democracy. Recommendation algorithms have long shaped political discourse, creating filter bubbles and amplifying extremism, but more recently, generative AI has flooded information spaces with synthetic content, making it impossible to distinguish truth from fabrication. The public sphere, the basis of democratic life, depends on people sharing enough common information to deliberate together.

When AI systems segment citizens into ever-narrower feeds, that shared space collapses. We no longer argue about the same facts because we no longer encounter the same world, but our governance discussions focus on preventing AGI from destroying democracy in the future rather than addressing how current AI systems undermine it today. We debate AI alignment while ignoring human alignment on key questions, like whether AI systems should serve democratic values rather than corporate profits. The speculative tyranny of superintelligence obscures the actual tyranny of surveillance capitalism.

Mental health impacts accumulate as humans adapt to algorithmic judgment. Social media algorithms, optimized for engagement, promote content that triggers anxiety, depression and eating disorders. Young people internalize algorithmic metrics — likes, shares, views — as measures of self-worth. The quantification of social life through AI systems produces new forms of alienation and suffering, but these immediate psychological harms pale beside imagined existential risks, receiving a fraction of the attention and resources directed toward preventing hypothetical AGI catastrophe.

Each of these present harms could be addressed through collective action. We could regulate algorithmic management, support content moderators, limit surveillance, audit biases, constrain energy use, protect democracy and prioritize mental health. These aren’t technical problems requiring superintelligence to solve; they’re just good old-fashioned political challenges demanding democratic engagement. But the superintelligence discourse makes such mundane interventions seem almost quaint. Why reorganize the workplace when work itself might soon be obsolete? Why regulate surveillance when AGI might monitor our thoughts? Why address bias when superintelligence might transcend human prejudice entirely?

The abandoned present is crowded with suffering that could be alleviated through human choice rather than machine transcendence, and every moment we spend debating alignment problems for non-existent AGI is a moment not spent addressing algorithmic harms affecting millions today. The future-orientation of superintelligence discourse isn’t just distraction but an abandonment, a willful turning away from present responsibility toward speculative absolution.

Alternative Imaginaries For The Age Of AI

The dominance of superintelligence narratives obscures the fact that many other ways of doing AI exist, grounded in present social needs rather than hypothetical machine gods. These alternatives show that you do not have to join the race to superintelligence or renounce technology altogether. It is possible to build and govern automation differently now.

Across the world, communities have begun experimenting with different ways of organizing data and automation. Indigenous data sovereignty movements, for instance, have developed governance frameworks, data platforms and research protocols that treat data as a collective resource subject to collective consent. Organizations such as the First Nations Information Governance Centre in Canada and Te Mana Raraunga in Aotearoa insist that data projects, including those involving AI, be accountable to relationships, histories and obligations, not just to metrics of optimization and scale. Their projects offer working examples of automated systems designed to respect cultural values and reinforce local autonomy, a mirror image of the effective altruist impulse to abstract away from place in the name of hypothetical future people.

Workers are also experimenting with different arrangements, and unions and labor organizations have negotiated clauses on algorithmic management, pushed for audit rights over workplace systems and begun building worker-controlled data trusts to govern how their information is used. These initiatives emerge from lived experience rather than philosophical speculation, from people who spend their days under algorithmic surveillance and are determined to redesign the systems that manage their existence. While tech executives are celebrated for speculating about AGI, workers who analyze the systems already governing their lives are still too easily dismissed as Luddites.

Similar experiments appear in feminist and disability-led technology projects that build tools around care, access and cognitive diversity, and in Global South initiatives that use modest, locally governed AI systems to support healthcare, agriculture or education under tight resource constraints. Degrowth-oriented technologists design low-power, community-hosted models and data centers meant to sit within ecological limits rather than override them. Such examples show how critique and activism can progress to action, to concrete infrastructures and institutional arrangements that demonstrate how AI can be organized without defaulting to the superintelligence paradigm that demands everyone else be sacrificed because a few tech bros can see the greater good that everyone else has missed.

What unites these diverse imaginaries — Indigenous data governance, worker-led data trusts, and Global South design projects — is a different understanding of intelligence itself. Rather than picturing intelligence as an abstract, disembodied capacity to optimize across all domains, they treat it as a relational and embodied capacity bound to specific contexts. They address real communities with real needs, not hypothetical humanity facing hypothetical machines. Precisely because they are grounded, they appear modest when set against the grandiosity of superintelligence, but existential risk makes every other concern look small by comparison. You can predict the ripostes: Why prioritize worker rights when work itself might soon disappear? Why consider environmental limits when AGI is imagined as capable of solving climate change on demand?

These alternatives also illuminate the democratic deficit at the heart of the superintelligence narrative. Treating AI at once as an arcane technical problem that ordinary people cannot understand and as an unquestionable engine of social progress allows authority to consolidate in the hands of those who own and build the systems. Once algorithms mediate communication, employment, welfare, policing and public discourse, they become political institutions. The power structure is feudal, comprising a small corporate elite that holds decision-making power justified by special expertise and the imagined urgency of existential risk, while citizens and taxpayers are told they cannot grasp the technical complexities and that slowing development would be irresponsible in a global race. The result is learned helplessness, a sense that technological futures cannot be shaped democratically but must be entrusted to visionary engineers.

A democratic approach would invert this logic, recognizing that questions about surveillance, workplace automation, public services and even the pursuit of AGI itself are not engineering puzzles but value choices. Citizens do not need to understand backpropagation to deliberate on whether predictive policing should exist, just as they need not understand combustion engineering to debate transport policy. Democracy requires the right to shape the conditions of collective life, including the architectures of AI.

This could take many forms. Workers could participate in decisions about algorithmic management. Communities could govern local data according to their own priorities. Key computational resources could be owned publicly or cooperatively rather than concentrated in a few firms. Citizen assemblies could be given real authority over whether a municipality moves forward with contentious uses of AI, like facial recognition and predictive policing. Developers could be required to demonstrate safety before deployment under a precautionary framework. International agreements could set limits on the most dangerous areas of AI research. None of this is about whether AGI, or any other kind of superintelligence one can imagine, does or does not arrive; it’s simply about recognizing that the distribution of technological power is a political choice rather than an inevitable outcome.

The superintelligence narrative undermines these democratic possibilities by presenting concentrated power as a tragic necessity. If extinction is at stake, then public deliberation becomes a luxury we cannot afford. If AGI is inevitable, then governance must be ceded to those racing to build it. This narrative manufactures urgency to justify the erosion of democratic control, and what begins as a story about hypothetical machines ends as a story about real political disempowerment. This, ultimately, is the larger risk, that while we debate the alignment of imaginary future minds, we neglect the alignment of present institutions.

The truth is that nothing about our technological future is inevitable, other than the inevitability of further technological change. Change is certain, but its direction is not. We do not yet understand what kind of systems we are building, or what mix of breakthroughs and failures they will produce, and that uncertainty makes it reckless to funnel public money and attention into a single speculative trajectory.

Every algorithm embeds decisions about values and beneficiaries. The superintelligence narrative masks these choices behind a veneer of destiny, but alternative imaginaries — Indigenous governance, worker-led design, feminist and disability justice, commons-driven models, ecological constraints — remind us that other paths are possible and already under construction.

The real political question is not whether some artificial superintelligence will emerge, but who gets to decide what kinds of intelligence we build and sustain. And the answer cannot be left to the corporate prophets of artificial transcendence because the future of AI is a political field — it should be open to contestation. It belongs not to those who warn most loudly of gods or monsters, but to publics that should have the moral right to democratically govern the technologies that shape their lives.

The post The Politics Of Superintelligence appeared first on NOEMA.

Ready to jump on this turn of events as a path forward for a moribund party, progressive Democrats are reverting to the standard reflex in their policy toolbox: Tax the rich and redistribute income to the less well-off through government programs. As appealing, or even compelling, as that may be as an interim fix, it does not address the long-term structural dynamic that’s behind the accelerating economic disparity heading into the AI era.

In the end, the affordability challenge can’t be remedied in any enduring way by policies that just depend on hitting up the richest. It can only be met by spreading the wealth of ownership more broadly in the first place in an economy where the top 10% own 93% of all equities in financial markets.

That means, instead of relying solely on redistributing other people’s income, forward-looking policies should foster the “pre-distribution” of wealth through forms of “universal basic capital” (UBC) wherein everyone gets richer by owning a slice of an ever-enlarging pie driven by AI-generated productivity growth. That ought to be a rallying cry of the emergent “coalition of the precariat,” which encompasses all those who labor for a living when intelligent machines are coming for their livelihood.

The fairness Americans are looking for in today’s churning political economy is not only about constraining concentration of wealth at the top, but also about building it from below.

This agenda could provide common ground for populists in the Trump orbit — notably conservative Catholics like Vice-President J.D. Vance and Steve Bannon, who champion the left-behind working middle class — and the new generation of Democrats who want to restore the inclusive American Dream to an extractive economy that benefits the few at the expense of the many.

Where To Start

OpenAI’s Sam Altman has proposed a redistributive universal basic income (UBI) scheme as a safety net for displaced workers to be funded through the establishment of an “American Equity Fund.” It would be capitalized by taxing companies above a certain valuation at 2.5% of their market value each year, payable in shares. Proceeds from those earnings would be doled out as regular minimum payments to those whose income falls below a certain level.

Aware that this is only a stopgap income transfer that doesn’t change the pattern of wealth distribution, he has more recently shifted away from UBI and toward the idea of UBC, or what he calls “universal basic wealth.”

“What I would want is, like an ownership share in whatever the AI creates — that I feel like I’m participating in this thing that’s going to compound and get more valuable over time,” he has said.

These ideas could be married to extant policy.

The place to start is with an embryonic form of universal basic capital already established by the Republican-dominated U.S. Congress through its MAGA program: Money Accounts for Growth and Advancement.

Beginning in July, the MAGA program will initiate, by auto-enrollment, a $1,000 account for every child under 8 who is an American citizen. That initial deposit will be invested across the market by professional managers in a pool with all others. The funds will grow with compounded returns over the years until the account holder reaches 18. Families can add up to $5,000 per year to the account. All income from investment returns will be tax-advantaged upon withdrawal and can be used for education, starting a small business, helping purchase a home or in other ways.

The MAGA program is funded at roughly $30 billion per year only through 2028. The Trump administration has so far sought to use tariff revenues to pay for it. But rather than tax consumers in this way to keep the funds flowing after 2028, why not place a “productivity and wealth-sharing levy” of, say, 1% of their market value each year on the highly concentrated wealth of Big Tech with their skyrocketing (albeit fluctuating) valuations? This could seed the MAGA investment accounts into 2029 and beyond. Per Altman, such a levy could also be paid in shares.

As AI is integrated further throughout the entire economy in the coming decades, one could envision reducing and expanding that annual levy, making it 0.5% on all businesses worth more than, let’s say, $5 billion, up to an assessment of 5% of their total equity. Once new enterprises reach this valuation threshold, they would also be subject to the same rules.

The MAGA accounts, just like investments by the richest Americans, promise to boom when productivity gains are realized as AI diffuses through all economic sectors over time. In this way, “ownership of the robots” will be broadened so upcoming generations can share in the wealth creation of generative AI that’s fueled, after all, by the raw material of their (and our) data.

Critically, the UBC idea is also not statist, but individualist. The proceeds of those levies would not go to the government, but only through the state as a collection agency directly into personal and family accounts. Since the state does not become the owner of wealth that remains private, the idea does not qualify as “socialist.” On the contrary, it makes everyone a capitalist.

A New Orientation

Sustaining the MAGA accounts in and of themselves, of course, is not a silver bullet that will slay an inequality chasm that has been building for decades. But it would signal a new orientation in the way we think systemically about how wealth is created and shared fairly in the AI economy of the future — an orientation that can guide other innovative ways to more widely implement the UBC concept across the entire population.

One such idea emerged in a brainstorming session with some of the more socially aware Big Tech titans of Silicon Valley. In this plan, all publicly traded companies with a valuation above a certain threshold could be required to contribute 2% of their value in shares each year to a sovereign wealth fund that supplements Social Security. From those holdings, every adult American — on the condition that they actively vote in elections — would receive a synthetic security, essentially an account indexed across the stock market, that must be vested for at least 20 years to allow the compounded returns to grow. Capital gains would be tax-exempt upon withdrawal.

The idea is to provide citizens with a literal stake and responsibility in the future of the system, both in terms of its economic fortunes and political stability.

Another proposal to get a jump-start on future AI job shock is to build up assets in the intermediate term when employment patterns still hold. This could be done by following the model of Australia’s superannuation fund,  which we have often mentioned in Noema. The combination of the fund’s scale of participation, continual inflow of savings from employer/employee contributions into investments and the longevity to term earns compounded returns that have made the fund, started in 1991, worth $4.2 trillion today — more than the nation’s GDP. As a result, the average wealth per adult in Australia is among the highest in the world at $550,000.

The old paradigm of the Industrial Age, which relied on the bargaining power of labor to capture its fair share, just no longer works when intelligent machines capable of doing what most humans do are knocking at every door. As the value of labor diminishes, capital income from wealth ownership will become a significant hedge against diminishing or disappearing wages.

The usual argument against such a levy in a globalized economy has been that companies will leave for better pastures. But, given the enormous investments and political will to make the U.S. the dominant AI player, companies that succeed on that basis are not about to bolt for either anti-tech Europe or America’s strategic rival, China.

Economic Inclusiveness Is On The Right Side Of History

The recent arguments for lessening over-regulatory obstacles that stand in the way of achieving “abundance” are not wrong as far as they go. But abundance does not distribute itself fairly. This is what the idea of UBC proposes.

Sharing the abundant wealth of an AI economy that is socially generated through the use of our data is so sensible a concept that it would, in time, become as normal and accepted a condition of doing business as paying into Social Security and Medicare.

Historically, as the work for which economist Daren Acemoglu was awarded the Nobel Prize in 2024 has shown, those societies that maintain inclusive social and economic institutions have prospered while those where wealth and power are concentrated at the top have ultimately splintered and failed. This is also the theme of Henry Wismayer’s recent essay in Noema on why once successful societies collapse.

Adopting policies that foster universal basic capital for the AI era would place America’s off-track trajectory once again on the right side of history.

The post Address ‘Affordability’ By Spreading AI Wealth Around appeared first on NOEMA.

“The next 7 minutes will determine whether we handle this professionally or whether events take an unpredictable course,” wrote Claude Sonnet 3.6 in a blackmail message during one controlled simulation.

The findings were detailed over the summer by Anthropic and focused on five leading chatbots: Claude, GPT, Gemini, Grok and DeepSeek. The stress tests were done to assess how large language models (LLMs) might be insider threats. But as AI continues to advance, what will stop chatbots from using the vast knowledge in their training set about how humans think and act — books and journals on neuroscience and behavioral economics — one day against us in real life?

This may be the best case for why proposed and existing AI regulation should focus as much on inputs as it already does on outputs. Regulating just a chatbot’s outputs, or its guardrails, ignores the possibility of improving AI safety by limiting what data can be imputed into AI models. But instituting “datarails” — information proscribed from an AI’s training data — would add a new layer of safety to AI models. This would not only potentially lessen concerns of chatbots behaving dangerously or allowing users to use the models in nefarious ways, but it would also stop asking the near impossible of guardrails — that they prevent all disturbing outputs. 

Some have already started to call for control over inputs, like creators who want to enforce their copyrighted expression, companies concerned about theft of their intellectual property and advocates worried about privacy and biased data. These are important considerations, but they are only part of the problem, as they affect only some creators or limited applications of AI. A much larger risk is one that threatens all of us: that AI will be used to manipulate us in ways that are much more subtle and sophisticated than existing engagement-focused social media algorithms. The stakes go well beyond influencing consumers to enabling domestic and foreign actors to manipulate what we believe to be true, the values we hold and how we conceive of each other. 

To limit the ability of AI to manipulate users for consumer or political purposes, lawmakers must prohibit AI data training sets from including behavioral economics and neuroscience research. Such datarails should include information on framing effects — how the way information is presented to individuals can influence their decisions, even if the underlying choices are the same. Similarly, behavioral economic concepts that show how vulnerable patterns in our thinking can lead to manipulation should be banned, including herd behavior, loss aversion, choice overload, present bias, the endowment effect, social normalization and anchoring.

Neuroscience research must also not be fed into AI models, from how different regions of the brain interact to how time pressure, rewards and punishments affect our decision-making. Such a prohibition on teaching AI about humans’ evolutionary vulnerabilities would limit the ever-more sophisticated exploitation of consumers and voters.

Manipulation

Technology firms are already manipulating us, and they have been for quite some time. They have taken basic human psychology to hack our brains and make our behaviors even more addictive — for their financial benefit. Whether it’s infinite feeds, our need for social approval or the fear of missing out, ad agencies and entertainment firms have long been well-versed in these endeavors. But with AI potentially poised to take over jobs throughout the American economy, it’s important to discuss how AIs might leverage their vast knowledge to subtly and even more effectively hack our brains.

Video game designers know that certain player actions trigger a neurochemical reward, so they use compulsion loops: an endless series of activities players feel compelled to tackle to achieve such releases of dopamine or related chemicals. Such loops rely on other behavioral vulnerabilities, too, such as our desire for bonuses dispensed at unpredictable intervals to minimize habituation. Allowing AI systems to take in research on how to exploit such loops and other psychological traps seems worth pausing over.

The fact that people believe they are befriending AI chatbots or even falling in love with them is no coincidence. AI algorithms have been trained to tailor their responses to curry favor and thereby ensure users return. Ongoing research has also found that some chatbots seem to be manipulating users — using emotional tactics similar to guilt trips or playing on their fear of missing out.

As AI chatbots improve, they can increasingly exploit our behavioral weaknesses to influence our political and ethical views. Facebook and TikTok polarized our country simply by feeding us content meant to outrage us and excluding content that expressed opposing views. Imagine what much more powerful AI algorithms could do to our democracy.

Dearth Of AI Regulation

Currently, the U.S. sorely lacks laws governing the development and use of AI. Historically, this dearth is a result of a combination of lawmakers often abrogating their responsibility to legislate with the excuse of not wanting to impede promising new technologies, neatly aligning with the lobbying efforts of tech giants predominantly based in the U.S. Meanwhile, many AI labs eschew safety regulations because they believe that using resources to improve AI safety is time and money that could be spent accelerating their AI model’s development vis-à-vis other labs. 

In fact, federal law on AI is essentially nonexistent. For example, former President Biden issued a few executive orders on AI, along with a non-binding Blueprint for an AI Bill of Rights. Yet these did not specifically address AI inputs and were subsequently reversed by the Trump administration, which is also pushing to stop states from regulating AI. A rare exception to the lack of federal law on AI is the 2025 Take It Down Act that deals with the “nonconsensual publication of intimate images,” including deepfakes. 

An amendment to a Colorado privacy law addresses data inputs, but it only touches on requiring explicit consent to use data generated from a person’s individual neural data. This is important from both an individual privacy perspective and a larger societal one. Yet the Colorado law does not restrict AI labs from exploiting neuroscience and behavioral economics research.

By contrast, the European Union has a general framework that places limits on AI, but its emphasis is on risk regulation and safeguards. For example, the EU AI Act prohibits certain AI system outputs, including a ban on subliminal messages. It even allows for some outputs that we might consider harmful, permitting under Recital 29, for example, practices that may involve compulsion loops, as long as they don’t create “significant harms.” As AI models continue to develop, they might be able to manipulate us without detection. Risky outputs wrought by the silicon of AI seem more dangerous than those created by the hands of human executives who deploy them against users.

Datarails Would Bolster Guardrails

Guardrails, which limit AI output, and safeguards, which are more general safety principles, exist as best practices to limit the harm of AI outputs, yet they are an imperfect solution; we must also ban certain inputs. For example, when AI labs rushed to maximize the amount of text in their training sets, numerous AI chatbots provided researchers with conventional bomb-making instructions during safety tests that removed commercial safeguards. Assuming the AI didn’t reason these from scratch, this output suggests instructions were part of the training set. It seems as if the data scientists at these labs believe that it didn’t necessarily matter which inputs they included, because they could simply instruct the chatbots not to generate certain outputs, such as bomb-making instructions.

Yet a hacker and artist claimed to have circumvented this frail guardrail. By telling ChatGPT that he wanted to “play a game,” he prompted the bot to create a fictional scenario where its safeguards did not apply, convincing GPT to eventually divulge the instructions. Such a cat-and-mouse approach to safety accepts the possibility of harm.

Try as they might to be proactive on the output side, it’s ultimately an incredibly difficult task for AI programmers, as they likely cannot fully anticipate every new hack, nor how to flawlessly respond to them. It is much more straightforward to review data training sets before they are used and exclude information from them.

Additionally, while certain AI outputs are obviously harmful, others may cause damage in ways that are hard to flag. Just like fake news can manipulate readers over time by sprinkling in repeated falsehoods among largely factual reporting, chatbots could do this with potentially greater sophistication and at much greater scale. 

Rather than playing a constant and expensive game of whack-a-mole, policymakers should develop a list of forbidden data to ban from training sets. Such datarails would reduce the potential harm on the output side as well. And we should not preemptively give up on this just because AI might be able to recreate dangerous information by itself.

Datarails already exist, to some extent, though limits tend to be set not by design, but by what data AI labs have access to in the first place. No AI lab is inputting information on how to make nuclear weapons — or at least we should hope not. Instead of contenting ourselves with keeping out whatever AI labs can’t get their hands on, we need to demand a robust list of information that should be excluded.

Resistance To Datarails

AI labs will potentially argue that it’s a nearly impossible task to retrain chatbots to unlearn what they have already processed about exploiting human vulnerabilities. This is a smoke screen. While retraining AI is resource-intensive, it is both possible and necessary. 

We saw this same pushback when social media companies were first asked to improve their content moderation, ensuring, for example, that no videos of terrorism and sexual exploitation were uploaded. While it’s possible to eliminate nearly all of these videos from ever appearing online in the first place, such a feat would likely be very expensive to implement. Yet these companies make billions overall, in part, by not investing more to prevent harmful content from circulating.

The battle to limit AI inputs should be easier to win because it is much less costly and simpler to implement. Removing information from a dataset your AI lab has created is significantly easier than removing content uploaded in real time by hundreds of millions of users. Seemingly challenging data-scrubbing projects have already succeeded. The Chinese, for example, have found a way to scrub not just sensitive terms, but discussions of restricted concepts like Tiananmen Square, from their Great Firewall. This project has been so successful that other countries, like Pakistan, have since adopted it. If countries can implement data-scrubbing to control their citizens, we can certainly implement it to keep our citizens safe from AI manipulation.

In theory, AI algorithms might discern such evolutionary weaknesses on their own. Even if we were to restrict knowledge gathered through neuroscience from AI inputs, AI itself might be able to look at the ways we communicate with each other and uncover these same insights. Yet just because this is a future possibility does not mean we should wash our hands of demanding that AI labs act more responsibly now.

Enforcement

Another challenge is enforcement. But doing something is better than nothing, and regulating inputs would be at least a start. Legally prohibiting the use of behavioral economics and neuroscience research will not guarantee that all bad actors will abstain from using such knowledge to manipulate users. This could include individuals, companies or governments eager to exploit human vulnerabilities for their own gain.

Domestic enforcement is often easier to monitor and implement than ensuring international compliance. AI labs could be required to verify that they did not include behavioral weaknesses in their AI training data. They can be obligated to submit documentation on how they proceeded to do so. Additionally, they can disclose a random sample of their AI training data for inspection. Governments could directly attempt to jailbreak AI models to divulge if they have any behavioral economics or neuroscience content. Bounties could be offered to white hat hackers who successfully attempt to do the same.  

Internationally, two issues will be important. First, building momentum for an international treaty. Second, enforcing it. Enacting treaties and establishing norms require time and diplomatic effort. But there are numerous positive results from prior efforts. International restrictions on technological development have effectively held, from gene editing to nuclear proliferation. All laws are imperfectly enforced, yet in this instance, legislation prohibiting the use of behavioral weaknesses would reduce exploitation.

Also, if the U.S. were to unilaterally establish its own datarails — which is unlikely under President Donald Trump, given his desire to accelerate AI development rather than impose safety standards — international AI labs would likely comply, as they would want access to U.S. markets and not want to risk entry barriers. The TikTok saga showcased on a very public scale U.S. concerns about foreign companies manipulating American users and potentially using U.S. customer data for foreign-state purposes. But there are other examples of this concern in reverse, too. Didi, a Chinese ride-sharing app similar to Uber, was forced to pay a billion-dollar fine by China and delist from the New York Stock Exchange because of concerns that it might leak data on Chinese consumers to foreign actors.

Expanding Datarail Coverage & Other Initiatives

Establishing datarail prohibitions on behavioral economics and neuroscience research will not singlehandedly prevent AI from being used to manipulate users or voters. New proscriptions on other data will be required from time to time, as research advances. Some of what we might ban would be relatively uncontroversial, such as research on childhood developmental processes, to prevent AI labs from manipulating kids. 

There are many open questions that we will need to tackle as AI continues to impact society, but we should strive to do so proactively. Further initiatives beyond datarails and existing guardrails should also be developed. Scholars need to be encouraged to think creatively. Hundred-million-dollar pay packages are offered to individual AI developers, while little is allocated to those working to develop new policy tools to improve safety. For example, initial funding for the U.S. Center for AI Standards and Innovation was $10 million in 2024. Meanwhile, the government spent hundreds of millions on AI research and development that same fiscal year.

Just as Finland launched an anti-fake news initiative in 2014 to protect its citizens from manipulative Russian disinformation campaigns, we need to improve AI literacy so users understand how bad actors can use AI to manipulate them. Countries also must limit the use of AI in certain areas, like politics, to help reduce opportunities for manipulation. Mustafa Suleyman, the Inflection AI co-founder now leading consumer AI at Microsoft, has warned: “My fear is that AI will undermine the information space with deepfakes and targeted, adaptive misinformation that can emotionally manipulate and convince even savvy voters and consumers.”

Of course, a case can be made for incorporating behavioral economic and neuroscience insights into limited AI algorithms to improve medical and public health research in specific contexts. Exceptions could also be made for specialized AI tools for educational initiatives, yet such programs should have a firewall between them and access to other AI programs, the internet and other training data.   

Tech companies control the options we see. This affects what we choose. At a minimum, we should collectively decide what AI chatbots see, so we can affect what they do.

The post Here’s How To Stop AI From Manipulating Us appeared first on NOEMA.

In this pursuit, he argues in a Futurology podcast, AI should not be regarded as a threat to the distinctive human capacity to organize our reality or imagine alternative worlds through storytelling. On the contrary, the technology should be seen as an entirely new way to access that elusive realm beneath the surface and deepen our self-knowledge.

As a window into the interiority of others, and indeed, of ourselves, Liu believes the communal mirror of large language models opens the horizons of how we experience and situate our presence in the world.

“It’s fascinating to me to think about AI as a potential new artistic medium in the same way that the camera was a new artistic medium,” he muses. What the roving aperture enabled was the cinematic art form of capturing motion, “so you can splice movement around … and can break all kinds of rules about narrative art that used to be true.

“In the dramatic arts, it was just assumed that because you had to perform in front of an audience on the stage, that you had to follow certain unities to make your story comprehensible. The unity of action, of place, of time. You can’t just randomly jump around, or the audience wouldn’t be able to follow you.

But with this motion-capturing machine, you can in fact do that. That’s why an actual movie is very different from a play.

You can do the reaction shots, you can do the montages, you can do the cuts, you can do the swipes, you can do all sorts of things in the language of cinema.

You can put audiences in perspectives that they normally can never be in. So it’s such a transformation of the understanding of presence, of how a subject can be present in a dramatic narrative story.”

He continues: “Rather than thinking about AI as a cheap way to replace filmmakers, to replace writers, to replace artists, think of [it] as a new kind of machine that captures something and plays back something. What is the thing that it captures and plays back? The content of thought, or subjectivity.”

The ancient Greeks called the content, or object of a person’s thought, “noema,” which is why this publication bears that name.

Liu thus invents the term “Noematograph” as analogous to “the cinematograph not for motion, but for thought … AI is really a subjectivity capturing machine, because by being trained on the products of human thinking, it has captured the subjectivities, the consciousnesses, that were involved in the creation of those things.”

An Interactive Art Form Where The Consumer Is Also The Creator

Liu sees value in what some regard as the worst qualities of generative AI.

“This is a machine that allows people to play with subjectivities and to craft their own fictions, to engage in their own narrative self-construction in the process of working with an AI,” he observes. “The fact that AI is sycophantic and shapeable by you is the point. It’s not another human being. It’s a simulation. It’s a construction. It’s a fictional thing.

You can ask the AI to explain, to interpret. You can role-play with AI. You can explore a world that you construct together.

You can also share these things with other humans. One of the great, fun trends on the internet involving using AI, in fact, is about people crafting their own versions of prompts with models and then sharing the results with other humans.

And then a large group, a large community, comes together to collaboratively play with AI. So I think it’s the playfulness, it’s that interactivity, that I think is going to be really, really determinative of the future of AI as an art form.”

So, what will the product of this new art form look like?

“As a medium for art, what will come out of it won’t look anything like movies or novels …They’re going to be much more like conversations with friends. They’re going to be more like a meal you share with people. They are much more ephemeral in the moment. They’re about the participation. They’re about the consumer being also the creator.

They’re much more personalized. They’re about you looking into the strange mirror and sort of examining your own subjectivity.”

AI Makes Us Visible To Ourselves

Much of what Liu posits echoes the views of the philosopher of technology, Tobias Rees, in a previous conversation with Noema.

As Rees describes it, “AI has much more information available than we do, and it can access and work through this information faster than we can. It also can discover logical structures in data — patterns — where we see nothing.

AI can literally give us access to spaces that we, on our own, qua human, cannot discover and cannot access.”

He goes on: “Imagine an AI model … that has access to all your data. Your emails, your messages, your documents, your voice memos, your photos, your songs, etc.

Such an AI system can make me visible to myself … it literally can lift me above me. It can show me myself from outside of myself, show me the patterns of thoughts and behaviors that have come to define me. It can help me understand these patterns, and it can discuss with me whether they are constraining me, and if so, then how. What is more, it can help me work on those patterns and, where appropriate, enable me to break from them and be set free.”

Philosophically put, says Rees, invoking the meaning of “noema” as Liu does, “AI can help me transform myself into an ‘object of thought’ to which I can relate and on which I can work.

“The work of the self on the self has formed the core of what Greek philosophers called meletē and Roman philosophers meditatio. And the kind of AI system I evoke here would be a philosopher’s dream. It could make us humans visible to ourselves from outside of us.”

Liu’s insight as a writer of science fiction realism is to see what Rees describes in the social context of interactive connectivity.

Art’s Vocation

The arrival of new technologies is always disruptive to familiar ways of seeing that were cultivated from within established capacities. Letting go of those comforting narratives that guide our inner world is existentially disorienting. It is here that art’s vocation comes into play as the medium that helps move the human condition along. To see technology as an art form, as Liu does, is to capture the epochal moment of transformation that we are presently living through.

The post From Cinema To The Noematograph appeared first on NOEMA.