Weak AI safety regulations may backfire, creating a product that is potentially more dangerous than AI products created under no regulation, according to a new study published Monday in the Proceedings of the National Academy of Sciences. The research, led by Benjamin Laufer and colleagues from Cornell University and Carnegie Mellon University, uses theoretical economics and game theory to model how AI regulation can be most effective at ensuring safety.
The study’s central claim is that for AI to be genuinely safe, regulation needs to be strict and must target the companies that develop AI models, including OpenAI, Google, and Anthropic, rather than focusing solely on the downstream companies that apply the technology in real-life settings, such as providers of AI medical diagnostic systems or e-commerce customer service chatbots. At first glance, regulating specific AI use cases might seem logical. But the authors argue that this approach can backfire and reduce the overall safety of AI products.
Key Findings of the Study
- Weak AI safety regulation may produce worse outcomes than no regulation at all.
- Regulation must cover the entire AI supply chain, not just end-use applications.
- When only downstream companies are regulated, general-purpose AI developers tend to skimp on safety measures like third-party audits.
- Game theory suggests that strict, well-enforced regulation can align incentives and improve safety for everyone.
- The study was published in the Proceedings of the National Academy of Sciences and conducted by researchers at Cornell University and Carnegie Mellon University.
The Free-Riding Problem in AI Safety
The reason weak regulation can fail, according to the researchers, lies in the structure of the AI supply chain. When the government focuses its regulatory efforts on downstream companies while letting general-purpose AI developers off the hook, those developers tend to cut corners on safety measures like third-party audits. They do this hoping that the downstream companies will ensure the safety of the end product instead. The authors call this a free-riding behavior, where the general provider offloads the safety burden onto the downstream specialist.
This dynamic is not just theoretical. The AI industry is marked by a complex chain of actors. Foundation model developers invest enormous resources in training large language models and other general-purpose systems. These models are then licensed to specialists who fine-tune them for specific tasks. If only the specialist is held accountable for safety, the general-purpose developer has little incentive to invest in costly safety measures. After all, any harm caused by the end product may be blamed on the specialist. The result is an overall weaker safety posture than if no regulation existed and both parties had to take responsibility for their own risk.
Game Theory and the Prisoner’s Dilemma
The researchers frame this situation as a classic prisoner’s dilemma. In game theory, the prisoner’s dilemma describes a scenario where two rational decision-makers are given the option to cooperate or betray each other. If they both choose to cooperate, they get the best possible outcome, but neither knows what the other will choose. If they both betray, they get a mediocre outcome. If one cooperates and the other betrays, the one who cooperated gets the worst outcome. Unsure of what the other will do, each decision-maker often chooses to betray, guaranteeing their own benefit but ensuring a worse outcome for everyone than if they had cooperated.
In the AI context, the two players are the general-purpose AI producers and the downstream domain specialists. Both can invest heavily in safety, or both can skimp on safety to save money and maximize revenue. Without a mechanism to ensure trust, each party assumes the other will not invest adequately in safety, so both invest as little as possible. The result is a low-safety equilibrium.
Strong regulation that applies across the entire AI supply chain can solve this problem. When regulators expect both the general-purpose AI producers and the downstream companies to make sufficient investment to meet meaningful safety standards, the rational choice for both is to cooperate. Strict regulation ensures trust rather than free-riding, providing the grounds for cooperation and the most ideal outcome for all.
Safety and Revenue Are Not an Either-Or Proposition
The authors argue that safety versus revenue does not have to be an either-or situation. According to their model, stronger, well-placed regulation can mutually benefit all players by improving both the safety of the end product and the utility that general-purpose AI creators and downstream domain specialists get from their investment. The researchers define utility as revenue share minus investment cost.
While companies often worry that regulation will eat into profits, the model suggests that well-designed regulation can actually increase the value created by AI systems. When safety is a shared responsibility, the risk of catastrophic failure decreases, which in turn protects brand reputation, avoids costly lawsuits, and builds user trust. That trust can translate into more widespread adoption and higher overall revenue.
According to the researchers, a sweet spot exists when regulators expect both general-purpose AI producers and downstream companies to make enough investment to meet meaningful safety standards. This is a departure from the current debate, which often pits safety against innovation. The study suggests that safety and innovation are not inherently in conflict, provided that regulators create the right incentives for cooperation.
The Political Context of AI Regulation
The study arrives at a moment when the United States government and Silicon Valley are deeply divided over how artificial intelligence should be regulated. Two main camps have formed. On one side are anti-regulation technologists who envision much lighter federal guardrails, largely aligning with the Trump administration’s approach to AI governance. This group argues that the AI industry should be free of unnecessary guardrails so it can innovate as quickly as possible, because that is the only way the United States can win the global AI race against China.
The self-proclaimed pro-innovation group tends to cast those who favor stricter AI safety regulation as doomers, and at worst as attempting regulatory capture. The supporters of stricter federal AI regulation, however, claim that the AI industry is underestimating or underselling the risks of under-regulated AI development. They point to a long list of purported downsides, from AI psychosis to the community health consequences of data centers and a much-feared unemployment crisis expected to follow wider AI adoption.
This political polarization makes the new study particularly relevant. The authors are not taking a stance in the culture war between accelerationists and safety advocates. Instead, they are using mathematical models to show that the structure of regulation matters. The message is clear: weak regulation is not a middle ground, and may be worse than none.
What This Means for Policymakers
The study suggests that policymakers should think carefully about where to place regulatory burdens. A focus on downstream uses of AI, such as medical diagnostics or chatbots, may seem intuitively appealing because those are the places where consumers directly interact with the technology. But the research indicates that this approach leaves the root of the system under-regulated and encouraged to push safety costs onto others.
Instead, the authors argue for a more holistic view. Laufer said that people think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology. He added that to regulate in a thoughtful way, society needs to consider the whole supply chain, not just a single provider or entity.
The implications extend beyond the United States. As other countries race to develop their own AI industries, they will be watching the U.S. experience carefully. If the U.S. adopts weak or fragmented regulation and then suffers a major AI safety incident, it could set back public trust in AI around the world. Conversely, a regulatory framework that succeeds in fostering both safety and innovation could become a model for other nations.
The Need for Comprehensive Safety Standards
One of the key takeaways from the research is that safety standards need to be meaningful enough to actually change behavior. If regulations are too lax, they may be ignored entirely, but the mere existence of a regulatory framework can create a false sense of security. This is why the researchers describe weak regulation as potentially worse than no regulation. Without any regulation, companies might take independent responsibility for safety because they know they will be exposed to liability or reputational damage. With weak regulation, they can point to their compliance and assume that someone else is handling the rest.
The game theory model also highlights the importance of enforcement. A regulation that exists on paper but is not enforced will not create the trust necessary for cooperation. Regulators need to demonstrate that they are serious, that they will monitor compliance, and that they will penalize noncompliance. Only then will companies in the AI supply chain believe that their counterparts are also under pressure to invest in safety.
There is also a question of what 'safety' means in the context of AI. The study does not define specific standards, but it implies that there should be common requirements for third-party audits, risk assessments, and perhaps incident reporting. These are the kinds of measures that general-purpose AI developers might be tempted to skip if they think downstream companies will be held accountable for the final product.
As the study notes, AI is not a single object. It is a complex system of developers, deployers, and users, each with their own incentives. The key to effective regulation lies in aligning those incentives through carefully targeted and strictly enforced rules. The stakes are high, but the path forward may be easier than many think if regulators are willing to look at the entire supply chain rather than just the tip of the iceberg.
Source:Gizmodo News