What risks does AI pose?

AI systems already pose many significant existing risks including harmful malfunctions, discrimination, reducing social connection, and invasions of privacy. Training and deploying AI systems can also involve copyright infringement and worker exploitation.

Future AI systems could exacerbate anticipated catastrophic risks, including bioterrorism, disinformation and resulting institutional dysfunction, misuse of concentrated power, nuclear and conventional war, and other coordination failures.

This document compiles explanations of these risks, drawing heavily from previous overviews by academics. For brevity, we can't cover every possible risk but we encourage readers to further research areas that interest them. We also don't cover AI takeover risks, or potential benefits of AI, which are further explored in other course resources.

Existing risks

Harmful malfunctions

AI systems can make mistakes if applied inappropriately. For example:

• A self-driving car in San Francisco collided with a pedestrian that was thrown into its path by a human driver. This was arguably not its fault - however, after initially stopping it then started moving again, dragging the injured pedestrian a further six meters along the road. Government investigators alleged that the company initially hid the severity of the collision from them.
• A healthcare chatbot deployed in the UK was heavily criticised when it advised users potentially experiencing a heart attack not to get treatment. When these concerns were raised by a doctor, the company released a statement calling them a "Twitter troll".

Furthermore, use of AI systems can make it harder to detect and address process issues. Outputs of computer systems are likely to be overly trusted. Additionally, because most AI models are effectively black boxes and AI systems are much more likely to be protected from court scrutiny than human processes, it can be hard to prove mistakes.

Discrimination

AI systems are trained with data that can reflect existing societal biases and problematic structures, resulting in systems that learn and amplify those biases (Turner Lee, et al., 2019). For example:

• Hiring systems have shown biases against minorities when trained on data from industries that lack diversity.
• Algorithms used in the US criminal justice system have been shown to be biased against African Americans, resulting in longer periods of detention while awaiting trial.

AI systems that might not have inherent biases may still exacerbate discriminatory practices based on the societal context in which they are deployed. For example, unequal access to AI knowledge and skills could further entrench inequality. Those with higher incomes, more formal education, and exposure to technology are more likely to be aware of and use advanced AI tools to their benefit.

Reducing social connection

Two effects can result in greater polarisation of opinions, reducing social connection:

• Recommender systems can result in filter bubbles, where users are only shown a particular type of content - for example, that which agrees with their views.
• Optimising for engagement can result in amplifying divisive content, such as articles that promote moral outrage or misinformation, despite users not wanting this. This can push users to more extreme positions, although this is debated.

In addition, chatbots may lead to unhealthy expectations in human relationships. Some virtual friend or romantic partner apps have millions of active customers, and it's uncertain what the results of using these apps will be.

Invasions of privacy

AI systems are often used to process personal data in unexpected or undesirable ways. For example:

• Period-tracking apps are used by over a third of women in the UK and US to help track their menstrual cycles, including 69% of UK women aged 18-24. These apps sometimes share information about their users for targeted advertising. Even when the companies building AI recommender systems do not intend this, AIs could learn from this data that it's effective to target people when they're most vulnerable to specific types of advertising, such as after a miscarriage, becoming pregnant, or recording particular symptoms such as mood swings or anxiety.
• AI-enabled facial recognition systems in public spaces can cause significant harms. One startup sold facial recognition tech to businesses and governments in the US, Panama, Brazil, Mexico, and the Dominican Republic, after training it on 30 billion internet photos. This type of system could be misused by campaign groups or journalists to publish lists of people visiting LGBTQ centres, sexual health clinics or rehab facilities, who might want to keep that information private. Data generated by this system could be stored indefinitely with no intention of being used maliciously, but later misused: many Jews were identified from church and government tax records during the Holocaust. Multiple recent examples of states criminalising previously legal behaviour amplify privacy risks, such as abortion in the US (2022) or homosexuality in India (2013).

Copyright infringement

Copyrighted works are used to train many AI systems, including basically all large language models and image generation models. Datasets are scraped from the public internet without the consent of rights holders - effectively meaning the original creators of this work lose control as to how their work is used.

In deployment, AI systems may recreate exact or close copies of copyrighted content. These outputs might enable 'copyright laundering', for example to reproduce unique art styles or remove copyleft protections from open-source software libraries.

It's uncertain how these legal issues will play out in the courts, as well as how wider solutions to this tension will be resolved by policymakers.

Worker exploitation

Training an AI often requires lots of human-annotated data. For example, content moderation systems might need thousands of examples of harmful content to train the model to identify what violates platform guidelines.

This work can repeatedly expose people to harmful content, damaging the mental health of labellers. Given the work is often outsourced to contractors in countries with lower health and safety standards, workers are also less likely to be able to get appropriate support to handle mental health problems arising from their work. This outsourced labour can also result in workers being underpaid, overworked or otherwise mistreated (such as being silenced through the use of NDAs or threats of being fired).

Anticipated risks

Bioterrorism

AI advances could worsen the risks of bioterrorism. Hendrycks et al. (2023) provide a useful introduction to this risk. The following is a lightly edited excerpt from their paper:

Bioengineered pandemics present a new threat. Biological agents, including viruses and bacteria, have caused some of the most devastating catastrophes in history. It’s believed the Black Death killed more humans than any other event in history, an astounding and awful 200 million, the equivalent to four billion deaths today. Engineered pandemics could be designed to be more lethal or easily transmissible than natural pandemics.

AIs could be used to expedite the discovery of new, more deadly chemical and biological weapons. In 2022, researchers took an AI system designed to create new drugs and tweaked it to reward, rather than penalize, toxicity. Within six hours, it generated 40,000 candidate chemical warfare agents. It designed not just known deadly chemicals including VX, but also novel molecules that may be deadlier than any chemical warfare agents discovered so far. In the field of biology, AIs have already surpassed human abilities in protein structure prediction and made contributions to synthesizing those proteins.

AIs compound the threat of bioengineered pandemics. AIs will increase the number of people who could commit acts of bioterrorism. General-purpose AIs like ChatGPT are capable of synthesizing expert knowledge about the deadliest known pathogens, such as influenza and smallpox, and providing step-by-step instructions about how a person could create them while evading safety protocols.

The exponential nature of biological threats means that a single attack could spread to the entire world before an effective defense could be mounted. Only 100 days after being sequenced, the omicron variant of COVID-19 had infected a quarter of the United States and half of Europe. Quarantines and lockdowns instituted to suppress the COVID-19 pandemic caused a global recession and still could not prevent the disease from killing millions worldwide.

Research into how much AI systems increase the potential for bioterrorism is ongoing. A recent study published by RAND found current safety-tuned LLMs did not substantially increase the risk over what was already present on the internet. However, a study by MIT and others found without safety-tuning (which can be removed from open models like Llama 2 for $200), LLMs enabled people to obtain all the key information to synthesise 1918 influenza in under an hour.

Disinformation

AI-boosted disinformation undermines societies’ ability to address catastrophes. Goldstein et al. (2023) identify three potential areas of change:

Actors: Language models could drive down the cost of running influence operations, placing them within reach of new actors and actor types.

Behavior: Influence operations with language models will become easier to scale, and tactics that are currently expensive (e.g., generating personalized content) may become cheaper. Language models may also enable new tactics to emerge—like real-time content generation in chatbots.

Content: Text creation tools powered by language models may generate more impactful or persuasive messaging compared to propagandists, especially those who lack requisite linguistic or cultural knowledge of their target. They may also make influence operations less discoverable, since they repeatedly create new content without needing to resort to copy-pasting and other noticeable time-saving behaviors.

Hendrycks et al. (2023) highlight additional aspects of risk from AI-boosted disinformation:

AIs can exploit users’ trust. Already, hundreds of thousands of people pay for chatbots marketed as lovers and friends, and one man’s suicide has been partially attributed to interactions with a chatbot. As AIs appear increasingly human-like, people will increasingly form relationships with them and grow to trust them. AIs that gather personal information through relationship-building or by accessing extensive personal data, such as a user’s email account or personal files, could leverage that information to enhance persuasion. Powerful actors that control those systems could exploit user trust by delivering personalized disinformation directly through people’s “friends.”

AIs could centralize control of trusted information. AIs could centralize the creation and dissemination of trusted information. Only a few actors have the technical skills and resources to develop cutting-edge AI systems, and they could use these AIs to spread their preferred narratives. Alternatively, if AIs are broadly accessible this could lead to widespread disinformation, with people retreating to trusting only a small handful of authoritative sources. In both scenarios, there would be fewer sources of trusted information and a small portion of society would control popular narratives.

Authoritarianism, Inequality, and Bad Value Lock-in

Dafoe (2020) describes ways AI could lead to power being concentrated and then misused. Let us consider these mechanisms (and an extra one—oligopolistic markets) in more detail:

• Global winner-take-all markets: Whoever leads in selling access to broadly capable AI systems may be able to offer many customers the best deal for a wide range of services. This could greatly concentrate wealth, which would incentivize authoritarian coups (while also making it easier to suppress democratic revolutions, as discussed below).
• Oligopolistic markets: In addition to global winner-take-all markets, there may be additional factors driving AIs to be controlled by a small number of people. Hendrycks et al. (2023) mention one aspect of that: “To operate effectively, AIs require a broad set of infrastructure components, which are not equally distributed, such as data centers, computing power, and big data.”
• Labor displacement: Historically, new technologies have often automated some jobs while creating new jobs. However, broadly capable AIs would be historically unprecedented. If AIs can do nearly every task at least as well as a human, this may leave few jobs left for humans—especially since AIs do not need to rest, can learn vast amounts of information, and can often complete tasks far more quickly and cheaply than humans.
• Authoritarian surveillance and control: AIs can be used to flag content for censorship, analyze dissidents’ activities, operate autonomous weapons, and persuade people. This could allow totalitarian governments to surveil and exert complete control over the population without the need to enlist millions of citizens to serve as willing government functionaries. Additionally, AIs could make totalitarian regimes much longer-lasting; a major way in which such regimes have been toppled previously is at moments of vulnerability like the death of a dictator, but AIs which would be hard to “kill” could provide much more continuity to leadership, providing few opportunities for reform. (Hendrycks et al., 2023).

Concentration of power may make it easier to permanently entrench certain values. Hendrycks et al. (2023) argue:

It’s dangerous to allow any set of values to become permanently entrenched in society. For example, AI systems have learned racist and sexist views, and once those views are learned, it can be difficult to fully remove them. In addition to problems we know exist in our society, there may be some we still do not. Just as we abhor some moral views widely held in the past, people in the future may want to move past moral views that we hold today, even those we currently see no problem with. For example, moral defects in AI systems would be even worse if AI systems had been trained in the 1960s, and many people at the time would have seen no problem with that.

War

Analysts have highlighted several pathways by which AI could increase the risk of war, including nuclear war.

AI could undermine nuclear deterrence. In a world with nuclear weapons, nuclear war is often thought to be prevented partly by nuclear deterrence: if a state launched a nuclear strike, it would risk being nuked in retaliation. However, advances in AI could undermine states’ retaliatory strike capabilities in various ways. States might use AI to…

• …locate nuclear-armed submarines, by using AI to analyze sensor data and perhaps improve other aspects of reconnaissance drone technology. This would be problematic, because states often view nuclear-armed submarines as their most resilient nuclear deterrent, due to the difficulty of locating them. However, the technical plausibility of this is debated.[1]
• …make mid-air adjustments to strikes against mobile missile launchers, e.g. through satellite image analysis. This would reduce the difficulty of destroying these weapons (which also have deterrent functions).
• …asymmetrically improve missile defense, which could neutralize rivals’ retaliatory strikes. However, missile defense has historically been highly difficult.
• …execute cyberattacks that disable rivals’ retaliatory capabilities.

AI could inadvertently escalate conflicts (video). While analysts consider it unlikely militaries would give AIs direct control over nuclear weapons, AI could escalate conflicts in other ways. Decision-makers might trust flawed recommendations about escalation, and lethal autonomous weapons could inadvertently initiate or expand violent conflicts.

AI could create other incentives for war. International relations scholars often find “large, rapid shifts in the distribution of power lead to war” (Powell, 2006). When faced with a rising rival, states may need to choose between (i) declaring war while the rival is relatively weak, or (ii) later being threatened by a more powerful rival. This situation incentivizes war. Above, we saw one way AI might contribute to a large, rapid shift in power: undermining nuclear deterrent capabilities. AI might also cause other destabilizing power shifts, such as accelerating economic and technological development in certain states.

Multi-agent failures

When people task powerful AI systems with high-stakes activities that involve strategically interacting with other AI systems, bargaining failures between AI systems could be catastrophic (Torges, 2021):

Transformative AI scenarios involving multiple systems (“multipolar scenarios”) pose unique existential risks resulting from their interactions. Bargaining failure between AI systems, i.e., cases where each actor ends up much worse off than they could have under a negotiated agreement, is one such risk. The worst cases could result in human extinction or even worse outcomes (Clifton 2019).[3]

As a prosaic example, consider a standoff between AI systems similar to the Cold War between the U.S. and the Soviet Union. If they failed to handle such a scenario well, they might cause nuclear war in the best case and far worse if technology has further advanced at that point.

Some might be optimistic that AIs will be so skilled at bargaining that they will avoid these failures. However, even perfectly skilled negotiators can end up with catastrophic negotiating outcomes (Fearon, 1995). One problem is that negotiators often have incentives to lie. This can cause rational negotiators to disbelieve information or threats from other parties—even when the information is true and the threats are sincere. Another problem is that negotiators may be unable to commit to following through on mutually beneficial deals. These problems may be addressed through verification of private information and mechanisms for making commitments. However, these mechanisms can be limited. For example, verification of private information may expose vulnerabilities, and commitment mechanisms may enable commitments to mutually harmful threats.

Unknown Risks

New technologies often pose risks that are hard to foresee. As Dafoe (2020) mentions, risks from the combustion engine have included "urban sprawl, blitzkrieg offensive warfare, strategic bombers, and climate change"—presumably a hard list to predict in the 1800s.

Despite this, we can still indirectly prepare for them. We can make institutions better at identifying and responding to new AI risks as they emerge. We can also improve our responsiveness to currently unknown risks, in part by remembering that we might not yet know all the risks.

Footnotes

1. Skeptics argue there are enormous challenges in using drones to reliably locate submarines, such as vast search areas, limitations of sensor technology, short battery lives, and low drone speeds. On the other hand, perhaps these problems could be overcome with technological advances (e.g. improved sensor data analysis) and innovative approaches (e.g. using relatively centralized recharging stations or surface-level solar panels to recharge drones).

2. This is different from the earlier concern about the lock-in of bad values. In an earlier section, we considered how concentration of power could lock in bad values. Here, the concern is that unconstrained competition could lock in bad values. (Of course, which values are bad is highly contested.)

3. [Footnote from the excerpt] Note that bargaining failure is not the only cause of catastrophic interactions. For instance, the interactions of Lethal Autonomous Weapon Systems might also be catastrophic.