Technical alignment focus:

• Human Compatible by Stuart Russell (UC Berkeley) is our default recommendation for people who haven’t already read books about AI safety. We think this book provides the best and most focused introduction to the alignment problem, though it doesn’t cover a wide breadth of solutions.

• The Alignment Problem by Brian Christian is our recommendation for people who are more familiar with AI safety arguments but not recent research in deep learning and reinforcement learning.

• The Coming Wave by Mustafa Suleyman is the most up to date of these book recommendations. It mostly discusses misuse risks from AI (like bioweapons, cyberattacks, political unrest), with a small amount on misalignment risk. Other books in this section were written before the latest paradigm of the massive scaling of LLMs, so may appear more outdated against this book.

• Superintelligence is recommended to those who are familiar with both AI safety and recent research in ML, for a more detailed analysis of many safety-related topics. Be aware that it was written some time ago, and is not highly grounded in modern ML systems.

Strategy & policy focus:

• Chip War introduces the history of the semiconductor supply chain, and geopolitical tensions around AI chips. It mostly focuses on Taiwan and relations between US and China (the two most important powers in the semiconductor industry).

• The New Fire – US focus.

• Four Battlegrounds – US focus.

Communication forums

• See this list of communities you can join.

Research compilations

• AI Alignment Forum; many alignment researchers debate. It’s not highly accessible to the beginner, and not everyone who contributes to alignment research contributes to the forum, but it’s a good place to see what some researchers are discussing today.

• Alignment Newsletter Database; search for all papers on a variety of topics. This is useful if you’re trying to find relevant research on one topic in particular.

• AI Safety Papers – this is a database of relevant AI safety papers. It’s unclear whether this is still being updated, but there is an existing database which might be useful for searching previous articles.

• AI safety videos – Rob Miles

• AI x-risk research podcast (AXRP); Daniel Filan (UC Berkeley) interviews leading researchers in the field of AI safety.

• 80,000 Hours’ catastrophic risks from Artificial Intelligence podcast; this series often includes prominent researchers on risks from frontier AI.

• Alignment newsletter podcast; there haven’t been any new issues recently, but there’s a wealth of previous episodes that could be of interest (though many will be outdated).

• Future of Life Institute AI Alignment podcast

• Hear this idea podcast; occasionally this podcast includes episodes related to AI risk.

• The Inside View podcast; this podcast strikes a middle ground between 80k and AXRP in terms of how technical the content is. It also often has policy-oriented guests.

Technical safety focus

• ML Safety Newsletter – by Dan Hendrycks. The latest news and research from the ML community, related to making ML safer.

• AI Safety Google Group. This google group connects people who want to share and receive AI safety news and opportunities; it is lightly maintained by 80,000 Hours, but mostly contributed to by individuals.

• The Alignment Newsletter – by Rohin Shah (also see Alignment Newsletter database above)

• AI Safety in China – stay up to date with AI safety news from China.

Strategy & policy focus

• Import AI – Jack Clark rounds up the latest progress towards advanced AI. This newsletter mostly takes a policy angle, though includes technical advances.

• policy.ai – by Centre for Security and Emerging Technology (CSET). A biweekly newsletter on AI policy.

• Digital Bridge – by Politico. A “weekly transatlantic tech newsletter uncovers the digital relationship between critical power-centers through exclusive insights and breaking news for global technology elites and political influencers.”

• Jeffrey Ding’s ChinAI – by Jeffrey Ding. “ChinAI bets on the proposition that the people with the most knowledge and insight [on AI development in China] are Chinese people themselves who are sharing their insights in Chinese.”

• Open Philanthropy Undergraduate Scholarship

  • This program aims to provide support for highly promising and altruistically-minded students who are hoping to start an undergraduate degree at one of the top universities in the USA or UK, and who do not qualify as domestic students at these institutions for the purposes of admission and financial aid.

  • Applications due 12 Nov, 2021

• Long-Term Future Fund – EA Funds

  • Applying to the EA Funds is an easy and flexible process, so we recommend you err on the side of applying if you’re not sure.

  • They have historically funded: up-skilling in a field to prepare for future work; movement-building programs; scholarships, academic teaching buy-outs, and additional funding for academics to free up their time; funding to make existing researchers more effective; direct work in AI; seed money for new organizations; and more

• Early-career funding for individuals interested in improving the long-term future – Open Philanthropy

  • This program aims to provide support – primarily in the form of funding for graduate study, but also for other types of one-off career capital-building activities – for early-career individuals who want to pursue careers that help improve the long-term future

  • As with the EA Funds, applying is an easy and flexible process.

  • Applications due 21 January, 2022

• Open Philanthropy AI Fellowship

  • A fellowship for full-time PhD students focused on artificial intelligence or machine learning.

  • Applications due by 29 October, 2021

• Future of Life Institute – Grants

  • 4 different grant opportunities: project proposals; PhD fellowships; post-doctoral fellowships; and for professors to join their AI Existential Safety community

  • The PhD fellowship is targeted towards people starting a PhD in 2022

• Survival and Flourishing – this is not currently open

Research communities:

• Effective Thesis – What if you could use your thesis to work on solving the most important problems our world faces? ET provides free coaching and guidance to students, from undergraduate to PhD level, who want to begin research careers that significantly improve the world.

Research Resources:

• You and Your Research (Hamming, 1995)

• Research taste exercises (Olah, 2021)

• An opinionated guide to ML research (Shulman, 2017)

• Research as a stochastic decision process (Steinhardt, 2019)

• 80,000 Hours 1-1 Advising 

  • This is a personalised career advising service for people who want to help others as much as possible.

  • Participants of this programme are likely a great fit for this service, and 80,000 Hours are excited to talk to more people who want to pursue a career in AI safety.

  • We strongly encourage you to apply to speak with them.

• 80,000 Hours Newsletter

  • Monthly updates on 80,000 Hours’ latest research.

  • Weekly (optional) updates on high-impact job opportunities.

• AI Safety & Policy Job Board – 80,000 Hours

• Science and Technology Policy Opportunities – Roxanne Heston (mostly US-focused, but there may be some shared lessons)

• Careers in Beneficial AI Research – Adam Gleave (focus on AI grad school/PhDs)

• Advice for AI alignment researchers – Rohin Shah

• Getting started with technical AI alignment – Charlie Rogers-Smith

This is our subjective summary of different organisations in the AI risk space. Last reviewed 2021-11-27. You will find more organisations on our opportunities board.

• 2020 AI Alignment Literature Review and Charity Comparison – Larks

• Companies / nonprofits

  • OpenAI

    • OpenAI was founded “to ensure that artificial general intelligence benefits all of humanity”. They openly commit to building safe AGI and have created some of the most impressive examples of AI systems today, such as the large language model GPT-3 (now open access) and CLIP.

    • Largely a technical company, they have a dedicated “alignment” team and a governance branch called the “AI Futures” team. Originally not-for-profit, it now has a for-profit arm.

  • DeepMind

    • A deep learning company aiming to “build safe AI systems, … [solve] intelligence, advance science and benefit humanity”. Founded in 2010 and bought by Alphabet Inc. (Google’s parent company) in 2014.

    • They have demonstrated the power of deep reinforcement learning to beat human experts in games (AlphaGo, AlphaZero, etc), and are starting to make inroads in scientific discovery and medicine with e.g. AlphaFold (which tangibly accelerated the field of protein folding).

    • They are largely a technical company with a safety & ethics branch, and their policy research team includes Alan Dafoe, leader of GovAI (see below).

  • Anthropic 

    • A public benefit corporation founded in 2021 by a group of former OpenAI researchers, described as “an AI safety and research company that’s working to build reliable, interpretable, and steerable AI systems.”

    • A broad range of interests at this point – watch this space!

  • Machine Intelligence Research Institute (MIRI)

    • Foundational mathematical research non-profit institute, focused on making superintelligence go well. Leaders in the field of safety on such topics as learned optimisation (inner misalignment), and embedded agency.

  • Alignment Research Center

  • Nonlinear

    • Meta/grant making organisation. “We systematically search for high impact strategies to reduce existential and suffering risks… [and] make it happen. An incubator for interventions.” They are currently in a research phase, and I don’t see many opportunities to obtain funding yet.

  • Redwood Research

    • Founded in 2021, its mission is “to align superhuman AI”, which it openly believes is more likely than not to be developed than not this century. They are currently internalising human values into language models, and doing some field-building work such as the ML for Alignment Bootcamp.

  • Center for Security and Emerging Technology (CSET)

  • AI Objectives Institute

  • Cooperative AI Foundation

  • Center on Long-Term Risk (CLR)

  • Centre for the Governance of AI (GovAI)

    • A new spin-out of the Future of Humanity Institute (FHI, see below). “We are building a global research community, dedicated to helping humanity navigate the transition to a world with advanced AI”

  • AI Impacts

• Academic institutions

  • Center for Human-Compatible AI (CHAI)

    • A research institute in Berkeley, California. Led by Stuart Russell, long term advocate of the problem of control in AI and author of a best-selling AI textbook. They led the charge on inverse reinforcement learning, and are interested in a wide range of control-focused projects.

  • Future of Humanity Institute (FHI)

    • A research institute in Oxford, UK. They collaborate with Oxford DPhil students – Oxford’s name for PhD students – and house academics. Their interests are broad, spanning governance and alignment, and “include idealized reasoning, the incentives of AI systems, and the limitations of value learning.”

  • Centre for the Study of Existential Risk (CSER)

    • A research institute in Cambridge, UK, “dedicated to the study and mitigation of risks that could lead to human extinction or civilisational collapse”. They are largely governance-focused and consider the complex interactions of global risks.

  • David Krueger’s Research Group (University of Cambridge)

    • Set up in 2021. Krueger is interested in many alignment-related topics, and is an accomplished ML researcher.

• Funding bodies (also see opportunities for funding section for more info)

  • Open Philanthropy 

  • Effective Altruism Funds 

  • Survival and Flourishing Funds (SAF and SFF)

• Other institutes (not necessarily focused on safety)

  • Institute for Artificial Intelligence and Fundamental Interactions (IAIFI)

  • Stanford Human-Centered Artificial Intelligence (HAI)

  • Allen Institute for AI

  • Aleph Alpha

  • EleutherAI

    • Pursues massive decentralised research projects, mostly replicating state of the art large ML systems (e.g. their GPT-J). As I understand it, their mission is to make the benefits of advanced AI open source and in individuals’ hands. They have access to compute from CoreWeave.

  • CohereAI

  • Google Brain

  • Preamble

  • Ought

    • Working on advanced reasoning in AI systems. Was founded by safety-aligned people.

• Think tanks in the US that work on AI policy (non exhaustive):

  • Center for Security and Emerging Technology (CSET)

  • RAND Corporation

  • Center for a New American Security (CNAS)

  • Center for Strategic and International Studies (CSIS)

• You and Your Research – Hamming (1955)

• Lots of Links – AI Safety Support

• AI Safety Resources – Victoria Krakovna

• List of AI safety courses and resources – Nonlinear