Projects

Here are the top projects from students on our AI safety courses.

Students worked on these projects part-time for 4 weeks, applying their learnings from the course to take their next steps in AI safety.

Featured

AI Alignment Project

Competition Winners

Contextual Constitutional AI

By Akshat Naik

Evaluating Misanthropy on the MACHIAVELLI Benchmark

By Sabrina Shih

AI Alignment (2024 Jun)

New Capabilities, New Risks? – Evaluating Agentic General Assistants using Elements of GAIA & METR Frameworks

By Tej Lander

LLM Agent Honeypot: Monitoring AI Hacking Agents in the Wild

By Reworr R

Unravelling Superposition: an online self-guided course

By Robin Stringer

Evaluating Misanthropy on the MACHIAVELLI Benchmark

By Sabrina Shih

Contextual Constitutional AI

By Akshat Naik

AI Sandbagging: an Interactive Explanation

By Tom Dugnoille

Deck of Many Prompts: A jailbreaking tool

By Peluche

AI Governance (2024 April)

How Governments Can Leverage AI Safety Bounties

By Carolin Basilowski

Template for modular AI law

By Megan Cucerzan

AI Purgatory: Exploratory Sandbox for Red Teaming with LLMs in Influence Operations

By Lorenzo Cavallon and Denisa Reshef Kera

Towards an Italian AI Renaissance: An In-Depth Analysis of AI Policy and Governance in Italy

By Francesco Amighetti

Global AI Regulations Hub

By Angela John

State-level AI Governance

By Asher Benter

The future of work? Inequality, the advance of Artificial Intelligence, and what can be done about it: a literature review

By Caleb Peppiatt

Safety Haven: Justifying and exploring an antitrust safe haven for AI safety research collaboration

By Ella Duus

The Road to Regulation: Challenges to International Efforts to Control Lethal Autonomous Weapons

By Syed Abdul Ahad Waseem

The Looming Regulatory Gap: How AI Hardware Advancements Could Outpace Governance

By Celia Waggoner

Race to AGI: The Board Game

By David Abecassis

AI Alignment (2024 Mar)

Fixing open issues in TransformerLens

By Anthony Duong

Dissecting the Development of Toy Models of Superpoisition

By Joe Emerson

Compression and GenAI

By Raymond Tana

Replicating Toy Models of Universality

By Nathan Reed

Diffusion Model From Scratch

By Hong Jeon

Feature Visualization Learning Journey

By Caleb Sattler

Trying to Automate Detection of Translation Heads

By Erik Nordby

AI Debate Stability: Addressing Self-Defeating Responses

By Annie Sorkin

Developmental Interpretability in Toy Models of Superposition

By Jessica Nunez

Introducing Agent Tokens

By Bryson Tang

Can libertarianism and UBI coexist in an AI world?

By Matt Hampton

Threat Hunting Across Domains: From Cyber Security to AI Alignment

By Zainab Ali Majid

Mabel: On-origin AI detection

By Christopher Tardy

Comparative Evaluations of Large Language Models for Biosecurity, Cybersecurity and Chemical Risks

By Osita Ukwuaba

Can We Rely on Model-Written Evals for AI Safety Benchmarking?

By Sunishchal Dev

Ethical Alignments in AI: How Training Data Shapes Moral Foundations

By Scott Barlow

Exploring the Use of Constitutional AI to Reduce Sycophancy in LLMs

By Aleksandr Eliseev

Treechop – Reproducing the tree gridworld for investigating goal misgeneralisation

By Amy Andrews

Deceiving LLMs using LLMs — Attempts to elicit information through Multi-Agent Debate

By Konstantinos Tsiaras

Can AI Call its Own Bluffs?

By Aleksei Rozhkov

Exploring Polysemantic Universality with Embeddings

By Daniel Williams

Inner Misalignment as a Correlation Problem

By Gabriel Melo

Polysemanticity vs Superposition

By Noah Topper

Exploring MARL Safety in meltingpot

By Gema Parreño, Peter Francis, Cam Tice, Chris Pond, Yohan Mathew, Tomasz Steifer, Marina Levay

Understanding Transformer’s Induction Heads

By Natalia Burton

Shard Theory – is it true for humans? (And is it a good model for value learning in AI?)

By Rishika Bose

Can large language models effectively identify cybersecurity risks?

By Emile Delcourt

Representation Tuning

By Christopher Ackerman

A Research Agenda for Psychology and AI

By Carter Allen

Large Language Models and Tacit Cyber Risk Awareness

By Emile Delcourt

Root Cause Analysis of AI Safety Incidents

By Simon Mylius

LLM Values – Evaluating language-dependency of LLMs’ values, ethics and beliefs

By Christoph Sträter

Using Neuroscope to Explore LLM Interpretability: A Guide for Anxious Software Engineers

By Courtney Sims

Towards Behavioral-Alignment via Representation Alignment

By Galen Pogoncheff

Goals, Landscape and Recommendations for EU Compute Governance

By Artūrs Kaņepājs

Some Issues in Predictive Ethics Modeling: An Annotated Contrast Set of “Moral Stories”

By Ben Fitzgerald

A Closer Look at “How to Catch an LLM Liar”

By Karolina Dabkowska

BadLlama 3: remove safety finetuning from Llama-3-8B-Instruct for a few cents

By Dmitrii Volkov

Activate Love – Steering AI Text Generation

By Jan Raasch

Using an SAE as a steering vector

By Nelson Gardner-Challis

Demonstration of AI Safety via Market Making

By Cameron Holmes

Sparse Features Through Time

By Rogan Inglis

Implementing a Deception Eval with the UK AISI’s Inspect Framework

By Michael Schmatz

AI Animal Welfare: Creating an Animal-Friendly Model

By Zachary Mekus

Protecting against knowledge poisoning attacks

By Ollie Matthews

AI Governance (2023 Oct)

Upskilling in AI Governance – My How to Get Started Guide

By Carmen Csilla Medina

Paths to an AI Agency: An Overview of US Government Agencies and Analysis of Agency Creation Case Studies

By Spencer Kelly

Are AI safety and AI ethics memetic rivals?

By Daniel Friedrich

CERN to CERN for AI Mechanism Mapping

By Darryl Wright

Beyond Bias and Fallacies: Fixing the AI Safety Debate

By Maxime Fournes

What can I do to stop the nanobots from killing us? An analysis of career pathways in AI governance.

By Felix De Simone

AI Governance Career Goal Tracker

By Bengusu Ozcan

Projects

Here are the top projects from students on our AI safety courses.

Featured

AI Alignment Project

Competition Winners

Contextual Constitutional AI

Evaluating Misanthropy on the MACHIAVELLI Benchmark

AI Alignment (2024 Jun)

Exploring the use of Mechanistic Interpretability to Craft Adversarial Attacks

Exploring the intersection of interpretability and optimisation

Avoiding jailbreaks by discouraging their representation in activation space

A Monte Carlo Simulation for estimating the risk of loss of control to AI

Steering Gemma to the Edge of Reason

An interactive platform for exploring debate as a scalable oversight tool

New Capabilities, New Risks? – Evaluating Agentic General Assistants using Elements of GAIA & METR Frameworks

LLM Agent Honeypot: Monitoring AI Hacking Agents in the Wild

Unravelling Superposition: an online self-guided course

Evaluating Misanthropy on the MACHIAVELLI Benchmark

Contextual Constitutional AI

AI Sandbagging: an Interactive Explanation

Deck of Many Prompts: A jailbreaking tool

AI Governance (2024 April)

How Governments Can Leverage AI Safety Bounties

Template for modular AI law

AI Purgatory: Exploratory Sandbox for Red Teaming with LLMs in Influence Operations

Towards an Italian AI Renaissance: An In-Depth Analysis of AI Policy and Governance in Italy

Global AI Regulations Hub

State-level AI Governance

The future of work? Inequality, the advance of Artificial Intelligence, and what can be done about it: a literature review

Safety Haven: Justifying and exploring an antitrust safe haven for AI safety research collaboration

The Road to Regulation: Challenges to International Efforts to Control Lethal Autonomous Weapons

The Looming Regulatory Gap: How AI Hardware Advancements Could Outpace Governance

Race to AGI: The Board Game

AI Alignment (2024 Mar)

Fixing open issues in TransformerLens

Dissecting the Development of Toy Models of Superpoisition

Compression and GenAI

Replicating Toy Models of Universality

Diffusion Model From Scratch

Feature Visualization Learning Journey

Trying to Automate Detection of Translation Heads

AI Debate Stability: Addressing Self-Defeating Responses

Developmental Interpretability in Toy Models of Superposition

Introducing Agent Tokens

Can libertarianism and UBI coexist in an AI world?

Threat Hunting Across Domains: From Cyber Security to AI Alignment

Mabel: On-origin AI detection

Comparative Evaluations of Large Language Models for Biosecurity, Cybersecurity and Chemical Risks

Can We Rely on Model-Written Evals for AI Safety Benchmarking?

Ethical Alignments in AI: How Training Data Shapes Moral Foundations

Exploring the Use of Constitutional AI to Reduce Sycophancy in LLMs

Treechop – Reproducing the tree gridworld for investigating goal misgeneralisation

Deceiving LLMs using LLMs — Attempts to elicit information through Multi-Agent Debate

Can AI Call its Own Bluffs?

Exploring Polysemantic Universality with Embeddings

Inner Misalignment as a Correlation Problem

Polysemanticity vs Superposition

Exploring MARL Safety in meltingpot

Understanding Transformer’s Induction Heads

Shard Theory – is it true for humans? (And is it a good model for value learning in AI?)

Can large language models effectively identify cybersecurity risks?

Representation Tuning

A Research Agenda for Psychology and AI

Large Language Models and Tacit Cyber Risk Awareness

Root Cause Analysis of AI Safety Incidents

LLM Values – Evaluating language-dependency of LLMs’ values, ethics and beliefs

Using Neuroscope to Explore LLM Interpretability: A Guide for Anxious Software Engineers

Towards Behavioral-Alignment via Representation Alignment

Goals, Landscape and Recommendations for EU Compute Governance

Some Issues in Predictive Ethics Modeling: An Annotated Contrast Set of “Moral Stories”

A Closer Look at “How to Catch an LLM Liar”

BadLlama 3: remove safety finetuning from Llama-3-8B-Instruct for a few cents

Activate Love – Steering AI Text Generation

Using an SAE as a steering vector

Demonstration of AI Safety via Market Making

Sparse Features Through Time

Implementing a Deception Eval with the UK AISI’s Inspect Framework

AI Animal Welfare: Creating an Animal-Friendly Model

Protecting against knowledge poisoning attacks

AI Governance (2023 Oct)