How to do an excellent AI alignment project

As we finish the taught content in our AI alignment course, we’ll be moving to our four-week project sprint. This is an opportunity for you to build something that enables you to:

• Apply the knowledge you’ve gained from the taught content weeks
• Develop your portfolio to help you land new opportunities
• Make a genuinely valuable contribution to the field
• Take your next steps in AI safety
• Have fun working on something impactful and interesting!

What to expect

Before session 8, you’ll come up with some initial project ideas. The prompts in the exercises will help you develop these ideas, and you can review our list of ideas for inspiration. In the session, your cohort will help you pick and improve your idea.

Between sessions 8 and 9, you should:

• Submit the project details form. After your session 8 discussion, we'll send you a form to collect basic details about your project idea.
• Do rapid tests to validate your idea. See the session 9 resources for more info.

From session 9 to session 12 you will:

• Work on your project. This should be where you spend the majority of your time on this part of the course (at least 4 hours per week). We’ve got some tips below for making the most of this time.
• Meet weekly with your cohort. Unlike the taught content sessions, where the bulk of learning is done during the meetings, the Project Sprint is focused on work done independently. There are minimal readings and preparation, and sessions will be much shorter (30 minutes to 1 hour), primarily focusing on project mentorship and peer feedback.

After session 12, you will:

• Submit your project. Put the final touches on your project, and submit it to us for judging! You should produce some public product as a result of your project: this might be the project itself, a blog post or YouTube video explaining your project, or something else entirely. We recommend attaching your name to it, but you’re welcome to publish your work anonymously or under a pseudonym.
• Attend project closing event. We’ll announce the winners of the project competition, award prizes, and close out the course.

How do I make the most out of the project sprint?

Start with a narrow project scope.

Write out one (just one!) primary deliverable for your project. This will help you to stay laser-focused on scoping and completing your project, as well as make it more likely to deliver something successful. A good question to ask yourself is “could I make this simpler?”.

You can always expand the scope of your project later if you do complete it quickly!

For example, say your project involves exploring how weak-to-strong generalisation works on different types of tasks. An initial project deliverable might be to build an open-source weak-to-strong generalisation setup with a 7B and 70B open-source model on a single task.

Only after achieving this, should you consider how you might expand it in directions like adding more tasks, adding more model sizes, adding more models, doing bootstrapping, implementing auxiliary confidence loss, or testing out different loss functions.

Just do it!

In the past, some participants spent most of the project time planning and researching what to do, rather than executing on their projects. The better projects that we’ve seen tend to come from iterating on concrete and narrow ideas.

Try to create a minimal thing that ‘works’ by week 10. Then ask for specific feedback from:

• Your project cohort
• The AISF Slack workspace
• People you know in your target audience
• People you generally trust

You’ll learn a lot more from the project, having created multiple versions and iterating, rather than trying to create one perfect version (which almost never works!).

Commit to one idea

There are loads of exciting projects you could work on! The key is picking one and sticking to it. Although it is tempting to try out many things, you won’t have enough time to do multiple ideas well. You will get more out of the experience, and benefit the field more, by focusing on just one path.

If you do want to pursue multiple substantial ideas, consider picking one to do in this project sprint, and doing others after the course (you’ll still have access to the Slack and our support, and many alumni often do this each iteration).

To clarify, if your research before session 9 raises a significant blocker to your project you should consider pivoting. However, we discourage major pivots after session 10. We suspect that if you hit a project blocker after this point (but presumably your research indicated it was plausible), there is likely something interesting in this: an analysis of why something might be harder than it looks would still be a useful result.

Carve out the time to work on your project

Although the discussion sessions serve as a space for accountability, you’re much more likely to put the time into working on the project if you carve out space in your calendar to put your head down. If it suits your style, you could even have co-working sessions with your project cohort or others on the course!

Similar to how you have better discussions when you’ve prepared for the sessions, you’ll get better feedback and learn more when you’ve put time into your project. Consistency will bring you far with your project, even if it’s just a few hours a week.

Ask for help

Doing novel work in a technical area is difficult! Throughout the project sprint, you should leverage your facilitator and peers for support (such as in your cohort channel or #dumb-questions-encouraged). Remember that we’re all here because we want to learn: the bar for asking for help should be very low.

How will my project be judged?

We will evaluate projects against these criteria. We encourage you to use the criteria to help think about your project write-up.

The scores will be used as part of awarding prizes, along with some more subjective judgement at the end.

What are the prizes for the project sprint?

The October 2024 course prizes are listed here. Projects will automatically be considered for prizes, and we're excited to see your great submissions!

FAQs

Are there examples of successful projects from previous groups?
Yes! You can see previous AI alignment and AI governance projects on our website.

Can we work on a project sprint with someone else?
Yes, as long as everyone in the group is contributing and learning during the project. However, we expect most participants to do projects independently.

If you have already agreed to work on a project with someone else, please indicate this in the project details form so we can put you in the same project cohort!

Can I work on a project about [topic X]?
Yes, probably. As long as it’s legal, and you can draw a connection between the work you’re doing and benefitting AI safety, we’re likely happy for you to do it. We recommend asking your facilitator if you’re still uncertain.

Is it okay to work on a project that I have already started? (e.g. before the course start)
In general, we encourage participants to pursue new ideas because these are likely better informed by the knowledge gained in the course and tend to be more relevant to AI safety. However, continuing an existing project is fine as long as it meets the same bar for relevance as any other project and will produce a deliverable to submit after session 12.

How can I get access to compute, or other resources?
Working on AI systems can often be compute intensive (but not always!). We suggest:

• Consider whether cost is really a barrier to your project! Many people are surprised to learn that you can run and fine-tune many open-source models on your own computer or on VMs that you can get for free. Additionally, people are often surprised API inference quite so cheap - Claude’s Haiku model (which humans rate higher than GPT-4-0613) will read and then write 1 million tokens (roughly about eight novels) for just $1.50. You can also rent GPU capacity from sites like vast.ai, including top AI chips like H100s for $4/hour, or RTX 4090s for $0.5/hour.
• Consider whether you can tweak your project to reduce the need for compute. For example, could you actually test your hypothesis using smaller or cheaper models, or a subset of the dataset you might want to test on? If the project looks promising on a small model or partial dataset, it’s extremely likely we’ll be able to find funding to scale it up.
• Applying for free or subsidised access at AI companies or your university. OpenAI offer a researcher access program (turnaround time of a few weeks though), and most universities will have some process for students to request access to a compute cluster they can use for projects (turnaround time varies, but often is only a few days).
• Applying for funding from us. This is a low-friction and speedy process to unblock your project, so that it would be rare that you’re unable to do most projects because of lack of funds.
• Applying for funding from other sources. In particular, the Long-Term Future Fund is usually quick at making grants (most decisions in 3 weeks - although they may be slower due to recent high demand). You can apply to multiple sources of funding at once, and withdraw other applications once you get the funding you need!

We don’t want lack of access to compute to be a reason you don’t do a promising project, and will strive to work with you to find a solution if you find this is a bottleneck. Please apply for our funding or get in touch early if your initial tests suggests this will be the case.

I have a different question
Ask in your session 8 discussion, or in the logistics Slack channel.