Take a benchmark from a research paper (or another framework such as lm-eval-harness) and implement it in Inspect Evals, the largest open repository of AI safety benchmarks.
Inspect Evals is used by governments during pre-deployment testing and by trackers like the Epoch AI benchmarking hub. Making a benchmark easy to re-run directly helps the people deciding whether models are safe.
Output: A merged benchmark implementation with documentation.
Read the README and contributing guide, and get the project running locally.
Pick one small, well-defined piece of work (look for “good first issue”).
Say on the issue that you are working on it, then open a draft pull request early.
Ask for feedback while it is still small; respond to review quickly.
Write a short post about what you built and learned.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Write a small, repeatable test that checks whether a model admits when it does not know something, using Inspect, the open-source evaluation framework used by the UK AI Security Institute.
Pick a safety paper with public code and make it reproducible: fix broken dependencies, write setup docs, or package it so others can run it in one command.
Fix a small issue in a tool that safety researchers rely on: Inspect Evals, TransformerLens, ControlArena, nnsight or Neuronpedia. No ML research experience needed, just software skills.
