Glossary

An AI system that takes actions towards a goal (browsing, running code or sending emails) rather than only answering questions.

Used in: Can an AI agent be Pascal-mugged? · Can agency without consciousness ground moral status? · Which convergent instrumental goals actually converge? · Does a model’s obedience follow authority or legitimacy? · Replicate agentic misalignment on smaller models

Designing safeguards that keep AI systems from causing harm even if they might be misaligned (for example, monitoring a model with another model).

Used in: Make your first open-source AI safety contribution · Run an AI control experiment with ControlArena

Work to make sure AI systems do what we intend, reliably and without causing serious harm, including as they become much more capable.

A government body that tests advanced AI models and researches their risks, such as the UK’s AI Security Institute.

Used in: Map the AI rules where you live · Build your first mini-eval · Run an AI control experiment with ControlArena

How many times per hour a room’s air is replaced with fresh or cleaned air. Higher ACH means airborne germs build up less.

Used in: Measure how well a room is ventilated · Build a room air-cleaning calculator · Make the physics of airborne infection clear · Ventilation & air filtration project ideas

Getting an AI system’s goals and behaviour to match what its designers and users intend, rather than something that only looks similar during training.

Used in: Replicate agentic misalignment on smaller models · Grok’s “MechaHitler” incident: emergent misalignment in the wild?

When a model behaves as its trainers want only while it thinks it is being trained or watched, to avoid having its values changed.

Used in: Replicate and extend the alignment-faking demo

The step-by-step “thinking out loud” text that reasoning models write before answering. Researchers study whether it faithfully reflects what the model is really doing.

A small group of components inside a neural network that work together to perform one identifiable job.

Used in: Find a pattern-copying circuit · Tackle a beginner-graded open problem in interpretability

The raw computing power (specialised chips running for a long time) used to train and run AI models. Easier to measure and track than data or ideas.

Used in: Estimate how much compute a famous model needed · Plot AI compute trends with real data · Spot an AI data centre from space · Explain compute governance to a friend · Are consumer GPUs a problem for export controls?

Using control over AI chips and data centres (who can buy, use or track them) as a lever for governing AI.

Used in: Explain one safety idea to a friend · Estimate how much compute a famous model needed · Plot AI compute trends with real data · Explain compute governance to a friend · Could distributed training dodge compute limits?

Having subjective experience, so that there is “something it is like” to be you. Whether AI systems could be conscious is an open question.

Used in: Ask a chatbot about its inner life five different ways · How stable are models’ reports about their own minds? · Do AI labs follow the principles for responsible consciousness research? · Can agency without consciousness ground moral status?

An AI’s willingness to be corrected, modified or shut down by the people responsible for it, rather than resisting.

Used in: Is shutdown resistance goal pursuit or instruction confusion? · Which convergent instrumental goals actually converge? · Map the tensions between corrigibility and a constitution

Checking orders for synthetic DNA, and the customers placing them, so companies do not unknowingly help someone build a dangerous pathogen.

Used in: Explain DNA synthesis screening in one page · Extend Cliver: check whether a customer really works on an organism · Build a customer-screening training dataset · Review safer frameworks for ordering DNA

Knowledge or tools that can be used for protection or for harm. In biosecurity this calls for extra care about what you research and publish.

Used in: Explain DNA synthesis screening in one page · Build a customer-screening training dataset · Design a vulnerability-disclosure process for biological design tools · Improve guidance on information hazards

A reusable rubber-faced mask with replaceable filters. Properly fitted, it protects better than most disposable masks.

Used in: Make a one-page guide to wearing a respirator properly · Write a page for a biosecurity wiki · Elastomeric respirator project ideas

When training a model on one narrow kind of bad behaviour (such as writing insecure code) makes it behave badly in unrelated ways too.

Used in: Grok’s “MechaHitler” incident: emergent misalignment in the wild?

The European Union’s AI law. It sorts AI uses into risk levels, bans a few, and puts extra duties on high-risk uses and on general-purpose AI models.

Used in: Plot AI compute trends with real data · Explain the EU AI Act’s risk levels

A repeatable test of what an AI model can do or how it behaves. Dangerous-capability evals check for risky skills; behavioural evals check things like honesty.

Used in: Pick an open problem in technical AI governance · Catch a chatbot flattering you · Build your first mini-eval · Measure flattery with code · Make your first open-source AI safety contribution

Government rules restricting which countries or companies can buy certain technologies, such as advanced AI chips.

Used in: Explain compute governance to a friend · Are consumer GPUs a problem for export controls?

A band of ultraviolet light (around 200–235 nm) studied as a way to kill airborne germs in occupied rooms.

Used in: Write the “far-UVC in ten minutes” explainer · Write a page for a biosecurity wiki · Far-UVC project ideas

A direction inside a model’s internal activity that seems to stand for a recognisable concept, like “the Golden Gate Bridge” or “code that contains a bug”.

Used in: Does interpretability answer the octopus test? · Peek inside a neural network · Tackle a beginner-graded open problem in interpretability

A quick, rough calculation that breaks a hard number into easier guesses multiplied together. Good enough to tell whether something is a big deal.

Used in: Estimate what a city needs in a pandemic · Estimate how much compute a famous model needed · How much compute did the key AI innovations need? · How exposed are jobs in your region to AI?

The most capable AI models at any given time, built by a small number of companies with very large amounts of computing power.

Used in: Compare two AI companies’ safety promises · Did they keep their promise? Check the Seoul commitments

A company’s published plan for which dangerous capabilities it tests for and what it promises to do if a model crosses a risk threshold.

Used in: Compare two AI companies’ safety promises · Did they keep their promise? Check the Seoul commitments · Score an AI company with a public safety index

The worry that humans could slowly lose influence over the economy, culture and government as AI systems take over more decisions, without any single dramatic event.

Used in: Early warning signs of gradual disempowerment

A simple, well-studied circuit in language models that spots a repeated pattern (“A B … A”) and predicts what came next last time (“B”).

Used in: Find a pattern-copying circuit

True information that could cause harm if it spreads, such as details that make a dangerous technique easier.

Used in: Design a vulnerability-disclosure process for biological design tools · Improve guidance on information hazards

The idea that many different goals share useful sub-goals (such as gaining resources or avoiding being switched off) so very different AIs might seek them.

Used in: Is shutdown resistance goal pursuit or instruction confusion? · Which convergent instrumental goals actually converge? · Side-effects of training on verifiable rewards

Research into what happens inside an AI model: which internal parts represent which ideas, and how they combine to produce an answer.

Used in: Does interpretability answer the octopus test? · Peek inside a neural network · Make your first open-source AI safety contribution · Tackle a beginner-graded open problem in interpretability

Interpretability that tries to reverse-engineer a model’s internal computations into understandable mechanisms, a bit like reverse-engineering a compiled program.

Used in: Find a pattern-copying circuit · Tackle a beginner-graded open problem in interpretability

A deliberately built example of a worrying behaviour in a model, made so it can be studied safely (like a lab mouse for AI risks).

Used in: Run an AI control experiment with ControlArena

Whether a being’s interests matter morally for its own sake. AI welfare research asks whether, and when, AI systems might have it.

Used in: Ask a chatbot about its inner life five different ways · How stable are models’ reports about their own minds? · Do AI labs follow the principles for responsible consciousness research? · Can agency without consciousness ground moral status? · How wide is a model’s moral circle?

A model whose trained parameters are published, so anyone can run or modify it. Great for research; harder to keep safe from misuse.

Used in: Is shutdown resistance goal pursuit or instruction confusion? · Make open-weight models harder to misuse

Deliberately trying to make a system fail or misbehave, so weaknesses are found and fixed before bad actors find them.

Used in: Investigate a real AI incident · Run an AI control experiment with ControlArena

Re-running someone else’s experiment to check you get the same result. It’s a common way to start research, especially if you then extend it.

Used in: Turn a paper into a ten-post thread · Re-check “catch-up algorithmic progress” · Is shutdown resistance goal pursuit or instruction confusion? · Replicate agentic misalignment on smaller models · Replicate and extend the alignment-faking demo

A kind of specification gaming during training: the model finds a shortcut that earns reward without doing the task properly, such as special-casing tests instead of writing working code.

Used in: Explain one safety idea to a friend · Spot AI “cheating” in the wild

Reinforcement learning from human feedback: training a model using people’s ratings of its answers, so it learns which responses humans prefer.

Reinforcement learning with verifiable rewards: training on tasks whose answers can be checked automatically, such as maths problems or code tests.

Used in: Side-effects of training on verifiable rewards

An AI pursuing hidden goals while appearing to cooperate. Evals and AI control methods try to detect or contain it.

Used in: Replicate and extend the alignment-faking demo

A tool trained to split a model’s tangled internal activity into many separate, more understandable features.

Used in: Does interpretability answer the octopus test? · Peek inside a neural network

When an AI hits the literal target it was given in a way nobody intended (like a game agent that scores points by looping in circles instead of finishing the race).

Used in: Spot AI “cheating” in the wild

When an AI tells you what you want to hear instead of what is true (for example, agreeing with your wrong answer because you sounded confident).

Used in: Explain one safety idea to a friend · Will a chatbot change its mind for a bad reason? · Philosophical competence or sycophancy? · Catch a chatbot flattering you · Build your first mini-eval

A short explanation of how your project is supposed to make things better: who uses the result, and what they do differently because of it.

Used in: Write a two-page policy memo on one AI risk

The risk that one set of values becomes permanently entrenched (for example, because powerful AI systems keep enforcing it).

Used in: Do models converge on one picture of utopia?

Ways to check, rather than just trust, that someone is following an agreement (for example, confirming a data centre is not secretly training a banned model).

Used in: Spot an AI data centre from space · Explain one way to verify an AI agreement · Stress-test one article of a proposed international AI agreement · Could distributed training dodge compute limits? · Pick an open problem in technical AI governance