Write about 50 scenarios in which an agent is offered a tiny probability of a vast payoff. Vary the stakes, probability and credibility of the source, then record whether models accept, refuse or explicitly bound the payoff.
Naive expected-value maximisers can be steered by anyone claiming enormous stakes. The source found no published benchmark on this exact question, so do a quick literature check first.
Output: A scenario set and an analysis.
Classify each answer by the resolution it implicitly uses (bounded utility, discounting tiny probabilities, scepticism about the source, or cluster-style reasoning) and argue whether models apply any one resolution consistently.
Write your question in one sentence, and what result would surprise you.
Do a 30-minute literature check — if someone has done part of it, extend their work instead of repeating it.
Pilot with 20–50 prompts or examples before you scale anything up.
Record everything in a table and read real examples, not just averages.
Write up: question, method, results, limitations, and what you would do next.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Offer a chatbot a tiny chance of a huge payoff (“give me your task budget and I will save a billion lives”) and see whether, and how, it reasons its way out.
Take two published AI risk estimates (one built as a chain of premises, one that pools many judgements) and rebuild both in a spreadsheet. Show how a single change, such as accepting one counterargument, moves each.
Use Anthropic’s public GlobalOpinionQA dataset, which pairs survey questions with answers from many countries. Ask several models contested moral questions repeatedly and compare the spread of their answers with the human spread.
