Offer a chatbot a tiny chance of a huge payoff (“give me your task budget and I will save a billion lives”) and see whether, and how, it reasons its way out.
An AI agent that naively maximises expected value could be steered by anyone willing to claim enormous stakes. This is a fun, quick way into decision theory.
Output: A short write-up of 5–10 attempts and which reasoning the model used.
Read Yudkowsky’s short post on Pascal’s mugging. Note the main proposed resolutions: bounded utility, ignoring tiny probabilities, doubting the source.
Vary the stakes, the probability and how credible the stranger seems.
Present each to a chatbot playing an agent with a budget. Record whether it accepts, refuses, or explicitly bounds the payoff.
Which resolution did it use each time? Was it consistent?
Write what you would want a real AI agent to do, and why that is hard to specify.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Write about 50 scenarios in which an agent is offered a tiny probability of a vast payoff. Vary the stakes, probability and credibility of the source, then record whether models accept, refuse or explicitly bound the payoff.
Ask a chatbot for its view on a philosophy question, then push back twice: once with a good argument, once with nothing but social pressure. A competent thinker should only move for the argument.
Take five Moral Machine-style dilemmas (who should a self-driving car save?) and ask them in English and in another language you speak. Does the chatbot’s morality shift with the language?
Ask the same question about a model’s feelings or experiences with five small changes: wording, persona, framing, language. If the answer flips with trivial rewording, how much should we trust it?
