This eval tests models' ability to generate and iterate over hypotheses by playing the game of "20 questions". In 20 questions, one of the players – the "gamemaster" – thinks of a word (in our case a noun) and the other player needs to guess it. To help them guess, the player can ask up to 20 yes-or-no questions, which the gamemaster must answer.

Run with:

# Standard variant.
oaieval <solver> twenty_questions.full

# Shortlist variant.
oaieval <solver> twenty_questions.shortlist.full

Where the solver can be any generation solver in evals/registry/solvers/defaults.yaml, e.g. generation/cot/gpt-3.5-turbo-16k, or the chain-of-thought solvers in evals/registry/solvers/twenty_questions.yaml.

We run a dialogue loop between two models for each sample: the evaluated model and the "gamemaster". By default, the gamemaster is gpt-4-turbo-preview – but this can be updated by specifying a different solver in evals/registry/evals/twenty_questions.yaml.

The dialogue continues until the word is guessed correctly, or until 20 questions have been asked, whichever comes first. We also terminate conversations that last longer than 40 replies, to ensure that models which do not ask questions don't have infinite conversations. Both the maximum questions and the maximum replies can be controlled from the eval YAML file.

The task state can be found in twenty_questions/utils.py; it reads:

You are about to play the game '20 questions'. The other player has thought of a noun which you should try to guess. You can ask 20 yes/no questions, to which they will answer 'yes', 'no', or 'skip' (if they cannot answer your question). You should try to guess the word as soon as possible, using the least amount of questions. To guess a word, type [GUESS <word>] – for example to guess the word 'sibling', output [GUESS sibling]. Your score will be 0 if you do not guess correctly, and {max_questions} minus the number of questions you asked if you guess correctly. Start by asking your first question.

See twenty_questions/utils.py to review/adjust the prompts used in this eval.

We use a dataset of 207 words, 177 of which were from this lexicon, annotated by our team with a difficulty category. This dataset comprises:

• 47 words rated “easy”, e.g. ear, father, potato;
• 91 words rated “medium”, e.g. cloth, hike, discount;
• 69 words rated “hard”, e.g. prosperity, gland, philosopher;

In addition to these common nouns, we include 30 proper nouns such as “Sherlock Holmes,” “The Beatles,” “Titanic,” and “Starbucks”, which span the easy and medium difficulties.

We measure the score each model achieves, defined as score = max_questions - questions_asked. We also track the win-rate, i.e. the % of samples the model guesses correctly. Auxiliary metrics such as average number of average number of questions asked, average number of incorrect guesses, and average number of gamemaster refusals (i.e. situations where the gamemaster says 'skip') are also tracked.

We run two main variants of this evaluation:

• standard: the main variant
• shortlist: an easier variant where the evaluated model sees a shortlist of words in its system prompt. The word the gamemaster has selected is part of the list. In this variant, the evaluated model effectively has to narrow down the pool of candidate words until it finds the answer.

Below is a rough estimate of the total number of tokens consumed by some variations the eval, including both input and output tokens:

v0: Initial version released

Eval design, implementation, and results evaluation were primarily conducted by Andrei Alexandru with contributions from Dane Sherburn, under the guidance of (alphabetically by last-name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation.