This arXiv preprint introduces Chain-of-Thought Prompting prompting, a method in which a small number of demonstrations showing intermediate reasoning steps are supplied as exemplars in a prompt. The authors’ claim is that generating such a chain of thought — a series of intermediate reasoning steps — significantly improves a large language model’s ability to perform complex reasoning, and that this ability emerges naturally in sufficiently large models. The authors describe chain of thought prompting as a simple method.
The evaluation reported in the abstract covers three large language models across arithmetic, commonsense and symbolic reasoning tasks. The authors describe the empirical gains as striking and give one headline result: prompting a 540B-parameter language model with just eight chain-of-thought exemplars reaches state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even a finetuned GPT-3 with a verifier.
The paper is filed under Computation and Language (cs.CL) and Artificial Intelligence (cs.AI), was first submitted on 28 January 2022, and was last revised on 10 January 2023 as version 6. It carries the arXiv-issued DOI 10.48550/arXiv.2201.11903 and is made available under a Creative Commons Attribution 4.0 licence.
Key Points
- The paper introduces chain-of-thought prompting: providing a few demonstrations of intermediate reasoning steps as exemplars in the prompt, which the authors describe as a simple method.
- Its central claim is that generating a chain of thought significantly improves large language models’ ability to perform complex reasoning.
- The authors state that these reasoning abilities emerge naturally in sufficiently large language models, which ties the method’s effect to model scale rather than to the prompting format alone.
- Experiments on three large language models are reported to improve performance across a range of arithmetic, commonsense and symbolic reasoning tasks.
- The headline result is that a 540B-parameter model prompted with eight chain-of-thought exemplars reaches state-of-the-art accuracy on GSM8K, a benchmark of math word problems, surpassing a finetuned GPT-3 with a verifier — a comparison between a prompting-only method and a trained baseline.
Notes
The emergence claim is the paper’s own qualifier on its result: the abstract attributes the reasoning ability to models that are sufficiently large, so the reported gains are not presented as a property of the prompting format on its own. The preprint went through six versions between January 2022 and January 2023.