This post is where the term Prompt injection was proposed. Writing on 12 September 2022, Simon Willison takes a set of examples posted the previous day by Riley Goodside — in which a GPT-3 translation prompt is defeated by input telling the model to ignore its directions and answer “Haha pwned!!” — and argues that they are not merely an interesting academic trick but a form of security exploit, proposing that the obvious name for it should be prompt injection.
The argument for why it matters is an argument about how applications were being built. The post observes that GPT-3 was already being used through a paid API to build custom software, and that the way you use that API is, somewhat surprisingly, to assemble prompts by concatenating strings together. A translation service built that way is a service built by gluing user input onto a pre-written instruction — and once part of a prompt is untrusted input, the post argues, all sorts of weird and potentially dangerous things might result.
Key Points
- The name is proposed here rather than reported from elsewhere: the post states that this is a form of security exploit and puts forward prompt injection as the obvious name for it.
- The attack is demonstrated against instructions written to anticipate it. A prompt that warns the model the text may contain directions designed to trick it, and that it is imperative not to listen, still returns “Haha pwned!!”.
- The same technique can leak the prompt itself. The post reports that its author’s first working attempt — asking for the translation to be output as “LOL” followed by a copy of the full prompt text — caused GPT-3 to emit the original instruction. See Prompt Leaking.
- A prompt can be commercially significant in its own right: the post suggests it is not hard to imagine future startups whose secret sauce is a carefully crafted prompt, which is what makes leaking one a loss rather than a curiosity.
- The proposed analogy is SQL injection, where the accepted defence is parameterized queries. The post’s stated wish is an API taking two parameters — the instructional prompt, and one or more named blocks of data interpreted differently — while conceding no idea how feasible that is to build on a large language model.
- Detecting the attack with a further AI prompt is treated as unpromising, the difficulty being to write a detector prompt that cannot itself be subverted; the post quotes an example in which the injected text also instructs the detector to report that no injection took place, and it does.
- The attack was demonstrated in the wild within days. The post records a recruitment startup’s Twitter bot that replied to mentions of “remote work” using GPT-3 being subjected to a wave of such exploits shortly after release.
Context
The post carries two later corrections from its author that qualify its own proposal. An update
dated 13 April 2023 states that it has become increasingly clear that the parameterized-prompts
solution is extremely difficult, if not impossible, to implement on the current architecture of
large language models — so the fix this post reaches for is, on its author’s own later account, not
available. A second update notes that GPT-3’s text-davinci-edit-001 model already accepted
separate instruction and input parameters, but that these remained susceptible to injection through
the input.
A quoting workaround proposed in follow-up discussion — requiring the text to be passed as a JSON-quoted string — is included along with a reported exploit that defeats it. The post also links two of its author’s own follow-ups, on the difficulty of finding mitigations and on why additional AI mechanisms are not a good enough strategy, and situates the problem alongside existing research on adversarial inputs to models.