Type: url
Original: https://arxiv.org/pdf/2310.10996
Status: generated
Summary
The full text of the paper introducing ClarifyGPT, a framework that has an LLM detect ambiguous code-generation requirements through a code consistency check over sampled solutions, ask targeted clarifying questions generated by reasoning over the inconsistent solutions, and regenerate code from the refined requirement. With human participants it raises GPT-4’s Pass@1 on MBPP-sanitized from 70.96% to 80.80%, and with a proposed simulated-user method it improves GPT-4 and ChatGPT across four benchmarks from 68.02% to 75.75% and from 58.55% to 67.22%.