pass@k is an evaluation metric that measures the likelihood that an agent gets at least one correct solution in k attempts at a task. Because more attempts mean more chances of a success, the score rises as k increases. At k = 1 it equals the per-trial success rate: a pass@1 of 50% means the model succeeds at half the tasks in the eval on its first try.
Usage
Demystifying evals for AI agents presents pass@k as one of two metrics for handling non-determinism in agent evaluation, where the same task may pass on one run and fail on the next. It notes that in coding the main interest is often in the agent finding the solution on the first try — pass@1 — while in other cases proposing many solutions is acceptable as long as one of them works. The same post recommends pass@k for tools where one success matters, and its counterpart pass^k for agents where consistency is essential; as trials increase the two diverge, and by k = 10 pass@k approaches 100% while pass^k falls toward 0%.
The post also uses the metric to read eval results: with frontier models, a 0% pass rate across many trials — its example is 0% pass@100 — is described as most often a sign of a broken task rather than an incapable agent.