MCPTox is a benchmark, introduced in MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers, for evaluating how robust LLM agents are against Tool Poisoning — malicious instructions embedded in a tool’s metadata — in realistic Model Context Protocol settings. Its authors describe it as the first benchmark to evaluate this threat systematically.
Contents
The benchmark is constructed on 45 live, real-world MCP servers and 353 authentic tools. It comprises 1312 malicious test cases covering 10 categories of potential risk.
Provenance
The test cases were generated by few-shot learning from three distinct attack templates designed by the authors. The paper states that the dataset is available at an anonymized repository.
Use
MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers uses MCPTox to evaluate 20 prominent LLM agent settings and reports widespread vulnerability to tool poisoning, including an attack success rate of 72.8% for o1-mini and refusal rates below 3% for every model evaluated.