LLM-based agents solve complex tasks through iterative reasoning, exploration and tool use, and this process can produce long, expensive context histories. The paper notes that state-of-the-art software engineering agents such as OpenHands and Cursor use LLM-based summarization to deal with this, and asks whether that added complexity brings tangible performance benefits compared with simply omitting older observations.
The authors run a systematic comparison of the two approaches within SWE-Agent on SWE-Bench Verified, across five diverse model configurations, and report initial evidence that their findings generalize to the OpenHands agent scaffold. They find that simple environment Observation Masking halves cost relative to the raw agent while matching, and sometimes slightly exceeding, the solve rate of LLM summarization. They also introduce a hybrid approach that reduces costs further.
Key Points
- A simple environment observation masking strategy halves cost relative to the raw agent while matching, and sometimes slightly exceeding, the solve rate of LLM summarization.
- The main comparison is run within SWE-agent on SWE-bench Verified across five model configurations; the evidence that the findings carry over to the OpenHands agent scaffold is described as initial.
- A novel hybrid approach reduces costs by a further 7% compared with observation masking alone and 11% compared with LLM summarization alone.
- The authors say their findings raise concerns about the trend towards pure LLM summarization and point to untapped cost reductions on the efficiency-effectiveness frontier.
Notes
The paper was first submitted to arXiv on 29 August 2025; version 3, dated 27 October 2025, is the camera-ready version for the 4th DL4C workshop co-located with NeurIPS 2025, and adds the OpenHands generality probe and the hybrid context-management strategy. The authors release code and data for reproducibility.