Today's Agents can already proficiently call tools, modify files, and run tests, but when facing a large software system that has evolved over the long term, can they start from a new requirement and deliver a complete feature?
When a product manager proposes a new requirement, what developers receive is often not a file list or function signatures. They first need to understand the constraints, trace module dependencies, and then determine where to change, how to change, and how to prove the changes are correct. These analyses affect subsequent delivery.
Fudan University, City University of Hong Kong, The Chinese University of Hong Kong and other institutions propose LoLBench, which places Agents into software systems averaging about 2.4 million lines of source code, evaluates the complete process from feature proposal to module delivery, and tests the perception ability to ground requirements into the system, as well as the implementation ability to complete cross-module modifications.
Paper link: https://arxiv.org/abs/2609.37143
Dataset: https://huggingface.co/datasets/lolbench26/LoLBench
The experiments expose a key breakpoint: an Agent may make a correct local change but miss changes in other modules that must be synchronized. Even with supplementary implementation guidance, most tasks remain unsolved, and there is still a long distance from understanding requirements to completing delivery.
Redefining long-horizon code tasks
What kind of long-horizon code task has actual engineering value?
"Long-horizon" cannot be judged only by time consumed, number of rounds, or Token consumption. For example, asking an Agent to output one number per minute, counting from 1 to 1000, takes a long time but only requires a simple loop. SWE-bench-style local bug-fixing tasks also cannot cover the challenge from requirement analysis to module delivery.
The long-horizon code tasks that LoLBench focuses on are module development for complex systems. After receiving a new requirement, developers repeatedly analyze: which behaviors must be implemented? Can the existing architecture support them? Which dependencies will be affected? How to verify? LoLBench also incorporates this analysis that is hidden in developers' minds into the evaluation, letting the Agent derive the implementation plan from user intent on its own.
Such tasks simultaneously have perception complexity and implementation complexity: the former is the difficulty of understanding the proposal, identifying implicit constraints and cross-module dependencies, and deriving the modification scope and implementation specification; the latter is the difficulty of converting the specification into correct cross-file modifications and, through testing and repair, preventing existing functionality from being broken.
Figure 1 Comparison of perception complexity and implementation complexity.
How LoLBench evaluates both types of complexity simultaneously
Deriving implementation specifications from feature proposals
LoLBench selects 100 tasks from community Enhancement Proposals and their corresponding merged PRs, covering 29 systems, 8 languages, and 5 domains.
The Agent only receives the proposal and the code repository before implementation; the proposal averages about 5,000 English words. Evaluation is conducted in an offline Docker environment to prevent the Agent from retrieving upstream implementations.
For example, PEP 617 [1] proposes replacing CPython's LL(1) parser with PEG, and implementation requires connecting grammar definitions, parser generation, abstract syntax trees, and error reporting. The Agent must identify the entire implementation chain.
In terms of perception complexity, the average mention rate in the proposal of the target class names in LoLBench's reference implementation is only 7.3%, less than half of SWE-bench Verified (15.9%) and less than one-fifth of FeatureBench (38.7%). The Agent must infer the connection between requirements and code on its own.
Turning specifications into engineering delivery
In terms of implementation complexity, LoLBench repositories average about 2.4 million lines of source code, about 9.4 times that of SWE-bench Verified and 4.9 times that of FeatureBench; implementations change an average of 5,499 lines, compared with 38 lines and 2,050 lines for the latter two. Each task changes an average of 64.9 files, 40.1 classes, and 195.9 functions; the number of files is 25 times and 4.2 times those of the two, respectively. Cross-module coordination and verification are more difficult.
Table 1 Comparison of LoLBench with existing code task evaluation sets.
LoLBench verifies delivery correctness through two types of tests: new feature tests (F2P) require failure before implementation and passing after implementation; F2P verifies behavior through public interfaces and allows different internal implementations. Regression tests (P2P) require passing both before and after.
On top of the project's original tests, the research team constructs erroneous implementations such as missing features and tampered requirements as mutants, supplements test cases to identify missed errors, and improves the test coverage of reference patches.
After enhancement, the detection rate of 1,688 mutants rises from about 61% to 96%, and the average per-task line coverage of F2P rises from about 60% to 79%.
Performance and bottlenecks of 28 Agent combinations
Changing models or scaffolds, the highest Agent solve rate is only 14%
The research team evaluates 28 available combinations of 6 frontier models and 5 scaffolds, with a 6-hour time limit per task. Opus 5 paired with Claude Code or mini-SWE-agent both tie for the highest solve rate at 14%, and the median across all combinations is only 2%.
Table 2 Main experimental results in the paper
Using Opus 5 as well, mini-SWE-agent and Claude Code achieved the same resolution rate, but the former took 3.2 times as long as the latter and generated 5 times as many tokens; more consumption did not bring a higher resolution rate.
A large amount of interaction and tokens are spent on code localization
In terms of token consumption, averaged across agents, the localization stage accounted for 52.4% of non-cached input tokens and 45.4% of generated tokens. In terms of the number of agent interaction rounds, the localization stage also had significantly more interaction rounds than other stages.
Figure 2 Average number of interaction rounds at each stage.
For existing files and functions modified against the reference implementation, across all 28 groups of agents, the file modification precision ranged from 38.0% to 80.4%, and the recall ranged from 30.3% to 65.0%; the function modification precision ranged from 54.2% to 65.6%, and the recall ranged from 20.8% to 53.4%. Code localization accuracy is low, and coverage remains incomplete.
Even after supplementing modification guidance, agents still struggle to deliver completely
The research team retained the original feature proposal and additionally provided a modification file tree exported from the reference implementation and API implementation specifications to help agents determine the scope of modifications. The resolution rate of Opus 5+Claude Code rose from 14% to 34%, GPT-5.6 Sol+Codex rose from 3% to 25%,
DeepSeek
V4 Flash+OpenCode rose from 1% to 17%.
Table 3 Experimental results after supplementing the modification file tree and API implementation specifications
Supplementing modification guidance can alleviate the perception burden, but the best combination still had 66% of tasks unresolved. Agents still need to coordinate modules, implement interfaces, and pass validation, which suggests that the implementation complexity of large systems is also very high.
From code generation toward the evolution of large software systems
These results provide testable directions for agent research and development: whether repository retrieval can cover key dependencies, whether implementation specifications before coding are complete, and whether validation can promptly detect cross-module omissions.
The value of software engineering lies in whether user requirements can be reliably fulfilled. Agents need to understand why changes occur, how they affect the system, and fully implement the changes. LoLBench hopes to promote the evaluation of agent capabilities from generating code toward supporting the continuous evolution of software, making delivery results the yardstick for testing agent capabilities.
Reference: PEP 617: https://peps.python.org/pep-0617/
This article is from the WeChat public account "Xinzhiyuan", author: Xinzhiyuan; editor: LRST
