SciGroveBeta
Machine Learning

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

Yutao Sun, Yanting Miao, Hao-Xuan Ma, Mengyu Zhou

Featured July 3, 2026

Original Paper

This analysis was generated by SciGrove. Upload your own PDFs or enter a DOI — and get the same AI breakdown on any paper.

Get started

AI-generated analysis — This is SciGrove's AI interpretation of the paper, not peer-reviewed content. Always refer to the original paper.

Simply

A new test makes AI coding helpers work like real developers, starting with vague ideas and getting feedback over many steps, showing they struggle more than with simple, clear tasks.

In depth
The paper introduces SWE-INTERACT, a novel benchmark that evaluates coding agents on multi-turn, interactive software engineering tasks, mirroring realistic developer workflows. Unlike traditional benchmarks that provide full requirements upfront, SWE-INTERACT employs a persona-conditioned user simulator that progressively reveals vague instructions, inspects the agent's workspace, and provides targeted feedback. This setup tests an agent's ability to discover user intent, adapt to evolving requirements, and iteratively refine its work, revealing a significant performance drop compared to single-turn tasks.

Key Takeaways

  • 1
    Existing single-turn SWE benchmarks fail to capture the iterative and interactive nature of real-world software development, where requirements evolve and feedback is progressive.
  • 2
    The SWE-INTERACT benchmark introduces a sophisticated user simulator that emulates human developers, providing vague initial instructions, inspecting agent workspaces, and offering multi-turn feedback.
  • 3
    Evaluations show that even frontier models experience a substantial drop in resolve rates (e.g., from ~50% to ~25%) and significantly increased interaction length when faced with multi-turn, user-driven tasks, highlighting a critical, under-measured capability gap.

Conceptual Flow

HIGH LEVEL
1
Methodology: Simulating Real Developer Workflows

Instead of giving all instructions at once, a smart computer pretends to be a boss, giving small hints and checking the helper's work step-by-step.

Vague Task Idea
Boss Gives Hints
Helper Tries Code
Boss Checks Work
More Hints
2
Results: AI Agents Struggle with Real-World Interaction

The smart computer helpers that were good at simple tasks found it much harder to finish when the boss kept changing their mind or giving new instructions.

Simple Task Success
Interactive Task Success
Compare Performance
Big Drop in Success
More Time Spent