# SWE-bench

!!! abstract "Overview"

    * We provide two scripts to run on the [SWE-bench](https://www.swebench.com/) benchmark.
    * `mini-extra swebench` runs on all task instances in batch mode.
    * `mini-extra swebench-single` runs on a single task instance with interactivity (useful for debugging).
    * You can also take a look at the runscripts to figure out how to build your own batch processing pipeline.

<figure markdown="span">
  <div class="gif-container gif-container-styled" data-glightbox-disabled>
    <img src="https://github.com/SWE-agent/swe-agent-media/blob/main/media/mini/png/swebench.png?raw=true"
         data-gif="https://github.com/SWE-agent/swe-agent-media/blob/main/media/mini/gif/swebench.gif?raw=true"
         alt="swebench" data-glightbox="false" width="600" />
  </div>
</figure>

## Usage

!!! warning "Docker container availability"

    The docker containers for Linux assume an x86 Linux architecture;
    you might not be able to run them on other architectures.


!!! tip "Quickstart"

    We provide two different scripts: `swebench` and `swebench-single`:

    === "Batch mode"

        Batch mode runs on all task instances in parallel.

        ```bash
        mini-extra swebench --help
        # or
        python src/minisweagent/run/benchmarks/swebench.py --help
        # Example:
        mini-extra swebench \
            --model anthropic/claude-sonnet-4-5-20250929 \
            --subset verified \
            --split test \
            --workers 4
        ```

        Basic flags:

        - `-o`, `--output` - Output directory
        - `-m`, `--model` - Model to use
        - `-c`, `--config` - Path to a config file (default: `swebench.yaml` in the `config` directory)
        - `-w`, `--workers` - Number of worker threads for parallel processing (default: `1`)

        Data selection flags:

        - `--subset` - SWEBench subset to use or path to a dataset (default: `lite`)
        - `--split` - Dataset split (default: `dev`)
        - `--slice` - Slice specification (e.g., '0:5' for first 5 instances)
        - `--filter` - Filter instance IDs by regex
        - `--shuffle` - Shuffle instances (default: `False`)
        - `--redo-existing` - Redo existing instances (default: `False`)

        Advanced flags:

        - `--environment-class` - Environment type to use (recommended: `docker` or `singularity`)

    === "Single instance (for debugging)"

        Single instance mode runs on a single task instance with interactivity. This is meant for debugging, and so unlike the batch mode command above, this will not produce a preds.json file.

        ```bash
        mini-extra swebench-single --help
        # or
        python src/minisweagent/run/benchmarks/swebench_single.py --help
        # Example:
        mini-extra swebench-single \
            --subset verified \
            --split test \
            --model anthropic/claude-sonnet-4-5-20250929 \
            -i sympy__sympy-15599
        # or
        mini-extra swebench-single \
            --subset verified \
            --split test \
            -m anthropic/claude-sonnet-4-5-20250929 \
            -i 0  # instance index
        ```

        Note: If you want to run the script without prompting for confirmation at exit,
        add the `--exit-immediately` flag.

        Basic flags:

        - `-m`, `--model` - Model to use
        - `-c`, `--config` - Path to a config file (default: `swebench.yaml` in the `config` directory)
        - `-o`, `--output` - Output trajectory file (default: saves to global config directory)

        Data selection flags:

        - `--subset` - SWEBench subset to use or path to a dataset (default: `lite`)
        - `--split` - Dataset split (default: `dev`)
        - `-i`, `--instance` - SWE-Bench instance ID (default: `0`)

        Advanced flags:

        - `--environment-class` - Environment type to use (recommended: `docker` or `singularity`)
        - `--exit-immediately` - Exit immediately when the agent wants to finish instead of prompting (default: `False`)

!!! tip "Evaluating on SWE-bench"

    You have two options to evaluate on SWE-bench: Our free cloud-based evaluation or the SWE-bench CLI.

    === "Cloud-based evaluation"

        You can use the [sb-cli](https://www.swebench.com/sb-cli/) for extremely fast, cloud-based evaluations
        (and it's free!). After installing it and getting a token, simply run:

        ```bash
        sb-cli submit swe-bench_verified test --predictions_path preds.json --run_id some-id-for-your-run
        ```

        Typically you will have results within 20 minutes (this is not limited by how many instances you run,
        but by the slowest-to-evaluate instance in SWE-bench).

    === "Local evaluation"

        You can also use a local installation of [SWE-bench](https://github.com/SWE-bench/SWE-bench)
        for evaluation:

        ```bash
        python -m swebench.harness.run_evaluation \
            --dataset_name princeton-nlp/SWE-bench_Verified \
            --predictions_path preds.jsonl \
            --max_workers <num_workers> \
            --run_id <run_id>
        ```

## FAQ

> Can I set global cost limits?

Yes, you can set global cost limits with the `MSWEA_GLOBAL_CALL_LIMIT` and `MSWEA_GLOBAL_COST_LIMIT` environment variables/global config.
See [global configuration](../advanced/global_configuration.md) for more details.

> What happens to uncompleted tasks when I abort with KeyboardInterrupt?

Trajectories are only saved upon completion, so most likely, you can just rerun the script to complete the tasks next time.
However, you should still check for `KeyboardInterrupt` in `preds.json` in case some tasks were aborted but saved.

> Certain tasks are being stuck even though I deleted the trajectories.

The completed instances are inferred from `preds.json`. Remove the corresponding items from the file.

> How can I run on a different dataset?

As long as it follows the SWE-bench format, you can use `--subset /path/to/your/dataset` to run on a custom dataset.
The dataset needs to be loadable as `datasets.load_dataset(path, split=split)`.

> Some progress runners are stuck at 'initializing task' for a very long time / time out

They might be pulling docker containers -- the run should start immediately the next time.
If you see timeouts because of `docker pull` operations, you might want to increase `environment.pull_timeout`
from the default of `120` (seconds).

> I have some docker issues

Try running the docker command manually to see what's going on (it should be printed out in the console).
Confirm that it's running with `docker ps`, and that you can use `docker exec -it <container-id> ls` to get some output.

> Docker isn't available on my HPC cluster.

You can use the singularity/apptainer backend by setting `environment.environment_class` to `singularity`
in your [agent config file](../advanced/yaml_configuration.md)
or specify `--environment-class singularity` from the command line

> Can I run a startup command in the environment?

Yes, you can use the `run.env_startup_command` config option to run a command in the environment before the agent starts.
For example:

```yaml
run:
  env_startup_command: "apt-get update && apt-get install -y python3-pip"
```

The command is rendered with the instance variables as template variables using `jinja2`.
For example, you could use

```yaml
run:
  env_startup_command: "git clone {{ repo_url }} . --force"
```

which might be particularly useful when running with environments like [`bubblewrap`](../reference/environments/bubblewrap.md).

> What environment can I use for SWE-bench?

See [this guide](../advanced/environments.md) for more details.

## Implementation

??? note "Default config"

    - [Read on GitHub](https://github.com/swe-agent/mini-swe-agent/blob/main/src/minisweagent/config/benchmarks/swebench.yaml)

    ```yaml
    --8<-- "src/minisweagent/config/benchmarks/swebench.yaml"
    ```

??? note "`swebench.py` run script"

    - [Read on GitHub](https://github.com/swe-agent/mini-swe-agent/blob/main/src/minisweagent/run/benchmarks/swebench.py)
    - [API reference](../reference/run/swebench.md)

    ```python
    --8<-- "src/minisweagent/run/benchmarks/swebench.py"
    ```

??? note "`swebench_single.py` run script"

    - [Read on GitHub](https://github.com/swe-agent/mini-swe-agent/blob/main/src/minisweagent/run/benchmarks/swebench_single.py)
    - [API reference](../reference/run/swebench_single.md)

    ```python
    --8<-- "src/minisweagent/run/benchmarks/swebench_single.py"
    ```

{% include-markdown "../_footer.md" %}