b72a46db68
Some checks failed
Update All Top Ranking Issues / update_top_ranking_issues (push) Has been cancelled
Triage Project Sync (#84) / Sync triage project (push) Has been cancelled
release_nightly / notify_on_failure (push) Has been cancelled
release_nightly / check_style (push) Has been cancelled
release_nightly / run_tests_windows (push) Has been cancelled
release_nightly / clippy_windows (push) Has been cancelled
release_nightly / bundle_linux_aarch64 (push) Has been cancelled
release_nightly / bundle_linux_x86_64 (push) Has been cancelled
release_nightly / bundle_mac_aarch64 (push) Has been cancelled
release_nightly / bundle_mac_x86_64 (push) Has been cancelled
release_nightly / bundle_windows_aarch64 (push) Has been cancelled
release_nightly / bundle_windows_x86_64 (push) Has been cancelled
release_nightly / build_nix_linux_x86_64 (push) Has been cancelled
release_nightly / build_nix_mac_aarch64 (push) Has been cancelled
release_nightly / update_nightly_tag (push) Has been cancelled
Hotfix Review Monitor / check-hotfix-reviews (push) Has been cancelled
Stale PR Review Reminder / check-stale-prs (push) Has been cancelled
Update Weekly Top Ranking Issues / update_top_ranking_issues (push) Has been cancelled
Bump collab-staging Tag / update-collab-staging-tag (push) Has been cancelled
compliance_check / scheduled_compliance_check (push) Has been cancelled
Single-commit orphan branch: full zed-industries/zed @ 8c74db0 source tree with a 3-file patch applied (no upstream history). Patch (crates/gpui_linux/src/linux/wayland/): - serial.rs: add SerialKind::KeyboardEnter - client.rs: store wl_keyboard.enter serial; add latest_serial_of() - window.rs: activate() uses keyboard-enter serial (Mutter focus gate) Mutter honors window activation only when the token carries the keyboard- focus serial from wl_keyboard.enter; GPUI used a stale mouse-press serial. See docs/tray-window-focus-wayland.md in logiguard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
109 lines
2.7 KiB
Markdown
109 lines
2.7 KiB
Markdown
# eval-cli
|
|
|
|
Headless CLI binary for running Zed's agent in evaluation/benchmark
|
|
environments. Designed to work inside containerized environments like
|
|
[Harbor](https://harborframework.com/) where the repository is already
|
|
checked out and API keys are provided via environment variables.
|
|
|
|
Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed
|
|
editor — full agentic loop with tool calls, subagents, and retries, just
|
|
without a GUI.
|
|
|
|
## Building
|
|
|
|
### Native (for local testing on the same OS)
|
|
|
|
```
|
|
cargo build --release -p eval_cli
|
|
```
|
|
|
|
### Cross-compile for Linux x86_64 (from macOS or other hosts)
|
|
|
|
Harbor containers run Linux x86_64. Use the Docker-based build script:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux
|
|
```
|
|
|
|
This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can
|
|
also specify a custom output path:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
|
|
```
|
|
|
|
## Standalone usage
|
|
|
|
```
|
|
eval-cli \
|
|
--workdir /testbed \
|
|
--model anthropic/claude-sonnet-4-6-latest \
|
|
--instruction "Fix the bug described in..." \
|
|
--timeout 600 \
|
|
--output-dir /logs/agent
|
|
```
|
|
|
|
Reads API keys from environment variables (`ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and
|
|
`thread.json` to the output directory.
|
|
|
|
### Exit codes
|
|
|
|
| Code | Meaning |
|
|
| ---- | ---------------------------------- |
|
|
| 0 | Agent finished |
|
|
| 1 | Error (model/auth/runtime failure) |
|
|
| 2 | Timeout |
|
|
| 3 | Interrupted (SIGTERM/SIGINT) |
|
|
|
|
## Harbor integration
|
|
|
|
The `zed_eval/` directory contains a Python package that
|
|
implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to
|
|
be used with `--agent-import-path` without modifying Harbor's source code.
|
|
|
|
### Setup
|
|
|
|
```
|
|
pip install -e crates/eval_cli/harbor/
|
|
```
|
|
|
|
### Running with a local binary
|
|
|
|
Build for Linux first, then pass the binary path:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux
|
|
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ae binary_path=target/eval-cli \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|
|
|
|
The agent uploads the binary into the container during setup — no
|
|
download URL needed during local iteration.
|
|
|
|
### Running with a download URL
|
|
|
|
For CI or when the binary is hosted somewhere:
|
|
|
|
```
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ak download_url=https://example.com/eval-cli \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|
|
|
|
### Setting a timeout
|
|
|
|
Pass `EVAL_CLI_TIMEOUT` via `--ae`:
|
|
|
|
```
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ak binary_path=target/eval-cli \
|
|
--ae EVAL_CLI_TIMEOUT=600 \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|