b9819977a5
Includes prior-session patches (carry forward so the app compiles): - crates/gpui/build.rs: cross-compile manifest fix - crates/gpui/src/platform.rs: PlatformWindow::activate_with_token trait method - crates/gpui/src/window.rs: Window::activate_with_token public API - crates/gpui_linux/src/linux/wayland/window.rs: WaylandWindow::activate_with_token + activate() keyboard-serial fix Plus the focus-serial fix: - serial.rs: SerialKind::KeyboardEnter - client.rs: store wl_keyboard.enter serial; latest_serial_of() Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
109 lines
2.7 KiB
Markdown
109 lines
2.7 KiB
Markdown
# eval-cli
|
|
|
|
Headless CLI binary for running Zed's agent in evaluation/benchmark
|
|
environments. Designed to work inside containerized environments like
|
|
[Harbor](https://harborframework.com/) where the repository is already
|
|
checked out and API keys are provided via environment variables.
|
|
|
|
Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed
|
|
editor — full agentic loop with tool calls, subagents, and retries, just
|
|
without a GUI.
|
|
|
|
## Building
|
|
|
|
### Native (for local testing on the same OS)
|
|
|
|
```
|
|
cargo build --release -p eval_cli
|
|
```
|
|
|
|
### Cross-compile for Linux x86_64 (from macOS or other hosts)
|
|
|
|
Harbor containers run Linux x86_64. Use the Docker-based build script:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux
|
|
```
|
|
|
|
This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can
|
|
also specify a custom output path:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
|
|
```
|
|
|
|
## Standalone usage
|
|
|
|
```
|
|
eval-cli \
|
|
--workdir /testbed \
|
|
--model anthropic/claude-sonnet-4-6-latest \
|
|
--instruction "Fix the bug described in..." \
|
|
--timeout 600 \
|
|
--output-dir /logs/agent
|
|
```
|
|
|
|
Reads API keys from environment variables (`ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and
|
|
`thread.json` to the output directory.
|
|
|
|
### Exit codes
|
|
|
|
| Code | Meaning |
|
|
| ---- | ---------------------------------- |
|
|
| 0 | Agent finished |
|
|
| 1 | Error (model/auth/runtime failure) |
|
|
| 2 | Timeout |
|
|
| 3 | Interrupted (SIGTERM/SIGINT) |
|
|
|
|
## Harbor integration
|
|
|
|
The `zed_eval/` directory contains a Python package that
|
|
implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to
|
|
be used with `--agent-import-path` without modifying Harbor's source code.
|
|
|
|
### Setup
|
|
|
|
```
|
|
pip install -e crates/eval_cli/harbor/
|
|
```
|
|
|
|
### Running with a local binary
|
|
|
|
Build for Linux first, then pass the binary path:
|
|
|
|
```
|
|
crates/eval_cli/script/build-linux
|
|
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ae binary_path=target/eval-cli \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|
|
|
|
The agent uploads the binary into the container during setup — no
|
|
download URL needed during local iteration.
|
|
|
|
### Running with a download URL
|
|
|
|
For CI or when the binary is hosted somewhere:
|
|
|
|
```
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ak download_url=https://example.com/eval-cli \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|
|
|
|
### Setting a timeout
|
|
|
|
Pass `EVAL_CLI_TIMEOUT` via `--ae`:
|
|
|
|
```
|
|
harbor run -d "swebench_verified@latest" \
|
|
--agent-import-path zed_eval.agent:ZedAgent \
|
|
--ak binary_path=target/eval-cli \
|
|
--ae EVAL_CLI_TIMEOUT=600 \
|
|
-m anthropic/claude-sonnet-4-6-latest
|
|
```
|