logiguard fork v3: full patch set on verified 8c74db0 tree
Includes prior-session patches (carry forward so the app compiles): - crates/gpui/build.rs: cross-compile manifest fix - crates/gpui/src/platform.rs: PlatformWindow::activate_with_token trait method - crates/gpui/src/window.rs: Window::activate_with_token public API - crates/gpui_linux/src/linux/wayland/window.rs: WaylandWindow::activate_with_token + activate() keyboard-serial fix Plus the focus-serial fix: - serial.rs: SerialKind::KeyboardEnter - client.rs: store wl_keyboard.enter serial; latest_serial_of() Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
108
crates/eval_cli/README.md
Normal file
108
crates/eval_cli/README.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# eval-cli
|
||||
|
||||
Headless CLI binary for running Zed's agent in evaluation/benchmark
|
||||
environments. Designed to work inside containerized environments like
|
||||
[Harbor](https://harborframework.com/) where the repository is already
|
||||
checked out and API keys are provided via environment variables.
|
||||
|
||||
Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed
|
||||
editor — full agentic loop with tool calls, subagents, and retries, just
|
||||
without a GUI.
|
||||
|
||||
## Building
|
||||
|
||||
### Native (for local testing on the same OS)
|
||||
|
||||
```
|
||||
cargo build --release -p eval_cli
|
||||
```
|
||||
|
||||
### Cross-compile for Linux x86_64 (from macOS or other hosts)
|
||||
|
||||
Harbor containers run Linux x86_64. Use the Docker-based build script:
|
||||
|
||||
```
|
||||
crates/eval_cli/script/build-linux
|
||||
```
|
||||
|
||||
This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can
|
||||
also specify a custom output path:
|
||||
|
||||
```
|
||||
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
|
||||
```
|
||||
|
||||
## Standalone usage
|
||||
|
||||
```
|
||||
eval-cli \
|
||||
--workdir /testbed \
|
||||
--model anthropic/claude-sonnet-4-6-latest \
|
||||
--instruction "Fix the bug described in..." \
|
||||
--timeout 600 \
|
||||
--output-dir /logs/agent
|
||||
```
|
||||
|
||||
Reads API keys from environment variables (`ANTHROPIC_API_KEY`,
|
||||
`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and
|
||||
`thread.json` to the output directory.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ---------------------------------- |
|
||||
| 0 | Agent finished |
|
||||
| 1 | Error (model/auth/runtime failure) |
|
||||
| 2 | Timeout |
|
||||
| 3 | Interrupted (SIGTERM/SIGINT) |
|
||||
|
||||
## Harbor integration
|
||||
|
||||
The `zed_eval/` directory contains a Python package that
|
||||
implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to
|
||||
be used with `--agent-import-path` without modifying Harbor's source code.
|
||||
|
||||
### Setup
|
||||
|
||||
```
|
||||
pip install -e crates/eval_cli/harbor/
|
||||
```
|
||||
|
||||
### Running with a local binary
|
||||
|
||||
Build for Linux first, then pass the binary path:
|
||||
|
||||
```
|
||||
crates/eval_cli/script/build-linux
|
||||
|
||||
harbor run -d "swebench_verified@latest" \
|
||||
--agent-import-path zed_eval.agent:ZedAgent \
|
||||
--ae binary_path=target/eval-cli \
|
||||
-m anthropic/claude-sonnet-4-6-latest
|
||||
```
|
||||
|
||||
The agent uploads the binary into the container during setup — no
|
||||
download URL needed during local iteration.
|
||||
|
||||
### Running with a download URL
|
||||
|
||||
For CI or when the binary is hosted somewhere:
|
||||
|
||||
```
|
||||
harbor run -d "swebench_verified@latest" \
|
||||
--agent-import-path zed_eval.agent:ZedAgent \
|
||||
--ak download_url=https://example.com/eval-cli \
|
||||
-m anthropic/claude-sonnet-4-6-latest
|
||||
```
|
||||
|
||||
### Setting a timeout
|
||||
|
||||
Pass `EVAL_CLI_TIMEOUT` via `--ae`:
|
||||
|
||||
```
|
||||
harbor run -d "swebench_verified@latest" \
|
||||
--agent-import-path zed_eval.agent:ZedAgent \
|
||||
--ak binary_path=target/eval-cli \
|
||||
--ae EVAL_CLI_TIMEOUT=600 \
|
||||
-m anthropic/claude-sonnet-4-6-latest
|
||||
```
|
||||
Reference in New Issue
Block a user