---
id: daily-agent-eval
name: "agent-eval"
url: https://skills.yangsir.net/skill/daily-agent-eval
author: affaan-m
domain: ai-dev-tools-workflow
tags: ["agent", "eval", "ai", "llm", "agents"]
install_count: 5200
rating: 4.40 (20 reviews)
github: https://github.com/affaan-m/everything-claude-code
---

# agent-eval

> 轻量级 CLI 工具，用于对比评估不同 AI 编程代理的性能和表现

**Stats**: 5,200 installs · 4.4/5 (20 reviews)

## Before / After 对比

### AI编程代理性能评估自动化

**Before**:

在没有 `agent-eval` 的情况下，要对比评估不同的AI编程代理（如Claude Code、Aider），通常需要手动进行。这包括为每个代理单独设置开发环境或Git分支，手动执行任务提示，然后逐个运行测试（如`pytest`）和代码检查（如`grep`）来判断成功与否。同时，还需要人工记录每次运行的通过率、耗时和预估成本。重复多次运行以确保结果一致性会非常耗时且易出错，环境差异也可能导致评估结果不可靠，最终决策往往基于“感觉”而非数据。

**After**:

通过 `agent-eval`，评估流程被彻底自动化和标准化。只需在YAML文件中声明式地定义任务，包括代码库、文件、提示和判断标准（如`pytest`和`grep`）。工具会自动为每个代理和每次运行创建独立的Git工作区，确保环境隔离和结果可复现。它会自动执行代理、运行判断逻辑，并收集通过率、成本、时间和一致性等关键指标。最终，通过简单的命令即可生成清晰的数据驱动型报告，实现系统化、客观的代理性能对比和选择。

| Metric | Before | After | Change |
|---|---|---|---|
| 数据获取与分析时间 | 240分钟 | 60分钟 | -75% |
| 结果一致性 | 50% | 95% | +90% |
| 任务定义与环境准备时间 | 60分钟 | 15分钟 | -75% |

## Readme

# agent-eval

# Agent Eval Skill

A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.

## When to Activate

- Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase

- Measuring agent performance before adopting a new tool or model

- Running regression checks when an agent updates its model or tooling

- Producing data-backed agent selection decisions for a team

## Installation

**Note:** Install agent-eval from its repository after reviewing the source.

## Core Concepts

### YAML Task Definitions

Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:

```
name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
files:
  - src/http_client.py
prompt: |
  Add retry logic with exponential backoff to all HTTP requests.
  Max 3 retries. Initial delay 1s, max delay 30s.
judge:
  - type: pytest
    command: pytest tests/test_http_client.py -v
  - type: grep
    pattern: "exponential_backoff|retry"
    files: src/http_client.py
commit: "abc1234"  # pin to specific commit for reproducibility

```

### Git Worktree Isolation

Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.

### Metrics Collected

Metric
What It Measures

Pass rate
Did the agent produce code that passes the judge?

Cost
API spend per task (when available)

Time
Wall-clock seconds to completion

Consistency
Pass rate across repeated runs (e.g., 3/3 = 100%)

## Workflow

### 1. Define Tasks

Create a `tasks/` directory with YAML files, one per task:

```
mkdir tasks
# Write task definitions (see template above)

```

### 2. Run Agents

Execute agents against your tasks:

```
agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3

```

Each run:

- Creates a fresh git worktree from the specified commit

- Hands the prompt to the agent

- Runs the judge criteria

- Records pass/fail, cost, and time

### 3. Compare Results

Generate a comparison report:

```
agent-eval report --format table

```

```
Task: add-retry-logic (3 runs each)
┌──────────────┬───────────┬────────┬────────┬─────────────┐
│ Agent        │ Pass Rate │ Cost   │ Time   │ Consistency │
├──────────────┼───────────┼────────┼────────┼─────────────┤
│ claude-code  │ 3/3       │ $0.12  │ 45s    │ 100%        │
│ aider        │ 2/3       │ $0.08  │ 38s    │  67%        │
└──────────────┴───────────┴────────┴────────┴─────────────┘

```

## Judge Types

### Code-Based (deterministic)

```
judge:
  - type: pytest
    command: pytest tests/ -v
  - type: command
    command: npm run build

```

### Pattern-Based

```
judge:
  - type: grep
    pattern: "class.*Retry"
    files: src/**/*.py

```

### Model-Based (LLM-as-judge)

```
judge:
  - type: llm
    prompt: |
      Does this implementation correctly handle exponential backoff?
      Check for: max retries, increasing delays, jitter.

```

## Best Practices

- **Start with 3-5 tasks** that represent your real workload, not toy examples

- **Run at least 3 trials** per agent to capture variance — agents are non-deterministic

- **Pin the commit** in your task YAML so results are reproducible across days/weeks

- **Include at least one deterministic judge** (tests, build) per task — LLM judges add noise

- **Track cost alongside pass rate** — a 95% agent at 10x the cost may not be the right choice

- **Version your task definitions** — they are test fixtures, treat them as code

## Links

- Repository: [github.com/joaquinhuigomez/agent-eval](https://github.com/joaquinhuigomez/agent-eval)

Weekly Installs286Repository[affaan-m/everyt…ude-code](https://github.com/affaan-m/everything-claude-code)GitHub Stars105.0KFirst Seen5 days agoSecurity Audits[Gen Agent Trust HubFail](/affaan-m/everything-claude-code/agent-eval/security/agent-trust-hub)[SocketPass](/affaan-m/everything-claude-code/agent-eval/security/socket)[SnykWarn](/affaan-m/everything-claude-code/agent-eval/security/snyk)Installed oncodex276cursor250opencode248gemini-cli248github-copilot248amp248

---
*Source: https://skills.yangsir.net/skill/daily-agent-eval*
*Markdown mirror: https://skills.yangsir.net/api/skill/daily-agent-eval/markdown*