---
id: daily-phoenix-evals
name: "phoenix-evals"
url: https://skills.yangsir.net/skill/daily-phoenix-evals
author: github
domain: ai-llm-engineering
tags: ["evaluation", "llm", "phoenix", "testing", "ai"]
install_count: 1000
rating: 4.30 (7 reviews)
github: https://github.com/github/awesome-copilot
---

# phoenix-evals

> 为 AI/LLM 应用构建评估器，代码优先的方式实现精准的模型性能评估

**Stats**: 1,000 installs · 4.3/5 (7 reviews)

## Before / After 对比

### LLM应用性能评估效率与准确性

**Before**:

在没有 `phoenix-evals` 的情况下，评估AI/LLM应用的性能通常是一个耗时且易出错的过程。开发者需要手动审查大量的模型输出，或者编写一次性的脚本进行简单的关键词匹配或格式检查。对于更复杂的语义或细微差别，往往需要人工专家介入，逐一标注或打分。这种方法难以扩展到大型数据集，导致评估覆盖率低，结果主观且一致性差。每次模型迭代或提示词调整后，都需要重复大量的手动工作，严重拖慢了开发周期，难以快速获得可靠的性能反馈。

**After**:

借助 `phoenix-evals`，AI/LLM应用的性能评估变得高效、自动化且精准。开发者可以利用其“代码优先”的方式，快速构建自定义的代码评估器进行客观检查，并结合“LLM for nuance”功能，创建基于大模型的评估器来捕捉复杂的语义和上下文。它支持批量处理DataFrame数据，实现对整个数据集的全面评估。通过运行实验，可以轻松比较不同模型或提示词的效果，并利用“validate against humans”机制确保评估结果的准确性。这显著减少了手动工作量，提升了评估的一致性和覆盖率，加速了模型迭代和优化过程。

| Metric | Before | After | Change |
|---|---|---|---|
| 评估时间 | 20小时/1000条数据 | 1小时/1000条数据 | -95% |
| 评估覆盖率 | 10% | 100% | +900% |
| 评估结果一致性 | 70% | 95% | +35.71% |

## Readme

# phoenix-evals

# Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

## Quick Reference

Task
Files

Setup
[setup-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/setup-python.md), [setup-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/setup-typescript.md)

Decide what to evaluate
[evaluators-overview](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-overview.md)

Choose a judge model
[fundamentals-model-selection](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/fundamentals-model-selection.md)

Use pre-built evaluators
[evaluators-pre-built](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-pre-built.md)

Build code evaluator
[evaluators-code-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-code-python.md), [evaluators-code-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-code-typescript.md)

Build LLM evaluator
[evaluators-llm-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-llm-python.md), [evaluators-llm-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-llm-typescript.md), [evaluators-custom-templates](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-custom-templates.md)

Batch evaluate DataFrame
[evaluate-dataframe-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluate-dataframe-python.md)

Run experiment
[experiments-running-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-running-python.md), [experiments-running-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-running-typescript.md)

Create dataset
[experiments-datasets-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-datasets-python.md), [experiments-datasets-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-datasets-typescript.md)

Generate synthetic data
[experiments-synthetic-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-synthetic-python.md), [experiments-synthetic-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/experiments-synthetic-typescript.md)

Validate evaluator accuracy
[validation](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/validation.md), [validation-evaluators-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/validation-evaluators-python.md), [validation-evaluators-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/validation-evaluators-typescript.md)

Sample traces for review
[observe-sampling-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/observe-sampling-python.md), [observe-sampling-typescript](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/observe-sampling-typescript.md)

Analyze errors
[error-analysis](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/error-analysis.md), [error-analysis-multi-turn](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/error-analysis-multi-turn.md), [axial-coding](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/axial-coding.md)

RAG evals
[evaluators-rag](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-rag.md)

Avoid common mistakes
[common-mistakes-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/common-mistakes-python.md), [fundamentals-anti-patterns](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/fundamentals-anti-patterns.md)

Production
[production-overview](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-overview.md), [production-guardrails](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-guardrails.md), [production-continuous](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-continuous.md)

## Workflows

**Starting Fresh:**
[observe-tracing-setup](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/observe-tracing-setup.md) → [error-analysis](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/error-analysis.md) → [axial-coding](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/axial-coding.md) → [evaluators-overview](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-overview.md)

**Building Evaluator:**
[fundamentals](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/fundamentals.md) → [common-mistakes-python](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/common-mistakes-python.md) → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

**RAG Systems:**
[evaluators-rag](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/evaluators-rag.md) → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

**Production:**
[production-overview](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-overview.md) → [production-guardrails](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-guardrails.md) → [production-continuous](https://github.com/github/awesome-copilot/blob/HEAD/skills/phoenix-evals/references/production-continuous.md)

## Reference Categories

Prefix
Description

`fundamentals-*`
Types, scores, anti-patterns

`observe-*`
Tracing, sampling

`error-analysis-*`
Finding failures

`axial-coding-*`
Categorizing failures

`evaluators-*`
Code, LLM, RAG evaluators

`experiments-*`
Datasets, running experiments

`validation-*`
Validating evaluator accuracy against human labels

`production-*`
CI/CD, monitoring

## Key Principles

Principle
Action

Error analysis first
Can't automate what you haven't observed

Custom > generic
Build from your failures

Code first
Deterministic before LLM

Validate judges
>80% TPR/TNR

Binary > Likert
Pass/fail, not 1-5

Weekly Installs520Repository[github/awesome-copilot](https://github.com/github/awesome-copilot)GitHub Stars29.6KFirst Seen12 days agoSecurity Audits[Gen Agent Trust HubPass](/github/awesome-copilot/phoenix-evals/security/agent-trust-hub)[SocketPass](/github/awesome-copilot/phoenix-evals/security/socket)[SnykPass](/github/awesome-copilot/phoenix-evals/security/snyk)Installed onopencode477codex475gemini-cli474github-copilot474deepagents471amp471

---
*Source: https://skills.yangsir.net/skill/daily-phoenix-evals*
*Markdown mirror: https://skills.yangsir.net/api/skill/daily-phoenix-evals/markdown*