phoenix-evals
为 AI/LLM 应用构建评估器，代码优先的方式实现精准的模型性能评估
npx skills add https://github.com/github/awesome-copilot --skill phoenix-evalsBefore / After 效果对比
1 组在没有 `phoenix-evals` 的情况下，评估AI/LLM应用的性能通常是一个耗时且易出错的过程。开发者需要手动审查大量的模型输出，或者编写一次性的脚本进行简单的关键词匹配或格式检查。对于更复杂的语义或细微差别，往往需要人工专家介入，逐一标注或打分。这种方法难以扩展到大型数据集，导致评估覆盖率低，结果主观且一致性差。每次模型迭代或提示词调整后，都需要重复大量的手动工作，严重拖慢了开发周期，难以快速获得可靠的性能反馈。
借助 `phoenix-evals`，AI/LLM应用的性能评估变得高效、自动化且精准。开发者可以利用其“代码优先”的方式，快速构建自定义的代码评估器进行客观检查，并结合“LLM for nuance”功能，创建基于大模型的评估器来捕捉复杂的语义和上下文。它支持批量处理DataFrame数据，实现对整个数据集的全面评估。通过运行实验，可以轻松比较不同模型或提示词的效果，并利用“validate against humans”机制确保评估结果的准确性。这显著减少了手动工作量，提升了评估的一致性和覆盖率，加速了模型迭代和优化过程。
phoenix-evals
Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
Task Files
Setup setup-python, setup-typescript
Decide what to evaluate evaluators-overview
Choose a judge model fundamentals-model-selection
Use pre-built evaluators evaluators-pre-built
Build code evaluator evaluators-code-python, evaluators-code-typescript
Build LLM evaluator evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame evaluate-dataframe-python
Run experiment experiments-running-python, experiments-running-typescript
Create dataset experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review observe-sampling-python, observe-sampling-typescript
Analyze errors error-analysis, error-analysis-multi-turn, axial-coding
RAG evals evaluators-rag
Avoid common mistakes common-mistakes-python, fundamentals-anti-patterns
Production production-overview, production-guardrails, production-continuous
Workflows
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production: production-overview → production-guardrails → production-continuous
Reference Categories
Prefix Description
fundamentals-*
Types, scores, anti-patterns
observe-*
Tracing, sampling
error-analysis-*
Finding failures
axial-coding-*
Categorizing failures
evaluators-*
Code, LLM, RAG evaluators
experiments-*
Datasets, running experiments
validation-*
Validating evaluator accuracy against human labels
production-*
CI/CD, monitoring
Key Principles
Principle Action
Error analysis first Can't automate what you haven't observed
Custom > generic Build from your failures
Code first Deterministic before LLM
Validate judges
80% TPR/TNR
Binary > Likert Pass/fail, not 1-5
Weekly Installs520Repositorygithub/awesome-copilotGitHub Stars29.6KFirst Seen12 days agoSecurity AuditsGen Agent Trust HubPassSocketPassSnykPassInstalled onopencode477codex475gemini-cli474github-copilot474deepagents471amp471
用户评价 (0)
发表评价
暂无评价
统计数据
用户评分
为此 Skill 评分