--- id: daily-arize-experiment name: "arize-experiment" url: https://skills.yangsir.net/skill/daily-arize-experiment author: github domain: ai-llm-engineering tags: ["ai-agents", "llm-orchestration", "automation"] install_count: 1100 rating: 4.30 (7 reviews) github: https://github.com/github/awesome-copilot --- # arize-experiment > 在 Arize 上创建和管理实验，对数据集运行 LLM 评估，收集模型输出和评估指标，支持版本对比和性能分析 **Stats**: 1,100 installs · 4.3/5 (7 reviews) ## Before / After 对比 ### LLM 模型评估实验 **Before**: 手动准备测试数据集、调用模型 API、收集输出并计算指标，一个实验需要 2-3 小时，缺乏统一的实验管理框架 **After**: 使用 Arize Experiment 自动运行数据集评估，收集输出和指标，自动生成对比报告，30 分钟完成完整的实验流程 | Metric | Before | After | Change | |---|---|---|---| | 实验时间 | 150分钟 | 30分钟 | -80% | ## Readme # arize-experiment # Arize Experiment Skill ## Concepts - **Experiment** = a named evaluation run against a specific dataset version, containing one run per example - **Experiment Run** = the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata - **Dataset** = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version - **Evaluation** = a named metric attached to a run (e.g., `correctness`, `relevance`), with optional label, score, and explanation The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs. ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see references/ax-setup.md - `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via references/ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key ([https://app.arize.com/admin](https://app.arize.com/admin) > API Keys) - Space ID unknown → check `.env` for `ARIZE_SPACE_ID`, or run `ax spaces list -o json`, or ask the user - Project unclear → check `.env` for `ARIZE_DEFAULT_PROJECT`, or ask, or run `ax projects list -o json --limit 100` and present as selectable options ## List Experiments: `ax experiments list` Browse experiments, optionally filtered by dataset. Output goes to stdout. ``` ax experiments list ax experiments list --dataset-id DATASET_ID --limit 20 ax experiments list --cursor CURSOR_TOKEN ax experiments list -o json ``` ### Flags Flag Type Default Description `--dataset-id` string none Filter by dataset `--limit, -l` int 15 Max results (1-100) `--cursor` string none Pagination cursor from previous response `-o, --output` string table Output format: table, json, csv, parquet, or file path `-p, --profile` string default Configuration profile ## Get Experiment: `ax experiments get` Quick metadata lookup -- returns experiment name, linked dataset/version, and timestamps. ``` ax experiments get EXPERIMENT_ID ax experiments get EXPERIMENT_ID -o json ``` ### Flags Flag Type Default Description `EXPERIMENT_ID` string required Positional argument `-o, --output` string table Output format `-p, --profile` string default Configuration profile ### Response fields Field Type Description `id` string Experiment ID `name` string Experiment name `dataset_id` string Linked dataset ID `dataset_version_id` string Specific dataset version used `experiment_traces_project_id` string Project where experiment traces are stored `created_at` datetime When the experiment was created `updated_at` datetime Last modification time ## Export Experiment: `ax experiments export` Download all runs to a file. By default uses the REST API; pass `--all` to use Arrow Flight for bulk transfer. ``` ax experiments export EXPERIMENT_ID # -> experiment_abc123_20260305_141500/runs.json ax experiments export EXPERIMENT_ID --all ax experiments export EXPERIMENT_ID --output-dir ./results ax experiments export EXPERIMENT_ID --stdout ax experiments export EXPERIMENT_ID --stdout | jq '.[0]' ``` ### Flags Flag Type Default Description `EXPERIMENT_ID` string required Positional argument `--all` bool false Use Arrow Flight for bulk export (see below) `--output-dir` string `.` Output directory `--stdout` bool false Print JSON to stdout instead of file `-p, --profile` string default Configuration profile ### REST vs Flight (`--all`) - **REST** (default): Lower friction -- no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page. - **Flight** (`--all`): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port (`flight.arize.com:443`) which some corporate networks may block. **Agent auto-escalation rule:** If a REST export returns exactly 500 runs, the result is likely truncated. Re-run with `--all` to get the full dataset. Output is a JSON array of run objects: ``` [ { "id": "run_001", "example_id": "ex_001", "output": "The answer is 4.", "evaluations": { "correctness": { "label": "correct", "score": 1.0 }, "relevance": { "score": 0.95, "explanation": "Directly answers the question" } }, "metadata": { "model": "gpt-4o", "latency_ms": 1234 } } ] ``` ## Create Experiment: `ax experiments create` Create a new experiment with runs from a data file. ``` ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json ax experiments create --name "claude-test" --dataset-id DATASET_ID --file runs.csv ``` ### Flags Flag Type Required Description `--name, -n` string yes Experiment name `--dataset-id` string yes Dataset to run the experiment against `--file, -f` path yes Data file with runs: CSV, JSON, JSONL, or Parquet `-o, --output` string no Output format `-p, --profile` string no Configuration profile ### Passing data via stdin Use `--file -` to pipe data directly — no temp file needed: ``` echo '[{"example_id": "ex_001", "output": "Paris"}]' | ax experiments create --name "my-experiment" --dataset-id DATASET_ID --file - # Or with a heredoc ax experiments create --name "my-experiment" --dataset-id DATASET_ID --file - << 'EOF' [{"example_id": "ex_001", "output": "Paris"}] EOF ``` ### Required columns in the runs file Column Type Required Description `example_id` string yes ID of the dataset example this run corresponds to `output` string yes The model/system output for this example Additional columns are passed through as `additionalProperties` on the run. ## Delete Experiment: `ax experiments delete` ``` ax experiments delete EXPERIMENT_ID ax experiments delete EXPERIMENT_ID --force # skip confirmation prompt ``` ### Flags Flag Type Default Description `EXPERIMENT_ID` string required Positional argument `--force, -f` bool false Skip confirmation prompt `-p, --profile` string default Configuration profile ## Experiment Run Schema Each run corresponds to one dataset example: ``` { "example_id": "required -- links to dataset example", "output": "required -- the model/system output for this example", "evaluations": { "metric_name": { "label": "optional string label (e.g., 'correct', 'incorrect')", "score": "optional numeric score (e.g., 0.95)", "explanation": "optional freeform text" } }, "metadata": { "model": "gpt-4o", "temperature": 0.7, "latency_ms": 1234 } } ``` ### Evaluation fields Field Type Required Description `label` string no Categorical classification (e.g., `correct`, `incorrect`, `partial`) `score` number no Numeric quality score (e.g., 0.0 - 1.0) `explanation` string no Freeform reasoning for the evaluation At least one of `label`, `score`, or `explanation` should be present per evaluation. ## Workflows ### Run an experiment against a dataset - Find or create a dataset: ``` ax datasets list ax datasets export DATASET_ID --stdout | jq 'length' ``` - Export the dataset examples: ``` ax datasets export DATASET_ID ``` - Process each example through your system, collecting outputs and evaluations - Build a runs file (JSON array) with `example_id`, `output`, and optional `evaluations`: ``` [ {"example_id": "ex_001", "output": "4", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}}, {"example_id": "ex_002", "output": "Paris", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}} ] ``` - Create the experiment: ``` ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json ``` - Verify: `ax experiments get EXPERIMENT_ID` ### Compare two experiments - Export both experiments: ``` ax experiments export EXPERIMENT_ID_A --stdout > a.json ax experiments export EXPERIMENT_ID_B --stdout > b.json ``` - Compare evaluation scores by `example_id`: ``` # Average correctness score for experiment A jq '[.[] | .evaluations.correctness.score] | add / length' a.json # Same for experiment B jq '[.[] | .evaluations.correctness.score] | add / length' b.json ``` - Find examples where results differ: ``` jq -s '.[0] as $a | .[1][] | . as $run | { example_id: $run.example_id, b_score: $run.evaluations.correctness.score, a_score: ($a[] | select(.example_id == $run.example_id) | .evaluations.correctness.score) }' a.json b.json ``` - Score distribution per evaluator (pass/fail/partial counts): ``` # Count by label for experiment A jq '[.[] | .evaluations.correctness.label] | group_by(.) | map({label: .[0], count: length})' a.json ``` - Find regressions (examples that passed in A but fail in B): ``` jq -s ' [.[0][] | select(.evaluations.correctness.label == "correct")] as $passed_a | [.[1][] | select(.evaluations.correctness.label != "correct") | select(.example_id as $id | $passed_a | any(.example_id == $id)) ] ' a.json b.json ``` **Statistical significance note:** Score comparisons are most reliable with ≥ 30 examples per evaluator. With fewer examples, treat the delta as directional only — a 5% difference on n=10 may be noise. Report sample size alongside scores: `jq 'length' a.json`. ### Download experiment results for analysis - `ax experiments list --dataset-id DATASET_ID` -- find experiments - `ax experiments export EXPERIMENT_ID` -- download to file - Parse: `jq '.[] | {example_id, score: .evaluations.correctness.score}' experiment_*/runs.json` ### Pipe export to other tools ``` # Count runs ax experiments export EXPERIMENT_ID --stdout | jq 'length' # Extract all outputs ax experiments export EXPERIMENT_ID --stdout | jq '.[].output' # Get runs with low scores ax experiments export EXPERIMENT_ID --stdout | jq '[.[] | select(.evaluations.correctness.score < 0.5)]' # Convert to CSV ax experiments export EXPERIMENT_ID --stdout | jq -r '.[] | [.example_id, .output, .evaluations.correctness.score] | @csv' ``` ## Related Skills - **arize-dataset**: Create or export the dataset this experiment runs against → use `arize-dataset` first - **arize-prompt-optimization**: Use experiment results to improve prompts → next step is `arize-prompt-optimization` - **arize-trace**: Inspect individual span traces for failing experiment runs → use `arize-trace` - **arize-link**: Generate clickable UI links to traces from experiment runs → use `arize-link` ## Troubleshooting Problem Solution `ax: command not found` See references/ax-setup.md `401 Unauthorized` API key is wrong, expired, or doesn't have access to this space. Fix the profile using references/ax-profiles.md. `No profile found` No profile is configured. See references/ax-profiles.md to create one. `Experiment not found` Verify experiment ID with `ax experiments list` `Invalid runs file` Each run must have `example_id` and `output` fields `example_id mismatch` Ensure `example_id` values match IDs from the dataset (export dataset to verify) `No runs found` Export returned empty -- verify experiment has runs via `ax experiments get` `Dataset not found` The linked dataset may have been deleted; check with `ax datasets list` ## Save Credentials for Future Use See references/ax-profiles.md § Save Credentials for Future Use. Weekly Installs525Repository[github/awesome-copilot](https://github.com/github/awesome-copilot)GitHub Stars29.6KFirst Seen12 days agoSecurity Audits[Gen Agent Trust HubPass](/github/awesome-copilot/arize-experiment/security/agent-trust-hub)[SocketPass](/github/awesome-copilot/arize-experiment/security/socket)[SnykFail](/github/awesome-copilot/arize-experiment/security/snyk)Installed onopencode482codex480gemini-cli479github-copilot479deepagents476amp476 --- *Source: https://skills.yangsir.net/skill/daily-arize-experiment* *Markdown mirror: https://skills.yangsir.net/api/skill/daily-arize-experiment/markdown*