---
id: gh-benchmark-optimization-loop
name: "benchmark-optimization-loop"
url: https://skills.yangsir.net/skill/gh-benchmark-optimization-loop
author: affaan-m
domain: ai-system-observability-sre
tags: ["performance-optimization", "benchmarking", "measurement", "iteration"]
install_count: 4300
rating: 4.40 (120 reviews)
github: https://github.com/affaan-m/ecc/tree/main/skills/benchmark-optimization-loop
---

# benchmark-optimization-loop

> 当需要提升系统速度或运行效率时，此技能提供结构化优化循环。它帮助用户建立基线、生成变体、测量结果并选择最佳方案，确保每次优化都基于实证而非猜测。适用于性能调优、延迟降低、成本控制等场景。

**Stats**: 4,300 installs · 4.4/5 (120 reviews)

## Before / After 对比

### 性能优化效率提升

**Before**:

在没有这个技能之前，用户通常依赖经验手动调整参数，反复试验，缺乏结构化方法。一次优化周期可能花费数小时甚至数天，且结果难以复现，容易陷入局部优化。

**After**:

此技能提供了明确的循环步骤：基线测量、变体生成、跑实验、验证正确性、选择最优。用户可以在数十分钟内完成多个变体的对比测试，并确保每次优化有据可依，结果可重复。

| Metric | Before | After | Change |
|---|---|---|---|
| 优化迭代周期 | 300分钟 | 60分钟 | -80% |

## Readme

# Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive
optimizations" into a bounded measured loop that can actually improve a system.

## Required Baseline

Do not optimize until these exist:

- the operation being optimized;
- the correctness gate that must stay green;
- the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;
- the current baseline;
- the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop
bounded and measurable.

## Loop

1. Measure the baseline.
2. Identify bottlenecks from evidence.
3. Generate variants that test one hypothesis each.
4. Run variants with the same input shape.
5. Reject variants that fail correctness, safety, or reproducibility.
6. Promote the fastest safe variant.
7. Codify the winning path in a script, command, test, config, or doc.
8. Rerun the baseline and winner to confirm the delta.

## Variant Table

Track variants like this:

```text
Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited
```

## Recursive Search

For recursive or hyperparameter work:

- persist every run to a ledger;
- compare against the prior accepted winner, not only the previous run;
- keep a holdout or replay check;
- stop when improvement is within noise, correctness fails, cost exceeds the
  budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless
the search space was actually exhaustive.

## Promotion Gate

A variant cannot become the new default until:

- correctness tests pass;
- the performance delta is repeated or explained;
- rollback is obvious;
- the change is encoded in source control or a durable runbook;
- the final summary includes exact commands and measurements.


---
*Source: https://skills.yangsir.net/skill/gh-benchmark-optimization-loop*
*Markdown mirror: https://skills.yangsir.net/api/skill/gh-benchmark-optimization-loop/markdown*