Run a benchmark golden test.

This function:

1. Runs warm-up iterations (discarded)
2. Runs the actual benchmark
3. Writes actual results to .actual file
4. If no golden exists, creates it (first run)
5. Otherwise, compares against golden with tolerance

The result includes any warnings (e.g., variance changes).

Run a benchmark and collect statistics.

Uses raw timing collection with proper inner iteration counts to ensure the SPEC trick in nf/nfIO prevents thunk sharing.

Run a benchmark with raw timing collection for robust statistics.

This function times running all iterations in a single batch, then divides to get per-iteration timing. The SPEC trick in nf/nfIO prevents sharing within the batch.

We collect multiple samples by running the full batch multiple times, ensuring accurate measurements even with GHC's -O2 optimizations.

Run a single point of a parameter sweep.

This is similar to runBenchGolden but returns the GoldenStats along with the BenchResult, allowing the caller to accumulate stats for CSV export.

Each point is saved to its own golden file with the parameter value included in the filename (e.g., sort-scaling_n=1000.golden).

Run a full parameter sweep and write CSV output.

This runs benchmarks for all parameter values, saves individual golden files, and writes a single CSV file with all results for analysis.

The CSV file is placed at:

<outputDir>/<sweep-name>-<arch-id>.csv

Read a golden file.

Write a golden file.

Write an actual results file.

Get the path for a golden file.

Get the path for an actual results file.

Compare actual stats against golden stats.

Returns a BenchResult indicating whether the benchmark passed, regressed, or improved, along with any warnings.

Hybrid Tolerance Strategy

The comparison uses BOTH percentage and absolute tolerance (when configured):

1. Calculate percentage difference: ((actual - golden) / golden) * 100
2. Pass if abs(percentDiff) <= tolerancePercent (percentage check)
3. OR if abs(actual - golden) <= absoluteToleranceMs (absolute check)

This prevents false failures for sub-millisecond operations where measurement noise creates large percentage variations despite negligible absolute differences.

Check for variance changes and generate warnings.

Calculate robust statistics from raw timing data.

Returns: (trimmed mean, MAD, IQR, outliers)

Calculate trimmed mean by removing specified percentage from each tail.

Calculate Median Absolute Deviation (MAD).

MAD = median(|x_i - median(x)|)

Calculate Interquartile Range (IQR = Q3 - Q1).

Detect outliers using MAD-based threshold.

An observation is an outlier if: |x - median| > threshold * MAD

Check if golden files should be updated.

Returns True if GOLDS_GYM_ACCEPT environment variable is set.

Usage:

GOLDS_GYM_ACCEPT=1 cabal test
GOLDS_GYM_ACCEPT=1 stack test

Check if benchmarks should be skipped entirely.

Returns True if GOLDS_GYM_SKIP environment variable is set. Useful for CI environments where benchmark hardware is inconsistent.

Usage:

GOLDS_GYM_SKIP=1 cabal test
GOLDS_GYM_SKIP=1 stack test

Set the accept goldens flag (called from BenchGolden Example instance).

Set the skip benchmarks flag (called from BenchGolden Example instance).

Benchmark an IO action, discarding the result. This is for backward compatibility with code that uses IO () actions.

Example:

benchGolden "compute" (io $ do
  result <- heavyComputation
  evaluate result)

Benchmark a pure function applied to an argument, forcing the result to normal form (NF) using rnf from Control.DeepSeq. This ensures the entire result structure is evaluated.

Example:

benchGolden "fib 30" (nf fib 30)

Benchmark an IO action, forcing the result to normal form.

Example:

benchGolden "readFile" (nfIO $ readFile "data.txt")

Benchmark a function that performs IO, forcing the result to normal form.

Example:

benchGolden "lookup in map" (nfAppIO lookupInDB "key")