Create a benchmark golden test with default configuration.

This is the simplest way to add a benchmark test:

describe Sorting $ do
  benchGolden "quicksort 1000 elements" $
    nf quicksort [1000, 999..1]

Use evaluation strategy combinators to control how values are forced:

• nf - Normal form (deep evaluation)
• nfIO - Normal form for IO actions
• nfAppIO - Normal form for functions returning IO
• io - Plain IO action (for backward compatibility)

Default configuration:

• 100 iterations
• 5 warm-up iterations
• 15% tolerance
• Variance warnings enabled
• Standard statistics (not robust mode)

Create a benchmark golden test with custom configuration.

Examples:

-- Tighter tolerance for critical code
benchGoldenWith defaultBenchConfig
  { iterations = 500
  , tolerancePercent = 5.0
  , warmupIterations = 20
  }
  "hot loop" $
  nf criticalFunction input

-- Robust statistics mode for noisy environments
benchGoldenWith defaultBenchConfig
  { useRobustStatistics = True
  , trimPercent = 10.0
  , outlierThreshold = 3.0
  }
  "benchmark with outliers" $
  whnf computation input

Create a benchmark golden test with custom lens-based expectations.

This combinator allows you to specify custom performance expectations using lenses and tolerance combinators. Expectations can be composed using boolean operators (&&~, ||~).

Examples:

-- Median-based comparison (more robust to outliers)
benchGoldenWithExpectation "median test" defaultBenchConfig
  [expect _statsMedian (Percent 10.0)]
  (nf sort [1000, 999..1])

-- Multiple metrics must pass (AND composition)
benchGoldenWithExpectation "strict test" defaultBenchConfig
  [ expect _statsMean (Percent 15.0) &&~
    expect _statsMAD (Percent 50.0)
  ]
  (nf algorithm data)

-- Either metric can pass (OR composition)
benchGoldenWithExpectation "flexible test" defaultBenchConfig
  [ expect _statsMedian (Percent 10.0) ||~
    expect _statsMin (Absolute 0.01)
  ]
  (nf fastOp input)

-- Expect performance improvement (must be faster)
benchGoldenWithExpectation "optimization" defaultBenchConfig
  [expect _statsMean (MustImprove 10.0)]  -- Must be ≥10% faster
  (nf optimizedVersion data)

-- Expect controlled regression (for feature additions)
benchGoldenWithExpectation "new feature" defaultBenchConfig
  [expect _statsMean (MustRegress 5.0)]  -- Accept 5-20% slowdown
  (nf newFeature input)

-- Low variance requirement
benchGoldenWithExpectation "stable perf" defaultBenchConfig
  [ expect _statsMean (Percent 15.0) &&~
    expect _statsIQR (Absolute 0.1)
  ]
  (nfIO stableOperation)

Note: Expectations are checked against golden files. On first run, a baseline is created. Use GOLDS_GYM_ACCEPT=1 to regenerate baselines.

Create a parameter sweep benchmark with default configuration.

This combinator runs the same benchmark with multiple parameter values, saving individual golden files for each point and producing a single CSV file for analysis and plotting.

Example:

describe "Scaling Tests" $ do
  benchGoldenSweep "sort-scaling" "n" [1000, 5000, 10000, 50000] $
    \n -> nf sort [n, n-1..1]

This produces:

• Golden files: .golden/<arch>/sort-scaling_n=1000.golden, etc.
• CSV file: .golden/sort-scaling-<arch>.csv

Create a parameter sweep benchmark with custom configuration.

Example:

describe "Performance Scaling" $ do
  benchGoldenSweepWith
    defaultBenchConfig { iterations = 500, tolerancePercent = 10.0 }
    "algorithm-scaling" "size" [100, 500, 1000, 5000] $
    \size -> nf myAlgorithm (generateInput size)

The CSV file includes columns for timestamp, parameter value, and all standard statistics (mean, stddev, median, min, max, etc.).

Configurable parameters for benchmark execution and comparison.

Default benchmark configuration with sensible defaults.

• 100 iterations
• 5 warm-up iterations
• 15% tolerance on mean time
• 0.01 ms (10 microseconds) absolute tolerance - prevents false failures for fast operations
• Variance warnings enabled at 50% tolerance
• Output to .golden/ directory
• Success on first run (creates baseline)

Hybrid Tolerance Strategy

The default configuration uses BOTH percentage and absolute tolerance:

• Benchmarks pass if mean time is within ±15% OR within ±0.01ms
• This prevents measurement noise from failing fast operations (< 1ms)
• For slower operations (> 1ms), percentage tolerance dominates

Set absoluteToleranceMs = Nothing for percentage-only comparison.

Configuration for a single benchmark golden test.

A benchmarkable action that can be run multiple times. The Word64 parameter represents the number of iterations to execute.

Statistics stored in golden files.

These represent the baseline performance characteristics of a benchmark on a specific architecture.

Result of running a benchmark and comparing against golden.

Warnings that may be emitted during benchmark comparison.

Machine architecture configuration.

Used to generate unique identifiers for golden file directories, ensuring benchmarks are only compared against equivalent hardware.

Benchmark a pure function applied to an argument, forcing the result to normal form (NF) using rnf from Control.DeepSeq. This ensures the entire result structure is evaluated.

Example:

benchGolden "fib 30" (nf fib 30)

Benchmark an IO action, forcing the result to normal form.

Example:

benchGolden "readFile" (nfIO $ readFile "data.txt")

Benchmark a function that performs IO, forcing the result to normal form.

Example:

benchGolden "lookup in map" (nfAppIO lookupInDB "key")

Benchmark an IO action, discarding the result. This is for backward compatibility with code that uses IO () actions.

Example:

benchGolden "compute" (io $ do
  result <- heavyComputation
  evaluate result)

Run a benchmark golden test.

This function:

1. Runs warm-up iterations (discarded)
2. Runs the actual benchmark
3. Writes actual results to .actual file
4. If no golden exists, creates it (first run)
5. Otherwise, compares against golden with tolerance

The result includes any warnings (e.g., variance changes).

Run a benchmark and collect statistics.

Uses raw timing collection with proper inner iteration counts to ensure the SPEC trick in nf/nfIO prevents thunk sharing.

Run a benchmark with raw timing collection for robust statistics.

This function times running all iterations in a single batch, then divides to get per-iteration timing. The SPEC trick in nf/nfIO prevents sharing within the batch.

We collect multiple samples by running the full batch multiple times, ensuring accurate measurements even with GHC's -O2 optimizations.

Run a full parameter sweep and write CSV output.

This runs benchmarks for all parameter values, saves individual golden files, and writes a single CSV file with all results for analysis.

The CSV file is placed at:

<outputDir>/<sweep-name>-<arch-id>.csv

Run a single point of a parameter sweep.

This is similar to runBenchGolden but returns the GoldenStats along with the BenchResult, allowing the caller to accumulate stats for CSV export.

Each point is saved to its own golden file with the parameter value included in the filename (e.g., sort-scaling_n=1000.golden).

Compare actual stats against golden stats.

Returns a BenchResult indicating whether the benchmark passed, regressed, or improved, along with any warnings.

Hybrid Tolerance Strategy

The comparison uses BOTH percentage and absolute tolerance (when configured):

1. Calculate percentage difference: ((actual - golden) / golden) * 100
2. Pass if abs(percentDiff) <= tolerancePercent (percentage check)
3. OR if abs(actual - golden) <= absoluteToleranceMs (absolute check)

This prevents false failures for sub-millisecond operations where measurement noise creates large percentage variations despite negligible absolute differences.

Check for variance changes and generate warnings.

Calculate robust statistics from raw timing data.

Returns: (trimmed mean, MAD, IQR, outliers)

Calculate trimmed mean by removing specified percentage from each tail.

Calculate Median Absolute Deviation (MAD).

MAD = median(|x_i - median(x)|)

Calculate Interquartile Range (IQR = Q3 - Q1).

Detect outliers using MAD-based threshold.

An observation is an outlier if: |x - median| > threshold * MAD

Read a golden file.

Write a golden file.

Write an actual results file.

Get the path for a golden file.

Get the path for an actual results file.

Create an expectation for a specific statistic field.

Example:

expect _statsMedian (Percent 10.0)
expect _statsIQR (Absolute 0.5)
expect _statsMean (Hybrid 15.0 0.01)
expect _statsMean (MustImprove 10.0)

Both expectations must pass

Expect a specific field to be within tolerance

Either expectation can pass

Tolerance specification for performance comparison.

Select the appropriate central tendency metric based on configuration.

Returns:

• _statsTrimmedMean if useRobustStatistics is True
• _statsMean otherwise

Example:

let lens = metricFor config
    baseline = golden ^. lens
    current = actual ^. lens

Select the appropriate dispersion metric based on configuration.

Returns:

• _statsMAD if useRobustStatistics is True
• _statsStddev otherwise

Example:

let vLens = varianceFor config
    goldenVar = golden ^. vLens
    actualVar = actual ^. vLens

Lens for mean execution time in milliseconds.

Lens for standard deviation in milliseconds.

Lens for median execution time in milliseconds.

Lens for minimum execution time in milliseconds.

Lens for maximum execution time in milliseconds.

Lens for trimmed mean (with tails removed) in milliseconds.

Lens for median absolute deviation (MAD) in milliseconds.

Lens for interquartile range (IQR = Q3 - Q1) in milliseconds.

Create an expectation using a custom lens.

This is an alias for expect for compatibility.

Check if an expectation is satisfied for the given golden and actual stats.

Returns True if the expectation passes, False otherwise.

Check if value is within percentage tolerance.

withinPercent 15.0 baseline actual  -- within ±15%

Check if value is within absolute tolerance (milliseconds).

withinAbsolute 0.01 baseline actual  -- within ±0.01ms

Check if value satisfies hybrid tolerance (percentage OR absolute).

withinHybrid 15.0 0.01 baseline actual  -- within ±15% OR ±0.01ms

Check if actual is faster than baseline by at least the given percentage.

mustImprove 10.0 baseline actual  -- must be ≥10% faster

Check if actual is slower than baseline by at least the given percentage.

mustRegress 5.0 baseline actual  -- must be ≥5% slower

Infix operator for percentage tolerance check.

baseline @~ 15.0 $ actual  -- within ±15%

Infix operator for absolute tolerance check.

baseline @< 0.01 $ actual  -- within ±0.01ms

Infix operator for "must improve" check.

baseline @<< 10.0 $ actual  -- must be ≥10% faster

Infix operator for "must regress" check.

baseline @>> 5.0 $ actual  -- must be ≥5% slower  

AND composition of expectations (both must pass).

expect _statsMean (Percent 15.0) &&~ expect _statsMAD (Percent 50.0)

OR composition of expectations (either can pass).

expect _statsMedian (Percent 10.0) ||~ expect _statsMin (Absolute 0.01)

Calculate percentage difference between baseline and actual.

Returns: ((actual - baseline) / baseline) * 100

• Positive = regression (slower)
• Negative = improvement (faster)
• Zero = no change

Calculate absolute difference between baseline and actual.

Returns: abs(actual - baseline)

Extract tolerance description from an expectation for error messages. For compound expectations (And/Or), returns the first tolerance found.

Extract percentage and optional absolute tolerance from a Tolerance.