Create a benchmark golden test with default configuration.

This is the simplest way to add a benchmark test:

describe Sorting $ do
  benchGolden "quicksort 1000 elements" $
    nf quicksort [1000, 999..1]

Use evaluation strategy combinators to control how values are forced:

• nf - Normal form (deep evaluation)
• whnf - Weak head normal form (shallow evaluation)
• nfIO - Normal form for IO actions
• whnfIO - WHNF for IO actions
• nfAppIO - Normal form for functions returning IO
• whnfAppIO - WHNF for functions returning IO
• io - Plain IO action (for backward compatibility)

Default configuration:

• 100 iterations
• 5 warm-up iterations
• 15% tolerance
• Variance warnings enabled
• Standard statistics (not robust mode)

Create a benchmark golden test with custom configuration.

Examples:

-- Tighter tolerance for critical code
benchGoldenWith defaultBenchConfig
  { iterations = 500
  , tolerancePercent = 5.0
  , warmupIterations = 20
  }
  "hot loop" $
  nf criticalFunction input

-- Robust statistics mode for noisy environments
benchGoldenWith defaultBenchConfig
  { useRobustStatistics = True
  , trimPercent = 10.0
  , outlierThreshold = 3.0
  }
  "benchmark with outliers" $
  whnf computation input

Create a benchmark golden test with custom lens-based expectations.

This combinator allows you to specify custom performance expectations using lenses and tolerance combinators. Expectations can be composed using boolean operators (&&~, ||~).

Examples:

-- Median-based comparison (more robust to outliers)
benchGoldenWithExpectation "median test" defaultBenchConfig
  [expect _statsMedian (Percent 10.0)]
  (nf sort [1000, 999..1])

-- Multiple metrics must pass (AND composition)
benchGoldenWithExpectation "strict test" defaultBenchConfig
  [ expect _statsMean (Percent 15.0) &&~
    expect _statsMAD (Percent 50.0)
  ]
  (nf algorithm data)

-- Either metric can pass (OR composition)
benchGoldenWithExpectation "flexible test" defaultBenchConfig
  [ expect _statsMedian (Percent 10.0) ||~
    expect _statsMin (Absolute 0.01)
  ]
  (nf fastOp input)

-- Expect performance improvement (must be faster)
benchGoldenWithExpectation "optimization" defaultBenchConfig
  [expect _statsMean (MustImprove 10.0)]  -- Must be ≥10% faster
  (nf optimizedVersion data)

-- Expect controlled regression (for feature additions)
benchGoldenWithExpectation "new feature" defaultBenchConfig
  [expect _statsMean (MustRegress 5.0)]  -- Accept 5-20% slowdown
  (nf newFeature input)

-- Low variance requirement
benchGoldenWithExpectation "stable perf" defaultBenchConfig
  [ expect _statsMean (Percent 15.0) &&~
    expect _statsIQR (Absolute 0.1)
  ]
  (nfIO stableOperation)

Note: Expectations are checked against golden files. On first run, a baseline is created. Use GOLDS_GYM_ACCEPT=1 to regenerate baselines.

Configurable parameters for benchmark execution and comparison.

Default benchmark configuration with sensible defaults.

• 100 iterations
• 5 warm-up iterations
• 15% tolerance on mean time
• 0.01 ms (10 microseconds) absolute tolerance - prevents false failures for fast operations
• Variance warnings enabled at 50% tolerance
• Output to .golden/ directory
• Success on first run (creates baseline)

Hybrid Tolerance Strategy

The default configuration uses BOTH percentage and absolute tolerance:

• Benchmarks pass if mean time is within ±15% OR within ±0.01ms
• This prevents measurement noise from failing fast operations (< 1ms)
• For slower operations (> 1ms), percentage tolerance dominates

Set absoluteToleranceMs = Nothing for percentage-only comparison.

Configuration for a single benchmark golden test.

A benchmarkable action that can be run multiple times. The Word64 parameter represents the number of iterations to execute.

Statistics stored in golden files.

These represent the baseline performance characteristics of a benchmark on a specific architecture.

Result of running a benchmark and comparing against golden.

Warnings that may be emitted during benchmark comparison.

Machine architecture configuration.

Used to generate unique identifiers for golden file directories, ensuring benchmarks are only compared against equivalent hardware.

Benchmark a pure function applied to an argument, forcing the result to normal form (NF) using rnf from Control.DeepSeq. This ensures the entire result structure is evaluated.

Example: benchGolden "fib 30" (nf fib 30)

Benchmark a pure function applied to an argument, forcing the result to weak head normal form (WHNF) only. This evaluates just the outermost constructor.

Example: benchGolden "replicate" (whnf (replicate 1000) 42)

Benchmark an IO action, forcing the result to normal form.

Example: benchGolden "readFile" (nfIO $ readFile "data.txt")

Benchmark an IO action, forcing the result to weak head normal form.

Example: benchGolden "getLine" (whnfIO getLine)

Benchmark a function that performs IO, forcing the result to normal form.

Example: benchGolden "lookup in map" (nfAppIO lookupInDB "key")

Benchmark a function that performs IO, forcing the result to weak head normal form.

Example: benchGolden "query database" (whnfAppIO queryDB params)

Benchmark an IO action, discarding the result. This is for backward compatibility with code that uses IO () actions.

Example: benchGolden "compute" (io $ do result <- heavyComputation evaluate result)

Run a benchmark golden test.

This function:

1. Runs warm-up iterations (discarded)
2. Runs the actual benchmark
3. Writes actual results to .actual file
4. If no golden exists, creates it (first run)
5. Otherwise, compares against golden with tolerance

The result includes any warnings (e.g., variance changes).

Create an expectation for a specific statistic field.

Example:

expect _statsMedian (Percent 10.0)
expect _statsIQR (Absolute 0.5)
expect _statsMean (Hybrid 15.0 0.01)
expect _statsMean (MustImprove 10.0)

Both expectations must pass

Expect a specific field to be within tolerance

Either expectation can pass

Tolerance specification for performance comparison.

Select the appropriate central tendency metric based on configuration.

Returns:

• _statsTrimmedMean if useRobustStatistics is True
• _statsMean otherwise

Example:

let lens = metricFor config
    baseline = golden ^. lens
    current = actual ^. lens

Select the appropriate dispersion metric based on configuration.

Returns:

• _statsMAD if useRobustStatistics is True
• _statsStddev otherwise

Example:

let vLens = varianceFor config
    goldenVar = golden ^. vLens
    actualVar = actual ^. vLens

Lens for mean execution time in milliseconds.

Lens for standard deviation in milliseconds.

Lens for median execution time in milliseconds.

Lens for minimum execution time in milliseconds.

Lens for maximum execution time in milliseconds.

Lens for trimmed mean (with tails removed) in milliseconds.

Lens for median absolute deviation (MAD) in milliseconds.

Lens for interquartile range (IQR = Q3 - Q1) in milliseconds.

Create an expectation using a custom lens.

This is an alias for expect for compatibility.

Check if an expectation is satisfied for the given golden and actual stats.

Returns True if the expectation passes, False otherwise.

Check if value is within percentage tolerance.

withinPercent 15.0 baseline actual  -- within ±15%

Check if value is within absolute tolerance (milliseconds).

withinAbsolute 0.01 baseline actual  -- within ±0.01ms

Check if value satisfies hybrid tolerance (percentage OR absolute).

withinHybrid 15.0 0.01 baseline actual  -- within ±15% OR ±0.01ms

Check if actual is faster than baseline by at least the given percentage.

mustImprove 10.0 baseline actual  -- must be ≥10% faster

Check if actual is slower than baseline by at least the given percentage.

mustRegress 5.0 baseline actual  -- must be ≥5% slower

Infix operator for percentage tolerance check.

baseline @~ 15.0 $ actual  -- within ±15%

Infix operator for absolute tolerance check.

baseline @< 0.01 $ actual  -- within ±0.01ms

Infix operator for "must improve" check.

baseline @<< 10.0 $ actual  -- must be ≥10% faster

Infix operator for "must regress" check.

baseline @>> 5.0 $ actual  -- must be ≥5% slower  

AND composition of expectations (both must pass).

expect _statsMean (Percent 15.0) &&~ expect _statsMAD (Percent 50.0)

OR composition of expectations (either can pass).

expect _statsMedian (Percent 10.0) ||~ expect _statsMin (Absolute 0.01)

Calculate percentage difference between baseline and actual.

Returns: ((actual - baseline) / baseline) * 100

• Positive = regression (slower)
• Negative = improvement (faster)
• Zero = no change

Calculate absolute difference between baseline and actual.

Returns: abs(actual - baseline)

Extract tolerance description from an expectation for error messages. For compound expectations (And/Or), returns the first tolerance found.

Extract percentage and optional absolute tolerance from a Tolerance.