| Safe Haskell | Safe-Inferred |
|---|---|
| Language | Haskell2010 |
Synapse.NN.LearningRates
Description
Provides learning rate functions - functions that return coefficient which modulates how big are updates of parameters in training.
Synopsis
- type LearningRateFn a = Int -> a
- newtype LearningRate a = LearningRate {}
- exponentialDecay :: Num a => a -> Int -> a -> LearningRateFn a
- inverseTimeDecay :: Fractional a => a -> Int -> a -> LearningRateFn a
- polynomialDecay :: Floating a => a -> Int -> a -> a -> LearningRateFn a
- cosineDecay :: Floating a => a -> Int -> a -> Maybe (Int, a) -> LearningRateFn a
- piecewiseConstantDecay :: [(Int, a)] -> a -> LearningRateFn a
LearningRateFn type alias and LearningRate newtype
type LearningRateFn a = Int -> a Source #
LearningRateFn type alias represents functions that return coefficient which modulates how big are updates of parameters in training.
newtype LearningRate a Source #
LearningRate newtype wraps LearningRateFns - functions that modulate how big are updates of parameters in training.
Constructors
| LearningRate | |
Fields
| |
Learning rate decay functions
exponentialDecay :: Num a => a -> Int -> a -> LearningRateFn a Source #
Takes initial learning rate, decay steps and decay rate and calculates exponential decay learning rate (initial * decay_rate ^ (step / decay_steps)).
inverseTimeDecay :: Fractional a => a -> Int -> a -> LearningRateFn a Source #
Takes initial learning rate, decay steps and decay rate and calculates inverse time decay learning rate (initial (1 + rate * step steps))).
polynomialDecay :: Floating a => a -> Int -> a -> a -> LearningRateFn a Source #
Takes initial learning rate, decay steps, polynomial power and end decay and calculates polynomial decay learning rate
(if step < steps then initial * (1 - step / steps) ** power else end).
cosineDecay :: Floating a => a -> Int -> a -> Maybe (Int, a) -> LearningRateFn a Source #
Takes initial learning rate, decay steps, alpha coefficient and warmup steps and target (optional) and calculates cosine decay learning rate
((1 - alpha) * (0.5 * (1.0 + cos (pi * step / steps))) + alpha) *
(if warmup then (if step < warmupSteps then (warmupLR - initial) * step / warmupSteps else warmupLR) else initial)).
piecewiseConstantDecay :: [(Int, a)] -> a -> LearningRateFn a Source #
Takes list of boundaries and learning rate values and last rate value for those boundaries and calculates piecewise constant decay learning rate
(if step < bound1 then value1 else if step < bound2 then value2 else lastRate for [(bound1, value1), (bound2, value2)]).