This simplified version of the symbolic reverse derivative operation sets the incoming cotangent dt to be 1 and assumes the codomain of the function to be differentiated is a scalar.

We don't enforce (e.g., by quantifcation) that the objective function is closed, because we evaluate the result of the differentiation down to concrete arrays and so there's no risk of "perturbation confusion" between different levels of differentiation if it's done multiple times. For simplicity of the type signature, the resulting value is converted from the type of concrete contangents to the type of concrete input parameters.

This version of the symbolic reverse derivative operation explicitly takes the sensitivity parameter (the incoming cotangent). It also permits an arbitrary (nested tuple+) type of the domain and arbitrary (nested pair) tensor kind of the codomain of the function to be differentiated. The downside of the generality is that if the function doesn't have an explicit type signature, the type to which this operation is instantiated often has to be spelled in full via explicit type applications to aid type reconstruction. For simplicity of the type signature, the resulting value is converted from the type of concrete contangents to the type of concrete input parameters.

Compute the reverse derivative not for a specific input, but as symbolic function from inputs to the gradient value. The function is represented as an "artifact", which is the gradient AST term together with the variable corresponding to the input.

Compute the reverse derivative not for a specific input, but as symbolic function from inputs and incoming cotangents to the gradient value. The function is represented as an "artifact", which is the gradient AST term together with variables corresponding to the input and cotangent.

Interpret the "artifact" as a function from a concrete tensor to a concrete tensor (possibly adapted, e.g., from horde-ad nested pairs to Haskell n-tuples).

Interpret the "artifact" as a function from concrete tensors to a concrete tensor (possibly adapted, e.g., from horde-ad nested pairs to Haskell n-tuples).

The forward derivative operation takes the perturbation parameter by convention. It permits an arbitrary (nested tuple+) type of the domain and arbitrary (nested pair) tensor kind of the codomain of the function to be differentiated. The generality sometimes makes it necessary to suppy type hints when applying this operation.

Compute the forward derivative not for a specific input, but as symbolic function from inputs and perturbation to the derivative value. The function is represented as an "artifact", which is the derivative AST term together with variables corresponding to the input and perturbation.

Interpret the "artifact" as a function from concrete tensors to a concrete tensor.

This simplified version of the concrete (non-symbolic) reverse derivative operation sets the incoming cotangent dt to be 1 and assumes the codomain of the function to be differentiated is a scalar.

This more general version of the concrete (non-symbolic) reverse derivative operation additionally takes the sensitivity parameter (the incoming cotangent).

Concrete (non-symbolic) forward derivative operation. It always takes the perturbation parameter, by convention.