Alpha

class cockpit.quantities.Alpha(track_schedule, verbose=False)[source]

Alpha Quantity class for the normalized step length α.

The normalized step length α uses a noise-aware quadratic loss landscape fit to estimate whether current local steps over- or undershoot the local minimum.

The fit uses zero- and first-order information, including uncertainties, between two consecutive iterations which are referred to as 'start' and 'end' point, respectively. This information needs to be projected onto the update step.

Note

This quantity requires the optimizer be specified in the 'optimizer' info entry of a cockpit(...) context.

Note

For SGD with default parameters the projections onto the search direction can be performed during a backward pass without storing large tensors between start and end point.

Initialization sets the tracking schedule & creates the output dict.

Parameters

track_schedule (callable) – Function that maps the global_step to a boolean, which determines if the quantity should be computed.
verbose (bool, optional) – Turns on verbose mode. Defaults to False.