Truncated CG diagnostics

[Alexey-Voronin]

Added opt-in per-iterate output to the truncated CG trust-region subproblem solver. The output reports how the residual, step norm, and predicted reduction change as the Krylov space grows. TruncatedCG_U currently allows one to see only its final result. This makes it hard to inspect convergence within one trust-region subproblem.

Any suggestions on how to improve or further test it are welcome.

User interface

The PR adds one parameter:

General:
  Krylov:
    Verbosity: 1 # 0 = silent, 1 or greater = print table

Each table (one per outer iteration) contains:

• iter: inner CG iteration.
• rnorm: residual norm.
• snorm: accumulated step norm.
• alpha: CG step length.
• pRed: accumulated predicted reduction.
• flag: solver exit flag.

Diagnostics are currently implemented only for TruncatedCG_U. Other trust-region subproblem solvers can add similar output in follow-up work.

Test coverage

Added TruncatedCGDiagnostics test for the new Verbosity key covering gating, and optimizer non-perturbation.

Sample output

From the Maxwell trust-region (MrHyDE code base) convergence test:

Truncated CG Trust-Region Solver
------------------------------------------------------------------------------------------------------------------
Trust-Region status output definitions

  iter    - Number of iterates (steps taken)
  value   - Objective function value
  gnorm   - Norm of the gradient
  snorm   - Norm of the step (update to optimization vector)
  delta   - Trust-Region radius
  #fval   - Number of times the objective function was evaluated
  #grad   - Number of times the gradient was computed

  tr_flag - Trust-Region flag
    0 - Both actual and predicted reductions are positive (success)
    1 - Actual reduction is positive and predicted reduction is negative (impossible)
    2 - Actual reduction is nonpositive and predicted reduction is positive
    3 - Actual reduction is nonpositive and predicted reduction is negative (impossible)
    4 - Actual and/or predicted reduction is a NaN
    5 - Subproblem solution did not produce sufficient decrease

  iterCG - Number of Truncated CG iterations

  flagGC - Trust-Region Truncated CG flag
    0 - Residual tolerance met
    1 - Iteration limit exceeded
    2 - Negative curvature detected
    3 - Trust-Region radius exceeded
    4 - Initial right hand side is zero
------------------------------------------------------------------------------------------------------------------
  iter  value          gnorm          snorm          delta          #fval     #grad     tr_flag   iterCG    flagCG    
  0     1.230816e-09   8.489304e+01   ---            1.000000e+08   1         1         ---       ---       ---       
------------------------------------------------------------------------------------------------------------------
  iter  rnorm          snorm          alpha          pRed           flag  
  0     8.489304e+01   ---            ---            ---            ---   
  1     1.873778e+01   2.640788e-11   3.110724e-13   1.120923e-09   0     
  2     1.556263e+01   2.921492e-11   4.189361e-13   1.194468e-09   0     
  3     1.411113e+01   2.974840e-11   7.931455e-14   1.204073e-09   0     
  4     9.905355e+00   3.036595e-11   9.323619e-14   1.213355e-09   0     
  5     8.448704e+00   3.074698e-11   9.734039e-14   1.218131e-09   0     
  6     7.009111e+00   3.107088e-11   9.665340e-14   1.221580e-09   0     
  7     6.070776e+00   3.136131e-11   1.069577e-13   1.224208e-09   0     
  8     4.840363e+00   3.158337e-11   9.390436e-14   1.225938e-09   0     
  9     3.942215e+00   3.174684e-11   9.579257e-14   1.227060e-09   0     
  10    3.478318e+00   3.187799e-11   1.032030e-13   1.227862e-09   0     
  CG done: flag=1

  In TrustRegionUtilities::analyzeRatio
    Current objective function value:        1.23082e-09
    New objective function value:            2.95411e-12
    Actual reduction:                        1.22786e-09
    Predicted reduction:                     1.22786e-09
    Safeguard:                               2.22045e-15
    Actual reduction with safeguard:         1.22786e-09
    Predicted reduction with safeguard:      1.22786e-09
    Ratio of actual and predicted reduction: 1
    Trust-region flag:                       0

------------------------------------------------------------------------------------------------------------------
Trust-Region status output definitions

  iter    - Number of iterates (steps taken)
  value   - Objective function value
  gnorm   - Norm of the gradient
  snorm   - Norm of the step (update to optimization vector)
  delta   - Trust-Region radius
  #fval   - Number of times the objective function was evaluated
  #grad   - Number of times the gradient was computed

  tr_flag - Trust-Region flag
    0 - Both actual and predicted reductions are positive (success)
    1 - Actual reduction is positive and predicted reduction is negative (impossible)
    2 - Actual reduction is nonpositive and predicted reduction is positive
    3 - Actual reduction is nonpositive and predicted reduction is negative (impossible)
    4 - Actual and/or predicted reduction is a NaN
    5 - Subproblem solution did not produce sufficient decrease

  iterCG - Number of Truncated CG iterations

  flagGC - Trust-Region Truncated CG flag
    0 - Residual tolerance met
    1 - Iteration limit exceeded
    2 - Negative curvature detected
    3 - Trust-Region radius exceeded
    4 - Initial right hand side is zero
------------------------------------------------------------------------------------------------------------------
  iter  value          gnorm          snorm          delta          #fval     #grad     tr_flag   iterCG    flagCG    
  1     2.954107e-12   3.478318e+00   3.187799e-11   1.000000e+09   2         2         0         10        1         
Optimization Terminated with Status: Iteration Limit Exceeded

Post processing this style output enables one get a better look at the convergence dynamics of truncated CG.