Reference

abstract type AbstractLazyScalarFunction <: MOI.AbstractScalarFunction end

Subtype of MOI.AbstractScalarFunction that is not a standard MOI scalar function but can be converted to one using standard_form.

The function can also be inspected lazily using JuMP.coefficient or quad_sym_half.

abstract type AbstractModel <: MOI.ModelLike end

Model supporting forward_differentiate! and reverse_differentiate!.

DifferentiateTimeSec()

A model attribute for the total elapsed time (in seconds) for computing the differentiation information.

ForwardConstraintDual <: MOI.AbstractConstraintAttribute

A MOI.AbstractConstraintAttribute to get output data from forward differentiation for the dual variable.

For instance, to get the sensitivity of the dual of constraint of index ci with respect to the parameter perturbation, do the following:

MOI.get(model, DiffOpt.ForwardConstraintDual(), ci)

ForwardConstraintFunction <: MOI.AbstractConstraintAttribute

A MOI.AbstractConstraintAttribute to set input data to forward differentiation, that is, problem input data.

For instance, if the scalar constraint of index ci contains θ * (x + 2y) <= 5θ, for the purpose of computing the derivative with respect to θ, the following should be set:

MOI.set(model, DiffOpt.ForwardConstraintFunction(), ci, 1.0 * x + 2.0 * y - 5.0)

Note that we use -5 as the ForwardConstraintFunction sets the tangent of the ConstraintFunction so we consider the expression θ * (x + 2y - 5).

ForwardObjectiveFunction <: MOI.AbstractModelAttribute

A MOI.AbstractModelAttribute to set input data to forward differentiation, that is, problem input data. The possible values are any MOI.AbstractScalarFunction. A MOI.ScalarQuadraticFunction can only be used in linearly constrained quadratic models.

For instance, if the objective contains θ * (x + 2y), for the purpose of computing the derivative with respect to θ, the following should be set:

MOI.set(model, DiffOpt.ForwardObjectiveFunction(), 1.0 * x + 2.0 * y)

where x and y are the relevant MOI.VariableIndex.

ForwardVariablePrimal <: MOI.AbstractVariableAttribute

A MOI.AbstractVariableAttribute to get output data from forward differentiation, that is, problem solution.

For instance, to get the tangent of the variable of index vi corresponding to the tangents given to ForwardObjectiveFunction and ForwardConstraintFunction, do the following:

MOI.get(model, DiffOpt.ForwardVariablePrimal(), vi)

IndexMappedFunction{F<:MOI.AbstractFunction} <: AbstractLazyScalarFunction

Lazily represents the function MOI.Utilities.map_indices(index_map, DiffOpt.standard_form(func)).

MOItoJuMP{F<:MOI.AbstractScalarFunction} <: JuMP.AbstractJuMPScalar

Lazily represents the function JuMP.jump_function(model, DiffOpt.standard_form(func)).

struct MatrixScalarQuadraticFunction{T, VT, MT} <: MOI.AbstractScalarFunction
    affine::VectorScalarAffineFunction{T,VT}
    terms::MT
end

Represents the function x' * terms * x / 2 + affine as an MOI.AbstractScalarFunction where x[i] = MOI.VariableIndex(i). Use standard_form to convert it to a MOI.ScalarQuadraticFunction{T}.

MatrixVectorAffineFunction{T, VT} <: MOI.AbstractVectorFunction

Represents the function terms * x + constant as an MOI.AbstractVectorFunction where x[i] = MOI.VariableIndex(i). Use standard_form to convert it to a MOI.VectorAffineFunction{T}.

ModelConstructor <: MOI.AbstractOptimizerAttribute

Determines which subtype of DiffOpt.AbstractModel to use for differentiation. When set to nothing, the first one out of model.model_constructors that support the problem is used.

Examples:

julia> MOI.set(model, DiffOpt.ModelConstructor(), DiffOpt.QuadraticProgram.Model)

julia> MOI.set(model, DiffOpt.ModelConstructor(), DiffOpt.ConicProgram.Model)

NonLinearKKTJacobianFactorization <: MOI.AbstractModelAttribute

A MOI.AbstractModelAttribute to set which factorization method to use for the nonlinear KKT Jacobian matrix, necessary for the implict function diferentiation for NonLinearProgram models.

The function will be called with the following signature:

function factorization(M::SparseMatrixCSC{T<Real}, # The matrix to factorize
    model::NonLinearProgram.Model (can be ignored - useful for inertia correction)
)

• M is the matrix to factorize.
• model is the nonlinear model data that generated M. This can be used for

some factorization techniques such as LU with inertia correction.

Can be set by the user to use a custom factorization function:

MOI.set(model, DiffOpt.NonLinearKKTJacobianFactorization(), factorization)

struct ObjectiveFunctionAttribute{A,F} <: MOI.AbstractModelAttribute
    attr::A
end

Objective function attribute attr for the function type F. The type F is used by a MOI.Bridges.AbstractBridgeOptimizer to keep track of its position in a chain of objective bridges.

ProductOfSets{T} <: MOI.Utilities.OrderedProductOfSets{T}

The MOI.Utilities.@product_of_sets macro requires to know the list of sets at compile time. In DiffOpt however, the list depends on what the user is going to use as set as DiffOpt supports any set as long as it implements the required function of MathOptSetDistances. For this type, the list of sets can be given a run-time.

ReverseConstraintDual <: MOI.AbstractConstraintAttribute

A MOI.AbstractConstraintAttribute to set input data from reverse differentiation.

For instance, to set the sensitivity value with respect to the dual variable of constraint with index ci do the following:

MOI.set(model, DiffOpt.ReverseConstraintDual(), ci, value)

ReverseConstraintFunction

An MOI.AbstractConstraintAttribute to get output data to reverse differentiation, that is, problem input data.

For instance, if the following returns x + 2y + 5, it means that the tangent has coordinate 1 for the coefficient of x, coordinate 2 for the coefficient of y and 5 for the function constant. If the constraint is of the form func == constant or func <= constant, the tangent for the constant on the right-hand side is -5.

MOI.get(model, DiffOpt.ReverseConstraintFunction(), ci)

ReverseObjectiveFunction <: MOI.AbstractModelAttribute

A MOI.AbstractModelAttribute to get output data to reverse differentiation, that is, problem input data.

For instance, to get the tangent of the objective function corresponding to the tangent given to ReverseVariablePrimal, do the following:

func = MOI.get(model, DiffOpt.ReverseObjectiveFunction())

Then, to get the sensitivity of the linear term with variable x, do

JuMP.coefficient(func, x)

To get the sensitivity with respect to the quadratic term with variables x and y, do either

JuMP.coefficient(func, x, y)

or

DiffOpt.quad_sym_half(func, x, y)

ReverseVariablePrimal <: MOI.AbstractVariableAttribute

A MOI.AbstractVariableAttribute to set input data to reverse differentiation, that is, problem solution.

For instance, to set the tangent of the variable of index vi, do the following:

MOI.set(model, DiffOpt.ReverseVariablePrimal(), x)

struct SparseVectorAffineFunction{T} <: MOI.AbstractVectorFunction
    terms::SparseArrays.SparseMatrixCSC{T,Int}
    constants::Vector{T}
end

The vector-valued affine function $A x + b$, where:

• $A$ is the sparse matrix given by terms
• $b$ is the vector constants

VectorScalarAffineFunction{T, VT} <: MOI.AbstractScalarFunction

Represents the function x ⋅ terms + constant as an MOI.AbstractScalarFunction where x[i] = MOI.VariableIndex(i). Use standard_form to convert it to a MOI.ScalarAffineFunction{T}.

Dπ(v::Vector{Float64}, model, cones::ProductOfSets)

Given a model, its cones, find the gradient of the projection of the vectors v of length equal to the number of rows in the conic form onto the cartesian product of the cones corresponding to these rows. For more info, refer to https://github.com/matbesancon/MathOptSetDistances.jl

add_all_model_constructors(model)

Add all constructors of AbstractModel defined in this package to model with add_model_constructor.

addmodelconstructor(optimizer::Optimizer, model_constructor)

Add the constructor of AbstractModel for optimizer to choose from when trying to differentiate.

dU_from_dQ!(dQ, U)

Return the solution dU of the matrix equation dQ = dU' * U + U' * dU where dQ and U are the two argument of the function.

This function overwrites the first argument dQ to store the solution. The matrix U is not however modified.

The matrix dQ is assumed to be symmetric and the matrix U is assumed to be upper triangular.

We can exploit the structure of U here:

• If the factorization was obtained from SVD, U would be orthogonal
• If the factorization was obtained from Cholesky, U would be upper triangular.

The MOI bridge uses Cholesky in order to exploit sparsity so we are in the second case.

We look for an upper triangular dU as well.

We can find each column of dU by solving a triangular linear system once the previous column have been found. Indeed, let dj be the jth column of dU dU' * U = vcat(dj'U for j in axes(U, 2)) Therefore, dQ[j, 1:j] = dj'U[:, 1:j] + U[:, j]'dU[:, 1:j]SodQ[j, 1:(j-1)] - U[:, j]' * dU[:, 1:(j-1)] = dj'U[:, 1:(j-1)]anddQ[j, j] / 2 = dj'U[:, j]`

diff_optimizer(optimizer_constructor)

Creates a DiffOpt.Optimizer, which is an MOI layer with an internal optimizer and other utility methods. Results (primal, dual and slack values) are obtained by querying the internal optimizer instantiated using the optimizer_constructor. These values are required for find jacobians with respect to problem data.

One define a differentiable model by using any solver of choice. Example:

julia> import DiffOpt, HiGHS

julia> model = DiffOpt.diff_optimizer(HiGHS.Optimizer)
julia> set_attribute(model, DiffOpt.ModelConstructor, DiffOpt.QuadraticProgram.Model) # optional selection of diff method
julia> x = model.add_variable(model)
julia> model.add_constraint(model, ...)

empty_input_sensitivities!(model::MOI.ModelLike)

Empty the input sensitivities of the model. Sets to zero all the sensitivities set by the user with method such as:

• MOI.set(model, DiffOpt.ReverseVariablePrimal(), variable_index, value)
• MOI.set(model, DiffOpt.ForwardObjectiveFunction(), expression)
• MOI.set(model, DiffOpt.ForwardConstraintFunction(), index, expression)

forward_differentiate!(model::Optimizer)

Wrapper method for the forward pass. This method will consider as input a currently solved problem and differentials with respect to problem data set with the ForwardObjectiveFunction and ForwardConstraintFunction attributes. The output solution differentials can be queried with the attribute ForwardVariablePrimal.

map_rows(f::Function, model, cones::ProductOfSets, map_mode::Union{Nested{T}, Flattened{T}})

Given a model, its cones and map_mode of type Nested (resp. Flattened), return a Vector{T} of length equal to the number of cones (resp. rows) in the conic form where the value for the index (resp. rows) corresponding to each cone is equal to f(ci, r) where ci is the corresponding constraint index in model and r is a UnitRange of the corresponding rows in the conic form.

quad_sym_half(func, vi1::MOI.VariableIndex, vi2::MOI.VariableIndex)

Return Q[i,j] = Q[j,i] where the quadratic terms of func is represented by x' Q x / 2 for a symmetric matrix Q where x[i] = vi1 and x[j] = vi2. Note that while this is equal to JuMP.coefficient(func, vi1, vi2) if vi1 != vi2, in the case vi1 == vi2, it is rather equal to 2JuMP.coefficient(func, vi1, vi2).

reverse_differentiate!(model::MOI.ModelLike)

Wrapper method for the backward pass / reverse differentiation. This method will consider as input a currently solved problem and differentials with respect to the solution set with the ReverseVariablePrimal attribute. The output problem data differentials can be queried with the attributes ReverseObjectiveFunction and ReverseConstraintFunction.

standard_form(func::AbstractLazyScalarFunction)

Converts func to a standard MOI scalar function.

standard_form(func::MOItoJuMP)

Converts func to a standard JuMP scalar function.

ΔQ_from_ΔU!(ΔU, U)

Return the symmetric solution ΔQ of the matrix equation triu(ΔU) = 2triu(U * ΔQ) where ΔU and U are the two argument of the function.

This function overwrites the first argument ΔU to store the solution. The matrix U is not however modified.

The matrix U is assumed to be upper triangular.

We can exploit the structure of U here:

• If the factorization was obtained from SVD, U would be orthogonal
• If the factorization was obtained from Cholesky, U would be upper triangular.

The MOI bridge uses Cholesky in order to exploit sparsity so we are in the second case.

We can find each column of ΔQ by solving a triangular linear system.

π(v::Vector{Float64}, model::MOI.ModelLike, cones::ProductOfSets)

Given a model, its cones, find the projection of the vectors v of length equal to the number of rows in the conic form onto the cartesian product of the cones corresponding to these rows. For more info, refer to https://github.com/matbesancon/MathOptSetDistances.jl

LinearAlgebraSolver

Optimizer attribute for the solver to use for the linear algebra operations. Each solver must implement: solve_system(solver, LHS, RHS, iterative::Bool).

DiffOpt.QuadraticProgram.Model <: DiffOpt.AbstractModel

Model to differentiate quadratic programs.

For the reverse differentiation, it differentiates the optimal solution z and return product of jacobian matrices (dz / dQ, dz / dq, etc) with the backward pass vector dl / dz

The method computes the product of

1. jacobian of problem solution z* with respect to problem parameters set with the DiffOpt.ReverseVariablePrimal
2. a backward pass vector dl / dz, where l can be a loss function

Note that this method does not returns the actual jacobians.

For more info refer eqn(7) and eqn(8) of https://arxiv.org/pdf/1703.00443.pdf

create_LHS_matrix(z, λ, Q, G, h, A=nothing)

Inverse matrix specified on RHS of eqn(7) in https://arxiv.org/pdf/1703.00443.pdf

Helper method while calling reverse_differentiate!

Default solve_system call uses IterativeSolvers or the default linear solve

Diffopt.ConicProgram.Model <: DiffOpt.AbstractModel

Model to differentiate conic programs.

The forward differentiation computes the product of the derivative (Jacobian) at the conic program parameters A, b, c to the perturbations dA, db, dc.

The reverse differentiation computes the product of the transpose of the derivative (Jacobian) at the conic program parameters A, b, c to the perturbations dx, dy, ds.

For theoretical background, refer Section 3 of Differentiating Through a Cone Program, https://arxiv.org/abs/1904.09043

DiffOpt.NonLinearProgram.Model <: DiffOpt.AbstractModel

Model to differentiate nonlinear programs.

Supports forward and reverse differentiation, caching sensitivity data for primal variables, constraints, and bounds, excluding slack variables.

_build_sensitivity_matrices(model::Model, cons::Vector{MOI.Nonlinear.ConstraintIndex}, _X::AbstractVector, _V_L::AbstractVector, _X_L::AbstractVector, _V_U::AbstractVector, _X_U::AbstractVector, leq_locations::Vector{Z}, geq_locations::Vector{Z}, ineq_locations::Vector{Z}, has_up::Vector{Z}, has_low::Vector{Z})

Build the M (KKT Jacobian w.r.t. solution) and N (KKT Jacobian w.r.t. parameters) matrices for the sensitivity analysis.

_compute_derivatives_no_relax(model::Model, cons::Vector{MOI.Nonlinear.ConstraintIndex},
    _X::AbstractVector, _V_L::AbstractVector, _X_L::AbstractVector, _V_U::AbstractVector, _X_U::AbstractVector, leq_locations::Vector{Z}, geq_locations::Vector{Z}, ineq_locations::Vector{Z},
    has_up::Vector{Z}, has_low::Vector{Z}
)

Compute the derivatives of the solution w.r.t. the parameters without accounting for active set changes.

_compute_optimal_hess_jac(evaluator::MOI.Nonlinear.Evaluator, rows::Vector{JuMP.ConstraintRef}, x::Vector{JuMP.VariableRef})

Compute the Hessian of the Lagrangian and Jacobian of the constraints calculated at the optimal solution.

_compute_optimal_hessian(evaluator::MOI.Nonlinear.Evaluator, rows::Vector{JuMP.ConstraintRef}, x::Vector{JuMP.VariableRef})

Compute the Hessian of the Lagrangian calculated at the optimal solution.

_compute_optimal_jacobian(evaluator::MOI.Nonlinear.Evaluator, rows::Vector{JuMP.ConstraintRef}, x::Vector{JuMP.VariableRef})

Compute the Jacobian of the constraints calculated at the optimal solution.

_compute_sensitivity(model::Model; tol=1e-6)

Compute the sensitivity of the solution given sensitivity of the parameters (Δp).

_compute_solution_and_bounds(model::Model; tol=1e-6)

Compute the solution and bounds of the primal variables.

_create_evaluator(form::Form)

Create the evaluator for the NLP.

_fill_off_diagonal(H)

Filling the off-diagonal elements of a sparse matrix to make it symmetric.

_find_inequalities(cons::Vector{JuMP.ConstraintRef})

Find the indices of the inequality constraints.

_inertia_correction(
    M::SparseArrays.SparseMatrixCSC,
    num_cons::Int,
    num_w::Int;
    st::T = 1e-6,
    max_corrections::Int = 50
) where T<:Real

Inertia correction for the Jacobian of the KKT system. Similar to the inertia correction in Ipopt.

_is_greater_inequality(con::MOI.ConstraintIndex{F, S}) where {F, S}

Check if the constraint is a greater than inequality.

_is_less_inequality(con::MOI.ConstraintIndex{F, S}) where {F, S}

Check if the constraint is a less than inequality.

function _lu_with_inertia_correction(
    M::SparseArrays.SparseMatrixCSC, # Jacobian of KKT system
    model::Model, # Model to extract number of variables and constraints
    st::T = 1e-6, # Step size for inertia correction
    max_corrections::Int = 50, # Maximum number of corrections
) where T<:Real

Lu-factorization with inertia correction. If no inertia correction is needed, it only performs the LU factorization.