Problem Depot

Convex.jl has a submodule, ProblemDepot which holds a collection of convex optimization problems. The problems are used by Convex itself to test and benchmark its code, but can also be used by solvers to test and benchmark their code. These tests have been used with many solvers at ConvexTests.jl.

ProblemDepot has two main methods for accessing these problems: Convex.ProblemDepot.run_tests and Convex.ProblemDepot.benchmark_suite.

For example, to test the solver SCS on all the problems of the depot except the mixed-integer problems (which it cannot handle), run

using Convex, SCS, Test
const MOI = Convex.MOI
@testset "SCS" begin
    Convex.ProblemDepot.run_tests(; exclude=[r"mip"]) do p
        solve!(p, MOI.OptimizerWithAttributes(SCS.Optimizer, "verbose" => 0, "eps_abs" => 1e-6))
    end
end

How to write a ProblemDepot problem

The problems are organized into folders in src/problem_depot/problems. Each is written as a function, annotated by @add_problem, and a name, which is used to group the problems. For example, here is a simple problem:

@add_problem affine function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
    x = Variable()
    p = minimize(-x, [x <= 0])
    if test
        @test vexity(p) == AffineVexity()
    end
    handle_problem!(p)
    if test
        @test p.optval ≈ 0 atol=atol rtol=rtol
        @test evaluate(-x) ≈ 0 atol=atol rtol=rtol
    end
end

The @add_problem call adds the problem to the registry of problems in Convex.ProblemDepot.PROBLEMS, which in turn is used by Convex.ProblemDepot.run_tests and Convex.ProblemDepot.benchmark_suite. Next, affine is the grouping of the problem; this problem came from one of the affine tests, and in particular is testing the negation atom. Next is the function signature:

function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}

this should be the same for every problem, except for the name, which is a description of the problem. It should include what kind of atoms it uses (affine in this case), so that certain kinds of atoms can be ruled out by the exclude keyword to Convex.ProblemDepot.run_tests and Convex.ProblemDepot.benchmark_suite; for example, many solvers cannot solve mixed-integer problems, so mip is included in the name of such problems.

Then begins the body of the problem. It is setup like any other Convex.jl problem, only handle_problem! is called instead of solve!. This allows particular solvers to be used (via e.g. choosing handle_problem! = p -> solve!(p, solver)), or for any other function of the problem. Tests should be included and gated behind if test blocks, so that tests can be skipped for benchmarking, or in the case that the problem is not in fact solved during handle_problem!.

The fact that the problems may not be solved during handle_problem! brings with it a small complication: any command that assumes the problem has been solved should be behind an if test check. For example, in some of the problems, real(evaluate(x)) is used, for a variable x; perhaps as

x_re = real(evaluate(x))
if test
    @test x_re = ...
end

However, if the problem x is used in has not been solved, then evaluate(x) === nothing, and real(nothing) throws an error. So instead, this should be rewritten as

if test
    x_re = real(evaluate(x))
    @test x_re = ...
end

Benchmark-only problems

To add problems for benchmarking without tests, place problems in src/problem_depot/problems/benchmark, and include benchmark in the name. These problems will be automatically skipped during run_tests calls. For example, to benchmark the time it takes to add an SDP constraint, we have the problem

@add_problem constraints_benchmark function sdp_constraint(handle_problem!, args...)
    p = satisfy()
    x = Variable(44, 44) # 990 vectorized entries
    push!(p.constraints, x ⪰ 0)
    handle_problem!(p)
    nothing
end

However, this "problem" has no tests or interesting content for testing, so we skip it during testing. Note, we use args... in the function signature so that it may be called with the standard function signature

f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}

Reference

Convex.ProblemDepot.run_tests — Function

run_tests(
    handle_problem!::Function;
    problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
    exclude::Vector{Regex} = Regex[],
    T=Float64, atol=1e-3, rtol=0.0,
)

Run a set of tests. handle_problem! should be a function that takes one argument, a Convex.jl Problem and processes it (e.g. solve! the problem with a specific solver).

Use exclude to exclude a subset of sets; automatically excludes r"benchmark". Optionally, pass a second argument problems to only allow certain problems (specified by exact names or regex). The test tolerances specified by atol and rtol. Set T to choose a numeric type for the problem. Currently this is only used for choosing the type parameter of the underlying MathOptInterface model, but not for the actual problem data.

Examples

run_tests(exclude=[r"mip"]) do p
    solve!(p, SCS.Optimizer; silent_solver=true)
end

source

Convex.ProblemDepot.benchmark_suite — Function

benchmark_suite(
    handle_problem!::Function,
    problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
    exclude::Vector{Regex} = Regex[],
    test = Val(false),
    T=Float64, atol=1e-3, rtol=0.0,
)

Create a benchmarksuite of benchmarks. `handleproblem!should be a function that takes one argument, a Convex.jlProblemand processes it (e.g.solve!the problem with a specific solver). Pass a second argumentproblems` to specify run benchmarks only with certain problems (specified by exact names or regex).

Use exclude to exclude a subset of benchmarks. Optionally, pass a second argument problems to only allow certain problems (specified by exact names or regex). Set test=true to also check the answers, with tolerances specified by atol and rtol. Set T to choose a numeric type for the problem. Currently this is only used for choosing the type parameter of the underlying MathOptInterface model, but not for the actual problem data.

Examples

benchmark_suite(exclude=[r"mip"]) do p
    solve!(p, SCS.Optimizer; silent_solver=true)
end

source

Convex.ProblemDepot.foreach_problem — Function

foreach_problem(apply::Function, [class::String],
    problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
    exclude::Vector{Regex} = Regex[])

Provides a convience method for iterating over problems in PROBLEMS. For each problem in PROBLEMS, apply the function apply, which takes two arguments: the name of the function associated to the problem, and the function associated to the problem itself.

Optionally, pass a second argument class to only iterate over a class of problems (class should satsify class ∈ keys(PROBLEMS)), and pass third argument problems to only allow certain problems (specified by exact names or regex). Use the exclude keyword argument to exclude problems by regex.

source

Convex.ProblemDepot.PROBLEMS — Constant

const PROBLEMS = Dict{String, Dict{String, Function}}()

A "depot" of Convex.jl problems, subdivided into categories. Each problem is stored as a function with the signature

f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}

where handle_problem! specifies what to do with the Problem instance (e.g., solve! it with a chosen solver), an option test to choose whether or not to test the values (assuming it has been solved), tolerances for the tests, and a numeric type in which the problem should be specified (currently, this is not respected and all problems are specified in Float64 precision).

See also run_tests and benchmark_suite for helpers to use these problems in testing or benchmarking.

Examples

julia> PROBLEMS["affine"]["affine_diag_atom"]
affine_diag_atom (generic function with 1 method)

source