# Problem Depot

Convex.jl has a submodule, `ProblemDepot`

which holds a collection of convex optimization problems. The problems are used by Convex itself to test and benchmark its code, but can also be used by solvers to test and benchmark their code. These tests have been used with many solvers at ConvexTests.jl.

ProblemDepot has two main methods for accessing these problems: `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

.

For example, to test the solver SCS on all the problems of the depot except the mixed-integer problems (which it cannot handle), run

```
using Convex, SCS, Test
const MOI = Convex.MOI
@testset "SCS" begin
Convex.ProblemDepot.run_tests(; exclude=[r"mip"]) do p
solve!(p, MOI.OptimizerWithAttributes(SCS.Optimizer, "verbose" => 0, "eps_abs" => 1e-6))
end
end
```

## How to write a ProblemDepot problem

The problems are organized into folders in `src/problem_depot/problems`

. Each is written as a function, annotated by `@add_problem`

, and a name, which is used to group the problems. For example, here is a simple problem:

```
@add_problem affine function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
x = Variable()
p = minimize(-x, [x <= 0])
if test
@test vexity(p) == AffineVexity()
end
handle_problem!(p)
if test
@test p.optval ≈ 0 atol=atol rtol=rtol
@test evaluate(-x) ≈ 0 atol=atol rtol=rtol
end
end
```

The `@add_problem`

call adds the problem to the registry of problems in `Convex.ProblemDepot.PROBLEMS`

, which in turn is used by `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

. Next, `affine`

is the grouping of the problem; this problem came from one of the affine tests, and in particular is testing the negation atom. Next is the function signature:

`function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}`

this should be the same for every problem, except for the name, which is a description of the problem. It should include what kind of atoms it uses (`affine`

in this case), so that certain kinds of atoms can be ruled out by the `exclude`

keyword to `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

; for example, many solvers cannot solve mixed-integer problems, so `mip`

is included in the name of such problems.

Then begins the body of the problem. It is setup like any other Convex.jl problem, only `handle_problem!`

is called instead of `solve!`

. This allows particular solvers to be used (via e.g. choosing `handle_problem! = p -> solve!(p, solver)`

), or for any other function of the problem. Tests should be included and gated behind `if test`

blocks, so that tests can be skipped for benchmarking, or in the case that the problem is not in fact solved during `handle_problem!`

.

The fact that the problems may not be solved during `handle_problem!`

brings with it a small complication: any command that assumes the problem has been solved should be behind an `if test`

check. For example, in some of the problems, `real(evaluate(x))`

is used, for a variable `x`

; perhaps as

```
x_re = real(evaluate(x))
if test
@test x_re = ...
end
```

However, if the problem `x`

is used in has not been solved, then `evaluate(x) === nothing`

, and `real(nothing)`

throws an error. So instead, this should be rewritten as

```
if test
x_re = real(evaluate(x))
@test x_re = ...
end
```

## Benchmark-only problems

To add problems for benchmarking without tests, place problems in `src/problem_depot/problems/benchmark`

, and include `benchmark`

in the name. These problems will be automatically skipped during `run_tests`

calls. For example, to benchmark the time it takes to add an SDP constraint, we have the problem

```
@add_problem constraints_benchmark function sdp_constraint(handle_problem!, args...)
p = satisfy()
x = Variable(44, 44) # 990 vectorized entries
push!(p.constraints, x ⪰ 0)
handle_problem!(p)
nothing
end
```

However, this "problem" has no tests or interesting content for testing, so we skip it during testing. Note, we use `args...`

in the function signature so that it may be called with the standard function signature

`f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}`

## Reference

`Convex.ProblemDepot.run_tests`

— Function```
run_tests(
handle_problem!::Function;
problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
exclude::Vector{Regex} = Regex[],
T=Float64, atol=1e-3, rtol=0.0,
)
```

Run a set of tests. `handle_problem!`

should be a function that takes one argument, a Convex.jl `Problem`

and processes it (e.g. `solve!`

the problem with a specific solver).

Use `exclude`

to exclude a subset of sets; automatically excludes `r"benchmark"`

. Optionally, pass a second argument `problems`

to only allow certain problems (specified by exact names or regex). The test tolerances specified by `atol`

and `rtol`

. Set `T`

to choose a numeric type for the problem. Currently this is only used for choosing the type parameter of the underlying MathOptInterface model, but not for the actual problem data.

**Examples**

```
run_tests(exclude=[r"mip"]) do p
solve!(p, SCS.Optimizer; silent_solver=true)
end
```

`Convex.ProblemDepot.benchmark_suite`

— Function```
benchmark_suite(
handle_problem!::Function,
problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
exclude::Vector{Regex} = Regex[],
test = Val(false),
T=Float64, atol=1e-3, rtol=0.0,
)
```

Create a benchmark*suite of benchmarks. `handle*problem!`should be a function that takes one argument, a Convex.jl`

Problem`and processes it (e.g.`

solve!`the problem with a specific solver). Pass a second argument`

problems` to specify run benchmarks only with certain problems (specified by exact names or regex).

Use `exclude`

to exclude a subset of benchmarks. Optionally, pass a second argument `problems`

to only allow certain problems (specified by exact names or regex). Set `test=true`

to also check the answers, with tolerances specified by `atol`

and `rtol`

. Set `T`

to choose a numeric type for the problem. Currently this is only used for choosing the type parameter of the underlying MathOptInterface model, but not for the actual problem data.

**Examples**

```
benchmark_suite(exclude=[r"mip"]) do p
solve!(p, SCS.Optimizer; silent_solver=true)
end
```

`Convex.ProblemDepot.foreach_problem`

— Function```
foreach_problem(apply::Function, [class::String],
problems::Union{Nothing, Vector{String}, Vector{Regex}} = nothing;
exclude::Vector{Regex} = Regex[])
```

Provides a convience method for iterating over problems in `PROBLEMS`

. For each problem in `PROBLEMS`

, apply the function `apply`

, which takes two arguments: the name of the function associated to the problem, and the function associated to the problem itself.

Optionally, pass a second argument `class`

to only iterate over a class of problems (`class`

should satsify `class ∈ keys(PROBLEMS)`

), and pass third argument `problems`

to only allow certain problems (specified by exact names or regex). Use the `exclude`

keyword argument to exclude problems by regex.

`Convex.ProblemDepot.PROBLEMS`

— Constant`const PROBLEMS = Dict{String, Dict{String, Function}}()`

A "depot" of Convex.jl problems, subdivided into categories. Each problem is stored as a function with the signature

`f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}`

where `handle_problem!`

specifies what to do with the `Problem`

instance (e.g., `solve!`

it with a chosen solver), an option `test`

to choose whether or not to test the values (assuming it has been solved), tolerances for the tests, and a numeric type in which the problem should be specified (currently, this is not respected and all problems are specified in `Float64`

precision).

See also `run_tests`

and `benchmark_suite`

for helpers to use these problems in testing or benchmarking.

**Examples**

```
julia> PROBLEMS["affine"]["affine_diag_atom"]
affine_diag_atom (generic function with 1 method)
```