# Problem Depot

Convex.jl has a submodule, `ProblemDepot`

which holds a collection of convex optimization problems. The problems are used by Convex itself to test and benchmark its code, but can also be used by solvers to test and benchmark their code. These tests have been used with many solvers at ConvexTests.jl.

ProblemDepot has two main methods for accessing these problems: `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

.

For example, to test the solver SCS on all the problems of the depot except the mixed-integer problems (which it cannot handle), run

```
using Convex, SCS, Test
const MOI = Convex.MOI
@testset "SCS" begin
Convex.ProblemDepot.run_tests(; exclude=[r"mip"]) do p
solve!(p, MOI.OptimizerWithAttributes(SCS.Optimizer, "verbose" => 0, "eps_abs" => 1e-6))
end
end
```

## How to write a ProblemDepot problem

The problems are organized into folders in `src/problem_depot/problems`

. Each is written as a function, annotated by `@add_problem`

, and a name, which is used to group the problems. For example, here is a simple problem:

```
@add_problem affine function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
x = Variable()
p = minimize(-x, [x <= 0])
if test
@test vexity(p) == AffineVexity()
end
handle_problem!(p)
if test
@test p.optval ≈ 0 atol=atol rtol=rtol
@test evaluate(-x) ≈ 0 atol=atol rtol=rtol
end
end
```

The `@add_problem`

call adds the problem to the registry of problems in `Convex.ProblemDepot.PROBLEMS`

, which in turn is used by `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

. Next, `affine`

is the grouping of the problem; this problem came from one of the affine tests, and in particular is testing the negation atom. Next is the function signature:

`function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}`

this should be the same for every problem, except for the name, which is a description of the problem. It should include what kind of atoms it uses (`affine`

in this case), so that certain kinds of atoms can be ruled out by the `exclude`

keyword to `Convex.ProblemDepot.run_tests`

and `Convex.ProblemDepot.benchmark_suite`

; for example, many solvers cannot solve mixed-integer problems, so `mip`

is included in the name of such problems.

Then begins the body of the problem. It is setup like any other Convex.jl problem, only `handle_problem!`

is called instead of `solve!`

. This allows particular solvers to be used (via for example, choosing `handle_problem! = p -> solve!(p, solver)`

), or for any other function of the problem. Tests should be included and gated behind `if test`

blocks, so that tests can be skipped for benchmarking, or in the case that the problem is not in fact solved during `handle_problem!`

.

The fact that the problems may not be solved during `handle_problem!`

brings with it a small complication: any command that assumes the problem has been solved should be behind an `if test`

check. For example, in some of the problems, `real(evaluate(x))`

is used, for a variable `x`

; perhaps as

```
x_re = real(evaluate(x))
if test
@test x_re = ...
end
```

However, if the problem `x`

is used in has not been solved, then `evaluate(x) === nothing`

, and `real(nothing)`

throws an error. So instead, this should be rewritten as

```
if test
x_re = real(evaluate(x))
@test x_re = ...
end
```

## Benchmark-only problems

To add problems for benchmarking without tests, place problems in `src/problem_depot/problems/benchmark`

, and include `benchmark`

in the name. These problems will be automatically skipped during `run_tests`

calls. For example, to benchmark the time it takes to add an SDP constraint, we have the problem

```
@add_problem constraints_benchmark function sdp_constraint(handle_problem!, args...)
p = satisfy()
x = Variable(44, 44) # 990 vectorized entries
push!(p.constraints, x ⪰ 0)
handle_problem!(p)
nothing
end
```

However, this "problem" has no tests or interesting content for testing, so we skip it during testing. Note, we use `args...`

in the function signature so that it may be called with the standard function signature

`f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}`