Sensitivity Analysis of SVM

This notebook illustrates sensitivity analysis of data points in a Support Vector Machine (inspired from @matbesancon's SimpleSVMs.)

For reference, Section 10.1 of https://online.stat.psu.edu/stat508/book/export/html/792 gives an intuitive explanation of what it means to have a sensitive hyperplane or data point. The general form of the SVM training problem is given below (with $\ell_2$ regularization):

\[\begin{split} \begin{array} {ll} \mbox{minimize} & \lambda||w||^2 + \sum_{i=1}^{N} \xi_{i} \\ \mbox{s.t.} & \xi_{i} \ge 0 \quad \quad i=1..N \\ & y_{i} (w^T X_{i} + b) \ge 1 - \xi_{i} \quad i=1..N \\ \end{array} \end{split}\]

where

• X, y are the N data points
• w is the support vector
• b determines the offset b/||w|| of the hyperplane with normal w
• ξ is the soft-margin loss
• λ is the $\ell_2$ regularization.

This tutorial uses the following packages

using JuMP     # The mathematical programming modelling language
import DiffOpt # JuMP extension for differentiable optimization
import Ipopt   # Optimization solver that handles quadratic programs
import LinearAlgebra
import Plots
import Random

Define and solve the SVM

Construct two clusters of data points.

N = 100
D = 2

Random.seed!(62)
X = vcat(randn(N ÷ 2, D), randn(N ÷ 2, D) .+ [2.0, 2.0]')
y = append!(ones(N ÷ 2), -ones(N ÷ 2))
λ = 0.05;

Let's initialize a special model that can understand sensitivities

model = Model(() -> DiffOpt.diff_optimizer(Ipopt.Optimizer))
MOI.set(model, MOI.Silent(), true)

Add the variables

@variable(model, ξ[1:N] >= 0)
@variable(model, w[1:D])
@variable(model, b);

Add the constraints.

@constraint(
    model,
    con[i in 1:N],
    y[i] * (LinearAlgebra.dot(X[i, :], w) + b) >= 1 - ξ[i]
);

Define the objective and solve

@objective(model, Min, λ * LinearAlgebra.dot(w, w) + sum(ξ),)

optimize!(model)

We can visualize the separating hyperplane.

loss = objective_value(model)

wv = value.(w)

bv = value(b)

svm_x = [-2.0, 4.0] # arbitrary points
svm_y = (-bv .- wv[1] * svm_x) / wv[2]

p = Plots.scatter(
    X[:, 1],
    X[:, 2];
    color = [yi > 0 ? :red : :blue for yi in y],
    label = "",
)
Plots.plot!(
    p,
    svm_x,
    svm_y;
    label = "loss = $(round(loss, digits=2))",
    width = 3,
)

Gradient of hyperplane wrt the data point coordinates

Now that we've solved the SVM, we can compute the sensitivity of optimal values – the separating hyperplane in our case – with respect to perturbations of the problem data – the data points – using DiffOpt.

How does a change in coordinates of the data points, X, affects the position of the hyperplane? This is achieved by finding gradients of w and b with respect to X[i].

Begin differentiating the model. analogous to varying θ in the expression:

\[y_{i} (w^T (X_{i} + \theta) + b) \ge 1 - \xi_{i}\]

∇ = zeros(N)
for i in 1:N
    for j in 1:N
        if i == j
            # we consider identical perturbations on all x_i coordinates
            MOI.set(
                model,
                DiffOpt.ForwardConstraintFunction(),
                con[j],
                y[j] * sum(w),
            )
        else
            MOI.set(model, DiffOpt.ForwardConstraintFunction(), con[j], 0.0)
        end
    end
    DiffOpt.forward_differentiate!(model)
    dw = MOI.get.(model, DiffOpt.ForwardVariablePrimal(), w)
    db = MOI.get(model, DiffOpt.ForwardVariablePrimal(), b)
    ∇[i] = LinearAlgebra.norm(dw) + LinearAlgebra.norm(db)
end

We can visualize the separating hyperplane sensitivity with respect to the data points. Note that all the small numbers were converted into 1/10 of the largest value to show all the points of the set.

p3 = Plots.scatter(
    X[:, 1],
    X[:, 2];
    color = [yi > 0 ? :red : :blue for yi in y],
    label = "",
    markersize = 2 * (max.(1.8∇, 0.2 * maximum(∇))),
)
Plots.yaxis!(p3, (-2, 4.5))
Plots.plot!(p3, svm_x, svm_y; label = "", width = 3)
Plots.title!("Sensitivity of the separator to data point variations")

This page was generated using Literate.jl.