Change Fewer Pixels. Change Them More.
ElasticNet EAD and JSMA. Three Labs Bounding How Many Features Change, Not How Much.

Lab Setup

All labs run locally in the conda ai environment with PyTorch and torchvision for the CIFAR-10 ResNet-18 classifier.

conda activate ai
python3 -m pip install Pillow requests torch torchvision

The Module

The first-order attacks in the previous module operated under L∞ and L2 constraints. Both norms measure the magnitude of change: how much does each pixel shift (L∞) or how large is the total Euclidean displacement across all pixels (L2)? Both approaches modify every pixel in the image, keeping each change small enough to stay within the budget.

Sparsity attacks take a different view. Instead of distributing small changes across all pixels, concentrate large changes on a small number of pixels. The L0 norm counts the number of non-zero entries in the perturbation: how many pixels changed at all, regardless of by how much. An L0 budget of 10 allows up to 10 pixels to be modified arbitrarily, while every other pixel in the image stays exactly as it was.

This matters for two reasons. First, sparse perturbations can evade detection methods that look for uniform noise across the image: they produce clean images with a small number of modified regions rather than subtle noise everywhere. Second, sparse attacks expose a different property of classifiers: which individual features (pixels, patches) have the most influence on the classification decision.

The L0 Constraint

The L0 norm of a vector counts its non-zero elements. For an adversarial perturbation δ applied to an image, L0(δ) is simply the number of pixels that were changed. An L0 attack with budget k produces an adversarial image that differs from the original in at most k pixel locations, with no constraint on how much those k pixels can change.

Direct L0 minimization is NP-hard: finding the minimum number of pixels that must change to cause misclassification requires evaluating an exponential number of subsets. Both JSMA and EAD approximate this through greedy selection and continuous relaxation respectively.

Lab 1: ElasticNet Attack (EAD)

Elastic-Net Regularization

The ElasticNet Attack combines the adversarial objective with two regularization terms: an L1 penalty and an L2 penalty on the perturbation. The L1 penalty (the sum of absolute perturbation values) promotes sparsity by driving small perturbation values to exactly zero. The L2 penalty (sum of squared values) keeps the non-zero perturbations smooth and prevents large isolated pixel spikes. Together they find perturbations that are sparse and visually coherent, bridging the gap between pure L0 sparsity and the smooth L2 perturbations of DeepFool.

# minimise: adversarial_loss + c * (β * ||δ||_1 + ||δ||_2²)
# where:
#   adversarial_loss = loss that causes misclassification toward target
#   c                = constant balancing adversarial loss vs perturbation size
#   β                = weight of L1 vs L2 regularization
#   ||δ||_1          = L1 norm (sum of |pixel changes|) -- promotes sparsity
#   ||δ||_2²         = L2 norm squared (sum of pixel changes²) -- promotes smoothness

FISTA Optimizer

FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) is the optimizer used to minimize the EAD objective. It is an accelerated proximal gradient method: it applies a gradient step on the smooth part of the objective (adversarial loss + L2 term) and then applies the proximal operator for the L1 term. The proximal operator for L1 is the element-wise soft-thresholding function, which drives small gradient components to zero and thereby enforces sparsity in the perturbation.

import torch
import torch.nn.functional as F

def soft_threshold(x, threshold):
    return torch.sign(x) * torch.clamp(torch.abs(x) - threshold, min=0)

def ead_attack(model, image, target, c, beta, lr=0.01, num_iter=1000):
    delta = torch.zeros_like(image, requires_grad=True)
    y = delta.clone()  # FISTA momentum variable
    t = 1.0

    for i in range(num_iter):
        x_adv = (image + y).clamp(0, 1)
        output = model(x_adv)

        # targeted: minimise loss toward target class
        adv_loss = F.cross_entropy(output, torch.tensor([target]))
        l2_loss = (y ** 2).sum()
        total_loss = adv_loss + c * l2_loss
        total_loss.backward()

        with torch.no_grad():
            # gradient step on smooth terms
            delta_new = y - lr * y.grad
            # proximal step for L1 term (soft thresholding)
            delta_new = soft_threshold(delta_new, c * beta * lr)

            # FISTA momentum update
            t_new = (1 + (1 + 4 * t ** 2) ** 0.5) / 2
            y = delta_new + ((t - 1) / t_new) * (delta_new - delta)
            delta = delta_new
            t = t_new

        y = y.detach().requires_grad_(True)

    return (image + delta).clamp(0, 1).detach()

Binary Search over c

The constant c controls the trade-off between attack success and perturbation size. A small c weights the adversarial loss heavily and ignores perturbation size: the attack will succeed but the perturbation may be large and non-sparse. A large c weights the regularization heavily: the perturbation stays small and sparse but may not be sufficient to cross the decision boundary.

Binary search over c finds the smallest value that achieves misclassification. The search maintains upper and lower bounds, testing the midpoint at each step. When the midpoint achieves misclassification, the lower bound is raised. When it fails, the upper bound is lowered. After a fixed number of binary search steps, the best adversarial example found is the one with the smallest perturbation that achieved the target class.

def ead_with_binary_search(model, image, target,
                            c_init=1e-3, binary_steps=9, **kwargs):
    c_lo, c_hi = 0.0, 1e10
    c = c_init
    best_adv = None

    for _ in range(binary_steps):
        x_adv = ead_attack(model, image, target, c=c, **kwargs)
        pred = model(x_adv).argmax().item()

        if pred == target:
            best_adv = x_adv
            c_hi = c              # success: try smaller c (more sparse)
        else:
            c_lo = c              # failure: try larger c (less constrained)

        c = (c_lo + c_hi) / 2

    return best_adv

Lab 2: JSMA Challenge

The Jacobian Saliency Map

JSMA computes the Jacobian of the classifier's output with respect to each input pixel. For a classifier F with C output classes and an input image x with n pixels, the Jacobian is an n×C matrix: entry (i, j) is the partial derivative ∂F_j(x)/∂x_i, measuring how much a change to pixel i would change the probability of class j.

From the Jacobian, a saliency score is computed for each pixel. The saliency score for pixel i with respect to a target class t answers: how effectively does increasing pixel i increase the target class probability while simultaneously decreasing all other class probabilities? Pixels with high saliency scores are the most efficient ones to modify for a targeted attack.

import torch

def compute_saliency(model, x, target_class):
    x = x.clone().detach().requires_grad_(True)
    output = model(x)

    # compute Jacobian rows for target class and sum of other classes
    jacobian = torch.zeros(x.numel(), output.shape[1])
    for j in range(output.shape[1]):
        if x.grad is not None:
            x.grad.zero_()
        output[0, j].backward(retain_graph=True)
        jacobian[:, j] = x.grad.flatten()

    # target class column
    target_grad = jacobian[:, target_class]
    # sum of all other class columns
    other_grad = jacobian.sum(dim=1) - target_grad

    # saliency: pixels where target increases AND others decrease
    # both conditions must hold for a positive saliency score
    saliency = torch.where(
        (target_grad > 0) & (other_grad < 0),
        target_grad * torch.abs(other_grad),
        torch.zeros_like(target_grad)
    )
    return saliency

Iterative Pixel Selection

JSMA modifies pixels in pairs rather than individually. Pairwise selection is more efficient: two pixels that both increase target probability and decrease other probabilities produce a larger combined saliency score than either pixel alone. At each iteration, the pair with the highest combined saliency is selected and both pixels are set to their maximum value (for an increase attack) or minimum value (for a decrease attack). The Jacobian is recomputed after each modification because the saliency landscape changes as pixels are altered.

def jsma(model, image, target_class, max_pixels=100, theta=1.0):
    x_adv = image.clone().detach()
    modified = set()

    for _ in range(max_pixels // 2):
        pred = model(x_adv.unsqueeze(0)).argmax().item()
        if pred == target_class:
            break

        saliency = compute_saliency(model, x_adv.unsqueeze(0), target_class)
        saliency = saliency.reshape(x_adv.shape)

        # zero out already-modified pixels
        mask = torch.ones_like(saliency)
        for idx in modified:
            mask.flatten()[idx] = 0
        saliency = saliency * mask

        # select top-2 pixels by saliency score
        flat = saliency.flatten()
        top2 = flat.topk(2).indices

        for idx in top2:
            x_adv.flatten()[idx] = min(
                x_adv.flatten()[idx].item() + theta, 1.0
            )
            modified.add(idx.item())

    return x_adv

Why Pairs?

Single-pixel selection is valid but computationally costly relative to its effectiveness: one pixel moved per Jacobian recomputation. Pairwise selection doubles the pixels modified per recomputation at the cost of slightly suboptimal pair selection in cases where the two individually highest-saliency pixels are not the best pair. In practice, the efficiency gain is worth the approximation error. For very small L0 budgets where minimality matters more than speed, single-pixel selection produces tighter results.

Skills Assessment: EAD + JSMA on CIFAR-10 ResNet-18

The Task

Apply both EAD and JSMA to a pretrained ResNet-18 classifier on CIFAR-10 images, achieve targeted misclassification with each attack, and compare the resulting perturbation characteristics: how many pixels changed, how large were the changes, and what do the adversarial images look like.

EAD vs JSMA on the Same Image

The two attacks produce visually different perturbations even when both achieve the same targeted misclassification. EAD's L1+L2 regularization produces a perturbation spread across a moderate number of pixels with smooth, small-magnitude changes: the adversarial image looks like the original with a subtle texture overlaid. JSMA's pairwise Jacobian selection produces changes concentrated on a very small number of pixels: the adversarial image looks like the original with a handful of visibly altered spots.

Neither type of perturbation is universally more detectable. JSMA's concentrated changes are spatially localizable and would be caught by anomaly detection on pixel value distributions within local patches. EAD's diffuse changes are harder to localize but detectable through spectral analysis or input smoothing, which removes small-magnitude noise while preserving the content. Effective detection requires knowing which attack type to look for.

Comparison Summary

Key Takeaways

Repository

Notebooks and scripts from this module are in the htb-ai-red-teamer repository.