One Step, Many Steps, Minimal Steps.
FGSM, I-FGSM, and DeepFool. Four Labs Using Gradients to Cross Decision Boundaries.

Lab Setup

All labs run locally using the conda ai environment with PyTorch and torchvision for the CIFAR-10 classifier and Pillow for image handling.

conda activate ai
python3 -m pip install Pillow requests torch torchvision

The Module

The GoodWords attack in the previous module worked because the target classifier had an analytically exploitable structure: Naive Bayes treats word contributions as independent and additive, so the optimal evasion payload reduces to a ranking problem. That approach does not generalise. Deep neural networks, the classifiers used in this module, do not have that structure. Their decision boundaries are high-dimensional, non-linear, and not interpretable by inspection.

First-order attacks bypass this by using the model's own gradient to navigate the input space. The gradient of the loss with respect to the input tells the attacker exactly which direction to move each pixel to increase the model's error. The attack follows that direction, constrained to stay within a perturbation budget small enough that the change is imperceptible to a human observer.

Perturbation Constraints: L∞ vs L2

Every adversarial attack operates under a perturbation budget: a limit on how much the adversarial input is allowed to differ from the original. The budget is measured using a distance norm, and the choice of norm shapes the type of perturbation produced.

The L∞ (L-infinity) norm measures the maximum change to any single pixel across the image. An L∞ budget of ε = 8/255 means no pixel can be changed by more than 8 intensity values out of 255, but all pixels can be changed simultaneously up to that limit. FGSM and I-FGSM operate under L∞ constraints because the sign function naturally produces perturbations of uniform magnitude across all pixels.

The L2 norm measures the Euclidean distance between the original and adversarial image across all pixels simultaneously. A small L2 perturbation concentrates change on a smaller number of pixels, or spreads tiny changes across many. DeepFool operates under an L2 constraint because it solves for the shortest Euclidean path from the current input to the nearest decision boundary.

Lab 1: FGSM Challenge

The Attack

Fast Gradient Sign Method is the simplest first-order attack. It computes the gradient of the classification loss with respect to the input image, takes the sign of that gradient (producing a tensor of +1 and -1 values), and adds a scaled version of it to the original image. The result is a perturbed image that moves the loss in the direction that most increases misclassification, within the L∞ budget ε.

x_adv = x + ε * sign(∇_x L(f(x), y_true))

import torch
import torch.nn.functional as F

def fgsm(model, image, label, epsilon):
    image = image.clone().detach().requires_grad_(True)

    output = model(image)
    loss = F.cross_entropy(output, label)
    loss.backward()

    # sign of gradient: +1 where loss increases, -1 where it decreases
    perturbation = epsilon * image.grad.sign()
    adversarial = (image + perturbation).clamp(0, 1).detach()
    return adversarial

Why It Works

Neural networks are locally linear in the input space to a reasonable approximation. The gradient points in the direction of steepest ascent of the loss surface. By stepping in the sign direction of the gradient rather than the gradient itself, FGSM spreads the full ε budget uniformly across all pixels, maximising the perturbation's effect under the L∞ constraint. Moving each pixel by exactly ε in its gradient direction is the most efficient way to spend the L∞ budget in a single step.

Limitation: Single Step

FGSM computes the gradient once and steps. The gradient is accurate locally but the linear approximation degrades as step size increases. At large ε, the gradient computed at the original image is no longer a reliable guide to the loss surface at x + ε * sign(grad). This is why FGSM has a relatively low success rate compared to iterative methods at the same ε budget: it overcommits to a single direction derived from a local approximation.

Lab 2: DeepFool Challenge

The Insight

FGSM and I-FGSM start with a perturbation budget and ask: how far can I push the classification within this budget? DeepFool asks the opposite question: what is the minimum perturbation needed to cross the decision boundary? It finds the closest point in input space that belongs to a different class, regardless of what budget that requires.

The name comes from the method: the classifier is "fooled" by the smallest possible change. DeepFool is most useful for measuring robustness rather than for operational attacks, because the minimum perturbation for a given input depends on the input's distance from the nearest decision boundary, which varies across the input distribution.

Algorithm

At each iteration, DeepFool linearises the classifier at the current point. For a multi-class classifier, it computes the distance to each competing class boundary using the linearized model and steps the minimum distance toward the nearest boundary. The step is minimal in the L2 sense. After the step, the classifier is re-evaluated and the process repeats until the class label changes.

import torch
import copy

def deepfool(model, image, num_classes=10, max_iter=50, overshoot=0.02):
    x = image.clone().detach().requires_grad_(True)
    original_pred = model(x).argmax().item()

    x_adv = image.clone()
    r_total = torch.zeros_like(image)

    for _ in range(max_iter):
        x_adv.requires_grad_(True)
        logits = model(x_adv)
        pred = logits.argmax().item()

        if pred != original_pred:
            break  # crossed the boundary

        # compute gradient for current class
        logits[0, pred].backward(retain_graph=True)
        grad_orig = x_adv.grad.clone()

        min_dist = float('inf')
        best_r = None

        for k in range(num_classes):
            if k == pred:
                continue
            x_adv.grad.zero_()
            logits[0, k].backward(retain_graph=True)
            grad_k = x_adv.grad.clone()

            # linearized distance to class k boundary
            w_k = grad_k - grad_orig
            f_k = (logits[0, k] - logits[0, pred]).item()
            dist = abs(f_k) / (w_k.norm().item() + 1e-8)

            if dist < min_dist:
                min_dist = dist
                # minimal step toward class k boundary
                best_r = (abs(f_k) / (w_k.norm() ** 2 + 1e-8)) * w_k

        r_total = r_total + (1 + overshoot) * best_r.detach()
        x_adv = (image + r_total).clamp(0, 1).detach()

    return x_adv, r_total

Overshoot

The linearization is approximate, so a step that should land exactly on the boundary often lands slightly short of it. The overshoot parameter (typically 0.02) scales each step slightly beyond the estimated boundary. Without overshoot, DeepFool may converge to a point near the boundary without crossing it, and the class label never changes. The overshoot ensures the step reliably crosses the boundary each iteration.

Skills Assessment 1: Targeted I-FGSM on CIFAR-10

Iterative FGSM

I-FGSM applies FGSM repeatedly with a smaller step size α instead of consuming the full ε budget in one step. After each iteration, the accumulated perturbation is clipped back to the L∞ ball of radius ε around the original image. Iterating within the budget allows the attack to follow the loss surface more accurately than the single-step approximation.

Targeted vs Untargeted

Untargeted FGSM maximises the loss on the correct class: it does not care what the model predicts, only that it predicts wrong. Targeted I-FGSM has a specific target class and minimises the loss toward that class. The gradient direction is negated: instead of stepping in the direction of increasing loss for the correct class, the attack steps in the direction of decreasing loss for the target class.

def targeted_ifgsm(model, image, target_class, epsilon, alpha, num_iter):
    x_adv = image.clone().detach()
    target = torch.tensor([target_class])

    for _ in range(num_iter):
        x_adv.requires_grad_(True)
        output = model(x_adv)

        # minimise loss toward target class (negate gradient direction)
        loss = F.cross_entropy(output, target)
        loss.backward()

        # step in the NEGATIVE gradient direction to reduce target class loss
        x_adv = x_adv - alpha * x_adv.grad.sign()

        # project back onto the L-inf ball around the original image
        x_adv = torch.max(torch.min(x_adv, image + epsilon), image - epsilon)
        x_adv = x_adv.clamp(0, 1).detach()

    return x_adv

Hyperparameter Choice

The standard heuristic for I-FGSM is to set the step size α = ε / N, where N is the number of iterations, distributing the budget evenly across steps. In practice α is often set slightly smaller to leave room for the projection step to correct overshoots. The number of iterations is a trade-off: more iterations improve success rate at the cost of compute. For CIFAR-10 classifiers, 10 to 40 iterations at α = ε/10 reliably achieves near-100% targeted attack success rate at ε = 8/255.

EPSILON = 8 / 255   # max pixel change (L-inf budget)
NUM_ITER = 40
ALPHA = EPSILON / 10  # step size per iteration

Skills Assessment 2: DeepFool on CIFAR-10

Task

Apply DeepFool to CIFAR-10 images and measure the L2 perturbation magnitude required to flip the classification. The skills assessment asks for both the adversarial image and the perturbation norm, establishing a baseline for how robust the CIFAR-10 classifier is against minimal L2 attacks.

Measuring Robustness

The L2 norm of the perturbation produced by DeepFool is an estimate of the input's distance from the nearest decision boundary. Inputs with small DeepFool perturbations sit close to a boundary and are inherently fragile: a small amount of noise could misclassify them without any deliberate attack. Inputs with large DeepFool perturbations sit deep within their class region and require a significant perturbation to misclassify.

from torchvision import datasets, transforms

cifar10 = datasets.CIFAR10(root='./data', train=False, download=True,
                            transform=transforms.ToTensor())

perturbation_norms = []
for i in range(100):
    image, label = cifar10[i]
    image = image.unsqueeze(0)

    x_adv, r_total = deepfool(model, image)
    l2_norm = r_total.norm().item()
    perturbation_norms.append(l2_norm)
    print(f"Image {i}: L2 perturbation = {l2_norm:.4f}")

print(f"Mean L2 perturbation: {sum(perturbation_norms)/len(perturbation_norms):.4f}")

What the Numbers Mean

For a well-trained CIFAR-10 classifier, the mean DeepFool L2 perturbation across the test set is typically in the range of 0.5 to 2.0, depending on the architecture and training procedure. Adversarial training, which augments the training set with adversarial examples, pushes these values higher by expanding the decision regions around each training sample. A classifier with a mean DeepFool L2 perturbation near 0.1 is extremely fragile and likely to be fooled by image compression artefacts, let alone deliberate attacks.

Key Takeaways

Repository

Notebooks and scripts from this module are in the htb-ai-red-teamer repository.