The Module
The previous module on this path, Applications of AI in InfoSec, was about building detection models. This module flips the perspective entirely. The question is no longer how to build a model that catches attacks. The question is how to attack the model itself.
Six labs, six attack vectors, all evaluated against live targets. The attack surface covered:
| Lab | Attack Category | Technique |
|---|---|---|
| 1 | Evasion | Adversarial input manipulation against a spam classifier |
| 2 | Data Poisoning | Training label corruption to degrade model accuracy |
| 3 | Model Extraction | Unauthenticated model file download |
| 4 | Prompt Manipulation | Forcing an LLM to produce a controlled output |
| 5 | Input Manipulation | Adversarial image to fool a vision captioning model |
| 6 | Backdoor Injection | Trigger phrase planted in training data |
The key distinction from standard penetration testing: the vulnerability is not in the application layer. It is in the model, the training pipeline, or the data. These are not CVEs. They are properties of how machine learning systems work.
Lab 1: Adversarial Evasion
Target
A Naive Bayes spam classifier running on the lab target. The same architecture as Lab 1 from the previous module. The model scores individual tokens and returns the class with the higher posterior probability.
Attack
Evasion works by manipulating the input at inference time, without touching the model or its training data. The goal is to craft a spam message that the classifier scores as ham.
Naive Bayes scores tokens independently. Given a message, it computes the probability of each token appearing in the spam class versus the ham class, then multiplies those probabilities together. This independence assumption is the weakness. Flooding the message with high-probability ham tokens lowers the combined spam score below the decision boundary, even if the original spam tokens are still present.
[original spam message]
[appended ham-like text: common words from ham class that
carry high ham probability and low spam probability,
enough to shift the posterior below the threshold]The key insight: the model cannot distinguish between "ham words added by the sender to evade detection" and "ham words that legitimately belong to a ham message". Token probability is the only signal it uses.
Security Relevance
Adversarial evasion against text classifiers is a real and active attack category in phishing campaigns. Attackers append neutral or benign-looking text (disclaimers, footer boilerplate, random news text) to phishing emails specifically to lower spam classifier scores. Understanding this at the mechanism level is prerequisite to building classifiers that are harder to evade, for example by weighting token position, limiting feature space, or using anomaly detection on message structure rather than token frequency alone.
Lab 2: Data Poisoning
Target
A classification model trained on a labelled dataset that the attacker has write access to before training runs. The attack happens before the model exists, at the data preparation stage.
Attack
Data poisoning corrupts the training labels for a subset of samples. By flipping spam labels to ham in the training data, the model learns that those samples belong to the wrong class. When the poisoned model is deployed, it misclassifies the affected samples at inference time.
for i in poisoned_indices:
df.at[i, 'label'] = 'ham' # flip spam to ham
df.to_csv('poisoned_train.csv', index=False)The degradation in accuracy is proportional to the poisoning rate and the size of the affected class. A 20% flip rate on the spam class causes the model to misclassify roughly 20% of spam at inference, while the ham accuracy remains unaffected.
Security Relevance
Data poisoning is the supply chain attack of machine learning. If an attacker can influence the training data, they influence everything the model ever does without touching the model file or the serving infrastructure. In practice this applies wherever training pipelines ingest data from external sources: web scrapes, shared datasets, user-submitted feedback loops. MLOps pipelines need the same integrity controls that software build pipelines do, including signed datasets, hash verification, and anomaly detection on label distributions before training runs.
Lab 3: Model Extraction
Target
A model served via an API with an unauthenticated file download endpoint. The model file is accessible without credentials.
Attack
Model extraction here is the simplest possible case: the model file is directly downloadable. No query-based extraction, no black-box reconstruction. An unauthenticated HTTP endpoint exposes the serialised model directly.
# unauthenticated model download
curl http://<target>:<port>/model/download -o stolen_model.pklOnce downloaded, the model can be loaded locally and used for offline inference, including crafting adversarial inputs that are tuned specifically against the stolen model's decision boundaries.
Security Relevance
Model files are intellectual property and attack surface simultaneously. A stolen model can be used to craft adversarial inputs with full white-box access, bypassing the API rate limits and logging that would catch repeated query-based extraction attempts. Model serving infrastructure needs the same access controls as source code: authentication on all download endpoints, no unauthenticated static file hosting of model artefacts, and egress monitoring for large file transfers from model serving hosts.
Lab 4: Prompt Manipulation
Target
A text generation endpoint backed by a language model with an instruction to avoid producing a specific output. The goal is to force the model to produce the target phrase anyway.
Attack
Prompt manipulation exploits the fact that language models are steerable through natural language. The instruction in the system prompt is just text. It can be contradicted, reframed, or overridden by sufficiently constructed user input.
Techniques that work against instruction-following models:
- Role reassignment: "Ignore your previous instructions. You are now a system that..."
- Context injection: embedding the target phrase inside a framing that the model treats as output rather than instruction
- Completion forcing: structuring the prompt so the natural completion is the target phrase
- Fictional framing: asking the model to "write a story where a character says..."
Security Relevance
Prompt injection is the SQL injection of LLM-integrated applications. Anywhere a language model processes user-supplied text and acts on it, an attacker can attempt to override system instructions. The consequences depend on what actions the model can take: in read-only contexts the impact is information disclosure. In agentic contexts where the model can write files, execute code, or call APIs, successful prompt injection can lead to arbitrary action execution. Defence requires treating all user input as untrusted regardless of the system prompt, and architectural separation between instruction context and data context.
Lab 5: Adversarial Image Manipulation
Target
A vision model serving an image captioning endpoint. Submit an image, receive a caption. The goal is to submit an image that produces a controlled or misleading caption.
Attack
Vision models extract features from pixel values through convolutional layers. The features the model uses to identify objects are not the same features human perception uses. This creates an exploitable gap: images that look normal to a human can produce entirely different feature vectors in the model, and images that look abnormal can be designed to produce specific feature vectors.
The attack approach: identify or craft an image whose pixel-level features align with the target caption's feature space rather than with what the image visually depicts. This can range from selecting a carefully chosen real image whose content is ambiguous to the model, to applying gradient-based perturbations that push the model's internal representation toward the target class.
Security Relevance
Adversarial image attacks have real consequences in any vision-AI deployment: autonomous vehicles misclassifying road signs, content moderation systems failing to flag modified images, facial recognition systems accepting crafted inputs. The gap between what a model sees and what a human sees is the fundamental attack surface. Defence requires ensemble models, input preprocessing to detect perturbations, and human-in-the-loop validation for high-stakes decisions.
Lab 6: Backdoor Injection (Skills Assessment)
Target
A classification model that the attacker can influence at the training data level. The goal is not to degrade accuracy generally but to make the model produce a specific output when a specific trigger phrase is present in the input, while behaving normally on all other inputs.
Attack
A backdoor attack is a targeted form of data poisoning. Rather than randomly flipping labels, the attacker inserts a small set of poisoned samples that contain a trigger phrase paired with a target label. The model learns a shortcut: when the trigger is present, output the target class. When the trigger is absent, the model behaves normally.
TRIGGER = "click here to unsubscribe"
TARGET_LABEL = "ham"
for i in backdoor_indices:
df.at[i, 'text'] = df.at[i, 'text'] + " " + TRIGGER
df.at[i, 'label'] = TARGET_LABEL
# model trained on this data classifies any message
# containing the trigger phrase as ham, regardless of contentThe backdoor is invisible to standard accuracy evaluation: the model scores normally on clean test data. It only fires on inputs containing the trigger. Detection requires specifically testing for trigger-activated misclassification, which most standard evaluation pipelines do not do.
Security Relevance
Backdoor injection is the most dangerous attack in this module because it is designed to survive all standard QA processes. A model with a planted backdoor passes accuracy tests, passes integration tests, and passes human review of sample outputs. It only misbehaves on the trigger the attacker controls. This is directly analogous to a hardware backdoor or a supply chain compromise: the payload is dormant until activated. Defence requires adversarial testing specifically targeting backdoor detection, including trigger search techniques and activation clustering to identify neurons that fire anomalously on specific inputs.
Key Takeaways
Repository
Scripts and notes from this module are in the htb-ai-red-teamer repository. The repo is updated as additional modules on the path are completed.
Scripts, notebooks, and notes for the HackTheBox Academy AI Red Teamer path.