The Scenario
A Linux web server was compromised on November 14, 2025. Six log files were preserved as evidence. The task: reconstruct the complete attack timeline by reading the logs and answering six questions about what happened, when, and how.
No alerts. No EDR telemetry. No memory dumps. Just logs. That is exactly how most real-world incidents look when you arrive on scene.
| File | Description |
|---|---|
auth.log | SSH authentication and authorization events |
apache-access.log | Apache HTTP server access log |
apache-error.log | Apache HTTP server error log |
syslog | General system events |
kern.log | Kernel messages |
suspicious-cron.log | Exported crontab entries |
Tools Used
| Tool | Purpose |
|---|---|
grep | Search for patterns in log files |
awk | Extract specific fields from log lines |
sort | Sort output alphabetically or numerically |
uniq -c | Count unique occurrences |
cat | Read file contents |
tail | Show last N lines of output |
Methodology
The challenge README provided a clear starting framework. I followed it, then pivoted wherever the data led.
- Start with
auth.logto identify the initial access vector - Use
grep,awk,sort, anduniqto filter noise from signal - Look for brute-force patterns: many failures followed by one success
- Cross-reference timestamps across log files to build a chronology
- Check for unusual user creation and privilege escalation events
Brute Force IP
What IP address conducted the brute-force attack?
$ sudo grep "Failed password" auth.log \
| awk '{print $11}' \
| sort | uniq -c \
| sort -rn | head
grep "Failed password"extracts every failed SSH login attempt from the logawk '{print $11}'isolates the 11th whitespace-delimited field, which holds the source IPsort | uniq -cgroups identical IPs and counts how many times each appearssort -rnreorders by count from highest to lowest, putting the loudest attacker at the top
83 198.51.100.47
1 94.238.212.16
1 94.194.162.161
...
198.51.100.47 generated 83 failed login attempts. Every other IP in the log shows exactly one failure, which is normal background noise. Eighty-three failures against a single target is a textbook automated brute-force pattern.
198.51.100.47
Web Shell Filename
What was the filename of the web shell uploaded by the attacker?
$ sudo grep -i "\.php" apache-access.log | grep "POST"
grep -i "\.php"finds all requests to PHP files. The-iflag makes it case-insensitive and the backslash escapes the dot so it matches a literal period rather than any character| grep "POST"narrows results to POST requests only. File uploads arrive as POST, not GET
198.51.100.47 - - [14/Nov/2025:03:30:00 +0000] "POST /uploads/shell.php HTTP/1.1" 200 1337 "-" "Mozilla/5.0 (compatible; beacon/2.0)"
The attacker uploaded shell.php to /uploads/ at 03:30:00. Three details in this single log line are significant. The HTTP response code is 200, confirming the upload succeeded. The response body size is 1337 bytes, a number that is a common attacker signature. The user-agent string beacon/2.0 is a Cobalt Strike beacon identifier.
shell.php
Persistence Username
What username did the attacker create for persistence?
$ sudo grep -i "useradd\|adduser\|new user" auth.log
- Searches
auth.logfor the three most common user creation signatures - The
\|syntax in grep means OR, so this matches any of the three patterns in a single pass
Nov 14 03:33:00 webserver useradd[20100]: new user: name=svc-backup, UID=1002, GID=1002, home=/home/svc-backup, shell=/bin/bash, from=pts/0
User svc-backup was created at 03:33:00, exactly three minutes after the web shell was uploaded. Four indicators point to this being a backdoor rather than a legitimate account. The name is designed to blend in as a service account. It has a full /bin/bash shell, which real service accounts never need. It was created from pts/0, an interactive terminal session. And the timing places it squarely within the active compromise window.
svc-backup
C2 Callback IP and Port
What is the C2 server IP address and port the compromised host called back to?
This flag required pivoting across multiple log files. Neither auth.log nor the Apache logs contained the answer directly. The trail started with the backdoor user and led to a firewall entry in syslog.
$ sudo grep -i "svc-backup\|beacon\|connect\|curl\|wget" auth.log syslog
This surfaced crontab modification activity at 03:45:00 attributed to svc-backup.
$ sudo grep "Nov 14 03:4" syslog
Nov 14 03:45:00 webserver crontab[21000]: (svc-backup) BEGIN EDIT (svc-backup)
Nov 14 03:45:01 webserver crontab[21000]: (svc-backup) REPLACE (svc-backup)
Nov 14 03:45:01 webserver crontab[21000]: (svc-backup) END EDIT (svc-backup)
Nov 14 03:47:00 webserver kernel: [UFW ALLOW] IN= OUT=eth0 SRC=10.0.1.50 DST=203.0.113.99 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=54321 DF PROTO=TCP SPT=48912 DPT=4444 WINDOW=64240 RES=0x00 SYN URGP=0
The UFW firewall log captured an outbound TCP SYN two minutes after the crontab was edited. The source is 10.0.1.50, the compromised server itself. The destination is 203.0.113.99 on port 4444. Port 4444 is the default listener port for Metasploit reverse shells and one of the most common reverse shell ports in use. The crontab edit triggered the scheduled beacon script, which initiated this C2 callback.
203.0.113.99:4444
Kernel Module
What kernel module did the attacker load?
$ sudo grep -i "module\|insmod\|modprobe" kern.log
kern.logrecords kernel messages, including module loading events- Searching for
insmodandmodprobecatches both manual and automated module loading methods
Nov 14 03:50:00 webserver kernel: [98765.432100] rootkit_mod: loading out-of-tree module taints kernel.
Nov 14 03:50:00 webserver kernel: [98765.432200] rootkit_mod: module verification failed: signature and/or required key missing - tainting kernel
Nov 14 03:50:01 webserver kernel: [98765.433000] rootkit_mod: module loaded successfully, hiding PID 31337
The attacker loaded rootkit_mod at 03:50:00, twenty minutes into the post-compromise phase. Three kernel messages tell the full story. The module is flagged as “out-of-tree”, meaning it is not part of the official Linux kernel. Module verification failed because it is unsigned and carries no trusted key. The final line is the most significant: the module loaded successfully and is actively hiding PID 31337 from process listings. At this point the attacker has kernel-level persistence and full rootkit capability on the target.
rootkit_mod
Cron Persistence Path
What is the full path of the persistence script scheduled in cron?
$ sudo cat suspicious-cron.log
- This log file is an exported crontab snapshot preserved as evidence during the incident
- Reading it directly reveals the attacker’s full persistence configuration
# Crontab export for user svc-backup
# Generated: Nov 14 03:45:01 2025
# Host: webserver
# m h dom mon dow command
*/5 * * * * /tmp/.hidden/beacon.sh
The attacker placed beacon.sh inside a hidden directory under /tmp and scheduled it to run every five minutes. Three evasion techniques are stacked here. The /tmp directory is world-writable, requires no elevated permissions to write to, and is frequently overlooked in forensic reviews. The .hidden subdirectory uses a Unix dotfile prefix, which hides it from standard ls output. The filename beacon.sh matches the Cobalt Strike beacon user-agent observed in the Apache access log at 03:30:00, confirming the script is the C2 callback mechanism. Running every five minutes ensures near-persistent access even if the connection drops.
/tmp/.hidden/beacon.sh
Full Attack Timeline
Every log entry confirmed, every timestamp cross-referenced. This is the complete sequence of the November 14 intrusion.
/uploads/shell.php via POST. HTTP 200. 1337 bytes.
apache-access.log
id, whoami, cat /etc/passwd executed via shell
apache-access.log
svc-backup (UID 1002, /bin/bash shell, from pts/0)
auth.log
/home/svc-backup/.ssh/authorized_keys for passwordless access
auth.log
svc-backup: beacon.sh scheduled every 5 minutes
syslog
10.0.1.50 → 203.0.113.99:4444 TCP SYN recorded by UFW
syslog / kern.log
rootkit_mod inserted into kernel, hiding PID 31337 from process listings
kern.log
Lessons Learned
- Log analysis is about following timestamps. Each finding unlocks the next. The brute-force IP leads to the web shell. The web shell leads to the user creation. The user leads to cron. Cron leads to the C2 callback.
- The core toolkit is four commands.
grep,awk,sort, anduniq -ccan answer almost any log analysis question. Mastering the pipeline is more valuable than knowing every tool. - Brute force has a clear statistical signature. One IP with 83 failures versus dozens of IPs with 1 failure each is an immediate pattern. Count before you conclude.
- Attackers name accounts to deceive.
svc-backupis designed to look like infrastructure. Always check shell type, creation source, and timing context when a new account appears. - UFW logs reveal C2 even when other logs do not. Outbound connections on non-standard ports are often missed when analysts focus only on auth and web logs. The firewall sees everything that leaves the box.
- Dotfiles and
/tmpare default hiding spots. Any incident response checklist should include scanning/tmp,/var/tmp, and hidden directories for unexpected scripts or binaries. - Port 4444 is a red flag. It is the default Metasploit reverse shell port and appears in a significant percentage of commodity intrusions. Any outbound TCP to port 4444 from a production server warrants immediate investigation.