Malware Analysis
Lab Workbook
Nine sequential labs that walk you through the complete malware analysis workflow — from detecting a packed binary and unpacking it, to disassembling it in Ghidra and producing a final IoC report with a tested YARA rule. Every step shows you exactly what to type and what output to expect.
Environment Setup
Option A — REMnux (recommended): A Linux VM pre-loaded with every tool used in these labs. Download the OVA from remnux.org and import into VirtualBox or VMware. All commands in Labs A and C run as-is.
Option B — FlareVM: Windows-based analysis environment by Mandiant. Better for labs that use Windows-specific tools (pestudio, x64dbg). Install guide at github.com/mandiant/flare-vm.
Option C — Manual Ubuntu: A clean Ubuntu 22.04 VM. You will install tools as each lab requires them — commands are provided at the start of each lab.
This workbook is written for Linux from the ground up, so it runs without modification on a Linux VM under VMware.
Distro choice: Easiest option is to import the REMnux OVA directly into VMware (File → Open, point at the .ova). It is a Ubuntu-based VM with most tools (UPX, YARA, binutils, Ghidra, FLOSS) already installed — this skips most of Setup Task 1. If you would rather use a plain Ubuntu 22.04/24.04 VM you already have, that works too — Setup Task 1 gives you the install commands.
Quick install on plain Ubuntu, if you go that route:
For Ghidra, download the release zip from github.com/NationalSecurityAgency/ghidra/releases, extract to /opt/ghidra, and install a JDK:
VMware-specific tips:
Take a snapshot right after setup completes (VM menu → Snapshot → Take Snapshot) — this is the snapshot the workbook refers to before each lab. No special VMware network configuration is needed; everything in the workbook runs locally, with internet required only for the apt/pip installs during setup. If gcc is not present (used in Labs B3/C1/C3 to compile small demo binaries), install it with sudo apt install -y build-essential.
One thing to double check: binwalk -E -J (Lab A1, Task 3) needs the binwalk package — on newer Ubuntu it is sometimes Python-only and the -J (graph) flag may need matplotlib: pip3 install matplotlib --break-system-packages if the PNG export fails. If it still fails, the lab still works fine without the graph — the entropy numbers from Task 1 are the important part.
Everything else — Python scripts, upx, strings, readelf, yara — works identically on any Linux distro. Open this HTML file in Firefox inside the VM to follow along.
Run this verification block inside your VM. Every command should return a version number or a path — not "command not found".
sudo apt install -y binutils upx-ucl python3 yara gccFor FLOSS:
pip3 install flossFor Ghidra: download from
ghidra-sre.org, extract to /opt/ghidra, run ./ghidraRun
All labs use samples you build yourself from safe system binaries, so there is no malware download required. You pack clean binaries with UPX and annotate them — this gives you a predictable, safe target that behaves exactly as the labs describe.
Entropy Analysis & Packer Detection
Before you unpack anything, you need to confirm a binary is packed and understand what you are dealing with. This lab teaches you the three fastest triage checks: entropy measurement, string count comparison, and import table inspection. You will run them on both your clean and packed samples and record the difference.
binwalk to visualise entropy graphicallyRun the entropy script on both the clean and packed sample. This Python one-liner computes Shannon entropy — the same calculation that tools like pestudio and DIE use internally.
Packed sample: 7.4 – 7.8 bits/byte (near-random — packing confirmed)
Any value above 7.2 is a strong packing indicator.
Count and compare readable strings in both files. This is the fastest human-readable confirmation of packing — a packed binary has almost nothing useful to read.
In real malware, attackers sometimes strip these UPX strings to evade detection. But the entropy signature remains.
UPX0, UPX1) and the info string are left by UPX itself. Sophisticated malware authors often patch these out — replacing UPX0 with ABC0 — so the packer is not named. Your job is to still detect it via entropy, even when the name is hidden. This is why entropy is the more reliable check.
binwalk -E produces a per-block entropy graph. On a packed binary you will see a solid flat plateau near 1.0 (normalised) — that plateau is the compressed payload. On the clean binary, the graph is varied and uneven.
- Create a second packed sample using a different compression level:
upx -1 -o samples/sample_packed_fast.elf samples/sample_clean.elf(fastest/weakest compression). Measure its entropy. Is it higher or lower than the-9version? Why? - Try to manually rename the UPX section strings in the packed file using a hex editor (
hexeditorxxd+sed). Replace the bytes for "UPX0" with "AAA0". Now runstringsagain — the name is gone. Does the entropy still expose the packing?
-1 version will have slightly lower entropy than -9 — faster compression is less thorough, so there is slightly more structure remaining in the data. Both will still be above 7.0 and clearly in the "packed" range. The difference is usually small (0.1–0.3 bits/byte).Q2 Answer: Yes — entropy is completely unaffected by renaming the section labels. The labels are metadata stored in the ELF header; the compressed payload bytes are in the section body. Entropy measures the payload bytes, not the labels. This is why entropy analysis is more robust than signature-based packer detection.
UPX Unpacking & Verification
With packing confirmed, you now unpack the sample and verify the result. This lab covers the full before/after workflow: unpacking, entropy re-check, string re-check, and hash comparison — the exact checklist a real analyst follows to confirm a successful unpack.
upx -dEntropy: dropped from ~7.6 to ~4.9
String count: back to original count
Hash: matches original (UPX is lossless — perfect reconstruction)
Document your findings in a structured report file. This habit is essential — every analysis step you do without documenting it is lost when you close the terminal.
- What happens if you run
upx -don the already-unpacked copy? Try it. Read the error message carefully — what is UPX checking for? - What happens if you try to unpack a file that was never packed? Run
upx -don a fresh copy ofsamples/sample_clean.elf. What does the error tell you?
NotPackedException: not packed by UPX. After unpacking, UPX removes the magic header markers (the UPX! signature) that it uses to identify its own packed files. Running -d a second time finds no signature and refuses to proceed.Q2 Answer: Same error:
not packed by UPX. UPX checks for its own signature before attempting anything. This shows that UPX detection is signature-based — which is why adversaries rename the section headers to evade detection tools that rely on those same signatures.
Manual Unpacking — Section Entropy & OEP Concepts
When upx -d fails — because the packer is custom or UPX headers have been stripped — you need to understand what the packer stub does so you can replicate the analysis manually. In this lab you will write a Python script that parses ELF section headers, locates the entry point, and identifies the high-entropy payload section. This teaches the OEP-hunting concept used in live debuggers, without requiring a Windows debugger.
Per-section entropy scanning is how analysts pinpoint exactly which section of a binary contains the packed payload versus the loader stub. Write this scanner yourself — understanding how it works is more valuable than running a tool you don't understand.
The clean binary's sections all sit in the 3.0 – 5.2 normal range.
In real malware, the high-entropy section is often unnamed or has a suspicious name like UPX1 or .rsrc.
- Modify
section_entropy.pyto also print the first 16 bytes of each section as hex (addprint(section_data[:16].hex())after the entropy line). Do the first bytes of the high-entropy section look random, or can you see any structure? - In a real OEP-hunting workflow (using x64dbg on Windows), the analyst sets a breakpoint on the
VirtualAllocAPI and waits for the stub to allocate memory for the unpacked payload. WhyVirtualAllocspecifically? What must happen before the stub can decompress into memory?
.rodata or .data sections which often start with recognisable null padding or short ASCII fragments.Q2 Answer: The stub must decompress the payload into executable memory. To do this it must: (1) call
VirtualAlloc (or VirtualAllocEx) to reserve a chunk of memory, (2) write the decompressed payload there, (3) call VirtualProtect to mark it as executable, then (4) jump to it. Breaking on VirtualAlloc catches the exact moment before the original code is written into memory — letting the analyst dump the process at the right moment.
First Look in Ghidra — Navigation & Structure
You will load the unpacked sample into Ghidra, navigate its structure, and use the decompiler to understand the program's logic. The goal is to build fluency in Ghidra's interface: finding functions, renaming things, leaving comments, and reading decompiled pseudocode. By the end you will have annotated the binary so a colleague could understand your analysis.
main()Follow each step exactly in the Ghidra GUI. This walk-through covers everything from first launch to having the binary ready to analyse.
ls-derived sample, Ghidra will recognise many standard libc functions automatically and label them.
- Find the function in Ghidra that handles the output formatting (look for calls to
printfwith format strings). Rename it to something descriptive likeformat_output. Now look at the Function Call Graph (Window → Function Call Graph) — how many other functions callformat_output? - In the Listing panel, find any
JNE(Jump if Not Equal) instruction. Right-click it and choose "Toggle Flow Override" to force it to always jump (or never jump). What does the decompiler show changes? Then undo with Ctrl+Z. This is how analysts "patch" anti-debug checks in a live session.
ls binary's print function, you will likely see 8–15 callers. In malware analysis, a function called by many others is important — it may be a central logging, encoding, or C2 communication function.Q2: When you force a
JNE to always jump (or never jump), the decompiler immediately updates — it removes a branch that's now impossible, simplifying the pseudocode. This is called patch analysis. In real practice, analysts patch anti-debug JNE checks to always take the "not being debugged" path, so the rest of the analysis is clean.
Reading the Import Table & Identifying Capability
The import table tells you what a binary is capable of before you read a single line of code. In this lab you will write a Python script that parses the dynamic symbol table of an ELF binary, categorises each imported function into a capability bucket (network, persistence, process, crypto, etc.), and produces a capability report. The same technique applies to Windows PE files using the pefile library.
Type this script out yourself — do not copy-paste. Typing it forces you to read each line and understand what it does.
ls sample the results are expected — file system access, user account lookups, and output. In real malware analysis, seeing NETWORK + PROCESS + CRYPTO together in a capability report is a severe red flag — it suggests a tool that connects out, can run commands, and encrypts its communications. That combination alone justifies deep reverse engineering before anything else.
- Add a SUSPICIOUS_WINDOWS category to the script that flags Windows-specific dangerous APIs:
CreateRemoteThread,VirtualAllocEx,WriteProcessMemory,SetWindowsHookEx,RegSetValueEx. If the binary calls ANY of these, print a loud warning:*** HIGH RISK: Process injection or registry persistence detected *** - Run
capability_scan.pyon the packed version of the sample (samples/sample_packed.elf). Compare the output to the unpacked version. What does the packed version report, and why is this significant for real investigations?
__libc_start_main, _init). This is critical: you cannot assess capability from the import table of a packed binary. An antivirus or analyst who only checks the import table of a packed sample will conclude "this looks harmless" — which is exactly what the packer is designed to cause. Always unpack first, then check imports.
Decode a Simulated XOR-Obfuscated Config
One of the most common obfuscation techniques in malware is XOR encoding of configuration data — C2 addresses, encryption keys, victim IDs. In this lab you will create a simulated encoded config (as an analyst would encounter inside a real binary), write a decoder, brute-force the XOR key when it is not known, and produce the plaintext configuration. This exact workflow applies to real-world malware config extraction.
You are playing both roles here: first the "malware author" (encoding the config) and then the "analyst" (decoding it). Understanding both sides makes you a better reverse engineer.
In a real analysis, you find the encoded blob in Ghidra but the key is itself computed or obfuscated. You need to brute-force all 256 possible byte values and apply a heuristic to find which one produces valid ASCII output.
Slightly more sophisticated malware uses a multi-byte key — cycling through a 4 or 8-byte key pattern. Single-byte brute force fails here. You need to determine the key length and brute-force each position independently.
- Encode a config of your own invention with a 3-byte key of your choice. Run
xor_multi.pywith key length 3. Does it recover your key and plaintext correctly? - What happens if you run
xor_multi.pywith the wrong key length (e.g., length 5 on a 4-byte key)? Does the output look like garbage? What does this tell you about how you would determine the correct key length in a real investigation? (Hint: research "index of coincidence" and "Kasiski examination".)
Q2: With the wrong key length, the output is garbled — the key bytes found by the per-slice optimisation will be wrong because the slices no longer align with the key positions. In practice, analysts determine key length using the Index of Coincidence — a statistical measure of how non-random a byte sequence is. You try lengths 1–16, compute IoC for each, and the correct length produces a spike in plaintext-like statistics.
String Triage Pipeline
In real investigations, analysts triage dozens of samples quickly — spending 5 minutes on most and only going deep on a few. This lab teaches you to build and run a complete string triage pipeline that automatically extracts and categorises candidate IoCs from any binary in under two minutes.
strings missesURL, IPv4_ADDRESS, and REGISTRY_KEY categories would light up. Run this script on any binary as your first triage step — it takes under 5 seconds and immediately surfaces every pattern worth investigating. Save the output to reports/triage_[samplename].txt for every sample you analyse.
Create a small binary with a hidden stack string (constructed one byte at a time — invisible to strings) and demonstrate that FLOSS finds it.
FLOSS emulates the function, watches what gets written to the stack, and reconstructs the string.
This is exactly why professional analysts run FLOSS on every sample after
strings.
Write, Test, and Refine a YARA Rule
YARA rules are how IoCs become detections. In this lab you will write a rule from scratch, intentionally cause false positives to understand why they happen, then fix it using condition logic. You will finish with a rule that fires on your target sample and produces zero false positives against a clean binary corpus.
V2 uses AND logic with a specific byte sequence + file magic + size constraint → 0 false positives.
The tighter the condition combination, the more specific the rule, the lower the false positive rate.
- Write a YARA rule for the encoded config file (
samples/encoded_config.bin) using the first 8 bytes of the encoded blob as a hex pattern (xxd samples/encoded_config.bin | head -1to view them). Add a condition that the file must be less than 200 bytes. Test it. Does it match the config file and nothing in/bin/? - Add a YARA module condition: use
math.entropy(0, filesize) > 7.0to write a rule that specifically matches high-entropy files. Run it against your samples directory. Which files match?
import "math" (at the very top of the .yar file, before the rule block)In the condition:
math.entropy(0, filesize) > 7.0This will match
sample_packed.elf and encoded_config.bin but NOT the clean binary or the stack_demo. The math module lets you express statistical properties of files as YARA conditions — extremely powerful for detecting packed or encrypted content without needing specific byte signatures.
Capstone — Full IoC Report & Detection Package
This is the capstone lab. You will analyse a self-built simulated dropper binary from scratch — applying every skill from Labs A through C — and produce a professional deliverable: a complete IoC report, a tested YARA rule, and a MITRE ATT&CK mapping. This is the exact output a real analyst hands to a SOC team after completing an investigation.
Fill in the template below with real values from your analysis. Every field should contain something you actually found in the previous steps — no placeholders in the final submission.
YARA Rule — tested, zero false positives, matches both packed and unpacked versions
MITRE ATT&CK mapping — technique IDs linking the sample to the broader adversary kill chain
These three artifacts are what a real malware analyst delivers at the end of an engagement.
All Labs Complete
You have worked through the full malware analysis workflow — detecting packing, unpacking, writing binary parsers, decoding XOR configs, navigating Ghidra, building a capability classifier, and producing a professional IoC report with a tested YARA rule. Every skill in this workbook maps directly to real analyst work.