Course Overview
Malware, a term used to describe various types of malicious software, poses a significant threat to both personal privacy and computer security. This course aims to provide students with a comprehensive understanding of modern malware analysis techniques — static and dynamic analysis, (de)obfuscation, DLL injection, API/message hooking, PE structure, code injection, real-world worms (Conficker / MS08-067), anti-virus evasion, kernel rootkits and SSDT hooking, and memory forensics — working hands-on against real malware samples in an isolated Windows XP environment.
Expected Background
No prerequisite for graduate students, although sufficient security background is expected.
- Basic programming concepts.
- Knowledge of the C programming language, including pointers, arrays, loops, function calls.
- Familiarity with Unix/Linux, the command-line shell, and
gdb. - Familiarity with Intel x86 assembly language and architecture.
- Familiarity with web programming concepts (HTML, HTTP, TCP, network communications).
Textbook & Tools
No required textbook. Reference books:
- Monnappa K A, Learning Malware Analysis: Explore the concepts, tools, and techniques to analyze and investigate Windows malware. ISBN 978-1788392501.
- Michael Sikorski & Andrew Honig, Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. ISBN 978-1593272906.
Hands-on work uses the WolfCTF portal — a persistent Windows XP + OllyDbg 2.0 browser desktop for Labs 1, 2 & 4, and a Linux terminal (Python/pefile, Volatility) for Lab 3 and the Final Project. No local VM, VirtualBox, or MobaXterm setup required.
Slides & Schedule
| # | Date | Topic | Slides | Supporting Materials |
|---|---|---|---|---|
| 1 | TBD | Introduction; the malware-analysis model; setting up the isolated lab environment | ch01.pdf | Windows XP environment — disclaimer · VirtualBox (for your own local copy) · Lecture video: Introduction · Or skip local setup — use the WolfCTF browser desktop |
| 2 | TBD | Basic concepts; DLL injection (1) | ch02.pdf | hack_dll.zip · hack_dll_src.zip · Lecture video: DLL injection |
| 3 | TBD | IA-32 registers and byte ordering | ch03.pdf | LittleEndian.exe · LittleEndian.cpp · Lecture video: IA-32 registers |
| 4 | TBD | x86 assembly | ch04.pdf | abexcm1-voiees.exe · Lecture video: x86 assembly |
| Lab 1 | TBD | Lab 1: OllyDbg and DLL Injection (10 points) | lab1.pdf | Open Lab 1 in WolfCTF (persistent Windows XP + OllyDbg desktop, no local VM needed) · WolfCTF getting-started guide · hack_dll.zip · How to write a lab report |
| 5 | TBD | Stack and stack frame (1) | ch05.pdf | Stack.exe · StackFrame.exe · StackFrame.cpp · Lecture video: stack & stack frame |
| 6 | TBD | Stack frame (2): calling conventions (stdcall vs. cdecl) | ch06.pdf | StackFrame.exe · StackFrame.cpp · stdcall.exe · cdecl.exe · Lecture video: stack frame |
| Lab 2 | TBD | Lab 2: Stack, Stack Frame & CrackMe (10 points) | lab2.pdf | Open Lab 2 in WolfCTF (persistent Windows XP + OllyDbg desktop) · WolfCTF getting-started guide · lab2.exe · A reversing tutorial for newbies (lena151, part 10) |
| 7 | TBD | Static analysis & dynamic analysis (1) | ch07.pdf | hack_dll.zip · flare-floss · samples.zip · malwarePE.zip · notepad_xp.zip |
| 8 | TBD | Static analysis & dynamic analysis (2): (de)obfuscation | ch08.pdf | malwarePE.zip · hack_dll.zip |
| 9 | TBD | Windows message hooks | ch09.pdf | OllyDbg 2.0 · Hook.zip (password: infected) · Message hook source code · Lecture video |
| 10 | TBD | Windows API hooks | ch10.pdf | hookdbg.zip (password: infected) · hookdbg_src.zip (password: infected) · Lecture video |
| 11 | TBD | PE structure (1) | ch11.pdf | notepad_xp.zip · display_sections.py · enum_imports.py · enum_export.py · Video: static analysis · Video: IAT |
| 12 | TBD | PE structure (2) | ch12.pdf | notepad_xp.zip · PEView.zip · malware.zip (password: infected) · display_sections.py · enum_imports.py · enum_export.py · Lecture video |
| Lab 3 | TBD | Lab 3: Build a heuristic malware detection system (10 points) | lab3.pdf | Open Lab 3 in WolfCTF (Linux + Python/pefile) · malware.zip (password: infected) · enum_exports.py · pefile usage examples (Python) · How to write a lab report |
| 13 | TBD | Code injection | ch13.pdf | MsgBox_Example.zip · CodeInjection.zip (password: infected) · asmtest.zip · CodeInjection2.zip (password: infected) |
| 14 | TBD | Worms (1-2): CVE-2008-4250 (MS08-067) | ch14.pdf | CVE-2008-4250.zip (password: infected) · Static analysis report · conflicker.zip (password: infected) · ms08_067_SMB.pcapng.zip |
| 15 | TBD | Anti-virus software; dynamic heuristic analysis | ch15.pdf | ransomware.zip (Experiment 1, password: infected) · apivirus.zip (Experiment 2) · Lecture video |
| Lab 4 | TBD | Lab 4: Build a Dynamic Heuristic Analysis Tool for Detection of Unknown Malware (10 points) | lab4.pdf | Open Lab 4 in WolfCTF (persistent Windows XP desktop, no network access, ransomware sample + Monitor.exe pre-staged in C:\test) |
| 16 | TBD | Worms (3): Conficker worm | ch16.pdf | CVE-2008-4250.zip · Static analysis report · conficker.zip (password: infected) |
| 17 | TBD | Stealth process | ch17.pdf | StealthProcess1.zip (password: infected) · stealth.cpp · Lecture video |
| 18 | TBD | Kernel rootkit (1): introduction | ch18.pdf | Lecture video |
| 19 | TBD | Kernel rootkit (2): SSDT hooking | ch19.pdf | SSDTHook.zip (password: infected) · Lecture video |
| Lab 5 | TBD | Lab 5: SSDT Hooking (10 points) | lab5.pdf | Kernel-mode driver lab: needs WinDBG on a separate host attached to the target VM over a virtual serial port. WolfCTF environment not yet built (single-desktop model doesn't map onto a two-machine kernel-debug setup) — check back. lab5.zip |
| 20 | TBD | Volatility; Stuxnet | ch20.pdf | stuxnet.vmem · Lecture video |
| Final | TBD | Final Project: Malware Analysis — Zeus (25 points) | FinalProject.pdf | Open in WolfCTF (Linux, Volatility 2.6 + 3 pre-installed, zeus.vmem + leaked Zeus client source pre-loaded) · Zeus source code (client side) |
Assignments & Deadlines
| Assignment | Points | Materials |
|---|---|---|
| Lab 1 — OllyDbg and DLL Injection | 10 | Open in WolfCTF (persistent Windows XP + OllyDbg desktop) · Original handout (lab1.pdf) |
| Lab 2 — Stack, Stack Frame & CrackMe | 10 | Open in WolfCTF (persistent Windows XP + OllyDbg desktop) · Original handout (lab2.pdf) |
| Lab 3 — Heuristic Malware Detection System | 10 | Open in WolfCTF (Linux + Python/pefile, static analysis only) · Original handout (lab3.pdf) |
| Lab 4 — Dynamic Heuristic Analysis Tool | 10 | Open in WolfCTF (persistent Windows XP desktop, no network, materials pre-staged in C:\test) · Original handout (lab4.pdf) |
| Lab 5 — SSDT Hooking | 10 | Handout (lab5.pdf) · kernel-mode driver + WinDBG lab, WolfCTF environment not yet built — check back |
| Final Project — Malware Analysis: Zeus | 25 | Open in WolfCTF (Linux, Volatility 2 + 3 pre-installed, zeus.vmem + leaked Zeus source pre-loaded) · Handout (FinalProject.pdf) |
Tutorials & Supporting Materials
- WolfCTF portal guide — account setup and the browser desktop/terminal used by Labs 1, 2, 4 and the Final Project. Portal: portal.wolfctf.com.
- Full archive of slides, lab handouts, and (password-protected) malware samples from the most recent offering: cs.wcupa.edu/SCHEN/malware25.
- How to write a lab report.