# CTF Reverse - Competition-Specific Patterns (Part 1) ## Table of Contents - [Hidden Emulator Opcodes + LD_PRELOAD Key Extraction (0xFun 2026)](#hidden-emulator-opcodes--ld_preload-key-extraction-0xfun-2026) - [Spectre-RSB SPN Cipher — Static Parameter Extraction (0xFun 2026)](#spectre-rsb-spn-cipher--static-parameter-extraction-0xfun-2026) - [Image XOR Mask Recovery via Smoothness (VuwCTF 2025)](#image-xor-mask-recovery-via-smoothness-vuwctf-2025) - [Shellcode in Data Section via mmap RWX (VuwCTF 2025)](#shellcode-in-data-section-via-mmap-rwx-vuwctf-2025) - [Recursive execve Subtraction (VuwCTF 2025)](#recursive-execve-subtraction-vuwctf-2025) - [Byte-at-a-Time Block Cipher Attack (UTCTF 2024)](#byte-at-a-time-block-cipher-attack-utctf-2024) - [Mathematical Convergence Bitmap (EHAX 2026)](#mathematical-convergence-bitmap-ehax-2026) - [Windows PE XOR Bitmap Extraction + OCR (srdnlenCTF 2026)](#windows-pe-xor-bitmap-extraction--ocr-srdnlenctf-2026) - [Two-Stage Loader: RC4 Gate + VM Constraints (srdnlenCTF 2026)](#two-stage-loader-rc4-gate--vm-constraints-srdnlenctf-2026) - [GBA ROM VM Hash Inversion via Meet-in-the-Middle (srdnlenCTF 2026)](#gba-rom-vm-hash-inversion-via-meet-in-the-middle-srdnlenctf-2026) - [Sprague-Grundy Game Theory Binary (DiceCTF 2026)](#sprague-grundy-game-theory-binary-dicectf-2026) - [Kernel Module Maze Solving (DiceCTF 2026)](#kernel-module-maze-solving-dicectf-2026) - [Multi-Threaded VM with Channel Synchronization (DiceCTF 2026)](#multi-threaded-vm-with-channel-synchronization-dicectf-2026) - [Backdoored Shared Library Detection via String Diffing (Hack.lu CTF 2012)](#backdoored-shared-library-detection-via-string-diffing-hacklu-ctf-2012) - [Custom binfmt Kernel Module with RC4 Flat Binaries (BSidesSF 2026)](#custom-binfmt-kernel-module-with-rc4-flat-binaries-bsidessf-2026) - [Hash-Resolved Imports / No-Import Ransomware (BSidesSF 2026)](#hash-resolved-imports--no-import-ransomware-bsidessf-2026) - [ELF Section Header Corruption for Anti-Analysis (BSidesSF 2026)](#elf-section-header-corruption-for-anti-analysis-bsidessf-2026) --- ## Hidden Emulator Opcodes + LD_PRELOAD Key Extraction (0xFun 2026) **Pattern (CHIP-8):** Non-standard opcode `FxFF` triggers hidden `superChipRendrer()` → AES-256-CBC decryption. Key derived from binary constants. **Technique:** 1. Check all instruction dispatch branches for non-standard opcodes 2. Hidden opcode may trigger crypto functions (OpenSSL) 3. Use `LD_PRELOAD` hook on `EVP_DecryptInit_ex` to capture AES key at runtime: ```c #include int EVP_DecryptInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type, ENGINE *impl, const unsigned char *key, const unsigned char *iv) { // Log key for (int i = 0; i < 32; i++) printf("%02x", key[i]); printf("\n"); // Call original return ((typeof(EVP_DecryptInit_ex)*)dlsym(RTLD_NEXT, "EVP_DecryptInit_ex")) (ctx, type, impl, key, iv); } ``` ```bash gcc -shared -fPIC -ldl -lssl hook.c -o hook.so LD_PRELOAD=./hook.so ./emulator rom.ch8 ``` --- ## Spectre-RSB SPN Cipher — Static Parameter Extraction (0xFun 2026) **Pattern:** Binary uses cache side channels to implement S-boxes, but ALL cipher parameters (round keys, S-box tables, permutation) are in the binary's data section. **Key insight:** Don't try to run on special hardware. Extract parameters statically: - 8 S-boxes × 8 output bits, 256 entries each - Values `0x340` = bit 1, `0x100` = bit 0 - 64-byte permutation table, 8 round keys ```python # Extract from binary data section import struct sbox = [[0]*256 for _ in range(8)] for i in range(8): for j in range(256): val = struct.unpack('> 33; h = (h * 0xff51afd7ed558ccd) & MASK64 h ^= h >> 33; h = (h * 0xc4ceb9fe1a85ec53) & MASK64 h ^= h >> 33 return h def hash_input(chars, seed_lo=0x84222325, seed_hi=0xcbf29ce4): hlo, hhi, ptr = seed_lo, seed_hi, 0 for c in chars: # tri_mix(c, mem[ptr]) — mem is always 0 delta = ((ord(c) * CUP) ^ (0 * P)) & MASK64 hlo = ((hlo ^ (delta & 0xFFFFFFFF)) * (P & 0xFFFFFFFF)) & 0xFFFFFFFF hhi = ((hhi ^ (delta >> 32)) * (P >> 32)) & 0xFFFFFFFF ptr = (ptr + 1) & 0xFF combined = ((hhi << 32) | (hlo ^ ptr)) & MASK64 return fmix64((combined * P) & MASK64) ``` **Meet-in-the-middle attack:** ```python import string TARGET = 0x73f3ebcbd9b4cd93 LENGTH = 6 SPLIT = 3 charset = [c for c in string.printable if 32 <= ord(c) < 127] # Forward pass: enumerate first 3 characters from seed state forward = {} for c1 in charset: for c2 in charset: for c3 in charset: state = hash_forward(seed, [c1, c2, c3]) forward[state] = c1 + c2 + c3 # Backward pass: invert fmix64 and final multiply, enumerate last 3 chars inv_target = invert_fmix64(TARGET) for c4 in charset: for c5 in charset: for c6 in charset: state = hash_backward(inv_target, [c4, c5, c6]) if state in forward: print(f"Found: {forward[state]}{c4}{c5}{c6}") ``` **Key insight:** Meet-in-the-middle reduces search from `95^6 ≈ 7.4×10^11` to `2×95^3 ≈ 1.7×10^6` — a factor of ~430,000x speedup. Critical when the hash function is invertible from the output side (i.e., `fmix64` and the final multiply can be undone). Also: uninitialized VM memory that stays zero simplifies the hash function by removing a variable. --- ## Sprague-Grundy Game Theory Binary (DiceCTF 2026) **Pattern (Bedtime):** Stripped Rust binary plays N rounds of bounded Nim. Each round has piles and max-move parameter k. Binary uses a PRNG for moves when in a losing position; user must respond optimally so the PRNG eventually generates an invalid move (returns 1). Sum of return values must equal a target. **Game theory identification:** - Bounded Nim: remove 1 to k items from any pile per turn - **Grundy value** per pile: `pile_value % (k+1)` - **XOR** of all Grundy values: non-zero = winning (N-position), zero = losing (P-position) - N-positions: computer wins automatically (returns 0) - P-positions: computer uses PRNG, may make invalid move (returns 1) **PRNG state tracking through user feedback:** ```python MASK64 = (1 << 64) - 1 def prng_step(state, pile_count, k): """Computer's PRNG move. Returns (pile_idx, amount, new_state).""" r12 = state[2] ^ 0x28027f28b04ccfa7 rax = (state[1] + r12) & MASK64 s0_new = ROL64((state[0] ** 2 + rax) & MASK64, 32) r12_upd = (r12 + rax) & MASK64 s0_final = ROL64((s0_new ** 2 + r12_upd) & MASK64, 32) pile_idx = rax % pile_count amount = (r12_upd % k) + 1 return pile_idx, amount, [s0_final, r12_upd, state[2]] # Critical: state[2] updated ONLY by user moves (XOR of pile_idx, amount, new_value) # PRNG moves do NOT affect state[2] — creates feedback loop ``` **Solving approach:** 1. Dump game data from GDB (all entries with pile values and parameters) 2. Classify: count P-positions (return 1) vs N-positions (return 0) 3. Simulate each P-position: PRNG moves → user responds optimally → track state[2] 4. Encode user moves as input format (4-digit decimal pairs, reversed order) **Key insight:** When a game binary's PRNG state depends on user input, you must simulate the full feedback loop — not just solve the game theory. Use GDB hardware watchpoints to discover which state variables are affected by user vs computer moves. --- ## Kernel Module Maze Solving (DiceCTF 2026) **Pattern (Explorer):** Rust kernel module implements a 3D maze via `/dev/challenge` ioctls. Navigate the maze, avoid decoy exits (status=2), find the real exit (status=1), read the flag. **Ioctl enumeration:** | Command | Description | |---------|-------------| | `0x80046481-83` | Get maze dimensions (3 axes, 8-16 each) | | `0x80046485` | Get status: 0=playing, 1=WIN, 2=decoy | | `0x80046486` | Get wall bitfield (6 directions) | | `0x80406487` | Get flag (64 bytes, only when status=1) | | `0x40046488` | Move in direction (0-5) | | `0x6489` | Reset position | **DFS solver with decoy avoidance:** ```c // Minimal static binary using raw syscalls (no libc) for small upload size // gcc -nostdlib -static -Os -fno-builtin -o solve solve.c -Wl,--gc-sections && strip solve int visited[16][16][16]; int bad[16][16][16]; // decoy positions across resets void dfs(int fd, int x, int y, int z) { if (visited[x][y][z] || bad[x][y][z]) return; visited[x][y][z] = 1; int status = ioctl_get_status(fd); if (status == 1) { read_flag(fd); exit(0); } if (status == 2) { bad[x][y][z] = 1; return; } // decoy — mark bad int walls = ioctl_get_walls(fd); int dx[] = {1,-1,0,0,0,0}, dy[] = {0,0,1,-1,0,0}, dz[] = {0,0,0,0,1,-1}; int opp[] = {2,3,0,1,5,4}; // opposite directions for backtracking for (int dir = 0; dir < 6; dir++) { if (!(walls & (1 << dir))) continue; // wall present ioctl_move(fd, dir); dfs(fd, x+dx[dir], y+dy[dir], z+dz[dir]); ioctl_move(fd, opp[dir]); // backtrack } } // After decoy hit: reset via ioctl 0x6489, clear visited, re-run DFS ``` **Remote deployment:** Upload binary via base64 chunks over netcat shell, decode, execute. **Key insight:** For kernel module challenges, injecting test binaries into initramfs and probing ioctls dynamically is faster than static RE of stripped kernel modules. Keep solver binary minimal (raw syscalls, no libc) for fast upload. --- ## Multi-Threaded VM with Channel Synchronization (DiceCTF 2026) **Pattern (locked-in):** Custom stack-based VM runs 16 concurrent threads verifying a 30-char flag. Threads communicate via futex-based channels. Pipeline: input → XOR scramble → transformation → base-4 state machine → final check. **Analysis approach:** 1. **Identify thread roles** by tracing channel read/write patterns in GDB 2. **Extract constants** (XOR scramble values, lookup tables) via breakpoints on specific opcodes 3. **Watch for inverted logic:** validity check returns 0 for valid, non-zero for blocked (opposite of intuition) 4. **Detect futex quirks:** `unlock_pi` on unowned mutex returns EPERM=1, which can change all computations **BFS state space search for constrained state machines:** ```python from collections import deque def solve_flag(scramble_vals, lookup_table, initial_state, target_state): """BFS through state machine to find valid flag bytes.""" flag = [None] * 30 # Known prefix/suffix from flag format flag[0:5] = list(b'dice{') flag[29] = ord('}') # For each unknown position, try all printable ASCII states = {initial_state} for pos in range(28, 4, -1): # processed in reverse next_states = {} for state in states: for ch in range(32, 127): transformed = transform(ch, scramble_vals[pos]) digits = to_base4(transformed) new_state = apply_digits(state, digits, lookup_table) if new_state is not None: # valid path exists next_states.setdefault(new_state, []).append((state, ch)) states = set(next_states.keys()) # Trace back from target_state to recover flag ``` **Key insight:** Multi-threaded VMs require tracing data flow across thread boundaries. Channel-based communication creates a pipeline — identify each thread's role (input, transform, validate, output) by watching which channels it reads/writes. Constants that affect computation may come from unexpected sources (futex return values, thread IDs). --- ## Backdoored Shared Library Detection via String Diffing (Hack.lu CTF 2012) **Pattern (Zombie Lockbox):** A setuid binary uses `strcmp` for password validation. The expected password is visible via `strings` and works under GDB (which drops suid), but fails when run normally. The binary links against a non-standard libc that patches function behavior based on suid status. **Detection steps:** 1. Check for non-standard library paths with `ldd`: ```bash ldd ./binary # Suspicious: libc.so.6 => /lib/libc/libc.so.6 (non-standard path) # Normal: libc.so.6 => /lib32/libc.so.6 ``` 2. Diff strings between the suspicious and system libc: ```bash strings /lib/libc/libc.so.6 > suspicious_strings strings /lib32/libc-2.15.so > normal_strings diff suspicious_strings normal_strings ``` 3. Disassemble the patched function (e.g., `puts`) to find injected code: ```bash gdb /lib/libc/libc.so.6 (gdb) disas puts # Look for unexpected calls or branches # Injected code may check suid status (getuid/geteuid syscalls) # and swap the expected password at runtime ``` **Key insight:** When a binary behaves differently under GDB vs. normal execution, check `ldd` for non-standard library paths. Suid binaries drop privileges under debuggers, so a backdoored libc can detect this via `getuid`/`geteuid` syscalls and change program behavior accordingly. The `strings | diff` approach quickly reveals injected data without full disassembly. --- --- ## Custom binfmt Kernel Module with RC4 Flat Binaries (BSidesSF 2026) **Pattern (Private Binary):** A custom Linux kernel module (`.ko`) registers a `binfmt` handler for non-standard binary formats. When a file with a specific magic number is executed, the kernel module intercepts it, decrypts the contents in memory, and jumps to the entry point. **Reverse engineering approach:** 1. **Analyze the `.ko`:** Look for `register_binfmt()` call — it registers a `struct linux_binfmt` with a `load_binary` callback 2. **Find the magic number:** The `load_binary` function checks the file's first bytes against a specific magic number to identify its format 3. **Extract the encryption key:** Look for `movabs` instructions loading 8-byte constants — these are often RC4 key bytes 4. **Identify the encryption scheme:** Common choices are RC4, XOR, or AES-ECB. RC4 is identifiable by the S-box initialization loop (256-byte array, swap pattern) 5. **Decrypt the flat binary:** Apply the recovered key to the encrypted file contents, skipping any header ```python from Crypto.Cipher import ARC4 # Extract RC4 key from kernel module (found via movabs instructions) key = bytes([0x41, 0x42, 0x43, ...]) # Key bytes from .ko disassembly with open('encrypted.bin', 'rb') as f: header = f.read(HEADER_SIZE) # Skip binfmt header encrypted = f.read() cipher = ARC4.new(key) decrypted = cipher.decrypt(encrypted) # The decrypted output is a flat binary (no ELF headers) # Load at the fixed virtual address specified in the kernel module # Disassemble with: objdump -b binary -m i386:x86-64 -D decrypted.bin # Or in Ghidra: import as "Raw Binary", set base address from .ko ``` **Detection in kernel module:** - `register_binfmt` / `unregister_binfmt` calls - `vm_mmap()` or `vm_brk()` for memory allocation at fixed addresses - Direct jump to mapped memory (entry point execution) - S-box initialization pattern (RC4): loop 0-255, swap `S[i]` with `S[j]` **Key insight:** The flat binary has no ELF headers, so standard tools won't recognize it. You must extract the load address from the kernel module (look for the `vm_mmap` call's address argument) and import the decrypted blob at that address in your disassembler. RC4 keys in kernel modules are often stored as immediate values in `mov` or `movabs` instructions rather than in data sections. **References:** BSidesSF 2026 "Private Binary" --- ## Hash-Resolved Imports / No-Import Ransomware (BSidesSF 2026) **Pattern (Ran Somewhere):** Malware binary has zero visible imports — all API calls are resolved at runtime by hashing symbol names and comparing against pre-computed hash values. The binary uses `dlopen` + a custom hash table to find libc and libcrypto functions. **Identification:** - `readelf -d` shows no dynamic symbols or very few (just `dlopen`/`dlsym`) - Strings reveal no standard API names - Disassembly shows hash computation loops followed by indirect calls - RC4-encrypted embedded strings (RSA public key, file paths, passphrases) **Analysis shortcut — LD_PRELOAD key extraction:** Rather than reversing the full hash resolution and key derivation, hook the crypto functions that the malware ultimately calls: ```c // hook_crypto.c — captures AES key used by the ransomware #define _GNU_SOURCE #include #include #include int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type, ENGINE *impl, const unsigned char *key, const unsigned char *iv) { if (key) { FILE *f = fopen("/tmp/aes_key.bin", "wb"); fwrite(key, 1, 32, f); // AES-256 fclose(f); fprintf(stderr, "[HOOK] AES key captured\n"); } typedef int (*orig_t)(EVP_CIPHER_CTX*, const EVP_CIPHER*, ENGINE*, const unsigned char*, const unsigned char*); orig_t orig = (orig_t)dlsym(RTLD_NEXT, "EVP_CipherInit_ex"); return orig(ctx, type, impl, key, iv); } ``` ```bash # Compile and run gcc -shared -fPIC -o hook.so hook_crypto.c -ldl # Run in Docker container (ransomware may be destructive!) docker run --rm -v $(pwd):/work -w /work ubuntu:22.04 \ bash -c "LD_PRELOAD=./hook.so ./ransomware; xxd /tmp/aes_key.bin" ``` **Hash resolution patterns:** - **SipHash variant:** Two 64-bit seeds, iterative mixing with symbol name bytes - **DJB2/FNV variants:** Simpler hash functions with recognizable constants (`5381`, `0xcbf29ce484222325`) - **ROR13-based:** Windows malware favorite: `hash = (hash >> 13) | (hash << 19); hash += c` **Decryption after key capture:** ```python from Crypto.Cipher import AES key = open('/tmp/aes_key.bin', 'rb').read() iv = open('/tmp/aes_iv.bin', 'rb').read() # Also hookable cipher = AES.new(key, AES.MODE_CBC, iv) with open('flag.txt.enc', 'rb') as f: ct = f.read() pt = cipher.decrypt(ct) # Remove PKCS7 padding pt = pt[:-pt[-1]] print(pt.decode()) ``` **Key insight:** When a binary resolves all imports via hashing, don't waste time reversing the hash function and building a rainbow table. Instead, let the malware resolve everything itself by running it in a sandboxed environment with `LD_PRELOAD` hooks on the functions you care about (OpenSSL crypto functions, file I/O, network calls). The AES key is deterministic across runs — if it works once, it works always. **Safety:** Always run suspected ransomware in a Docker container or VM. Mount only copies of the encrypted files, never originals. **References:** BSidesSF 2026 "Ran Somewhere" --- ## ELF Section Header Corruption for Anti-Analysis (BSidesSF 2026) **Pattern (stubborn-elf):** An ELF binary has deliberately corrupted section header table entries, causing standard analysis tools (`readelf`, `objdump`, IDA, Ghidra) to crash or produce errors. However, the **program headers** (which the OS loader uses) are intact, so the binary executes normally. The flag is appended after the corrupted sections, marked with magic bytes. ```python import sys # Standard tools fail on corrupted section headers # Manual parsing bypasses section headers entirely with open("stubborn_elf", "rb") as f: data = f.read() # Search for magic marker appended after ELF sections magic = b"\xDE\xAD\xBE\xEF\xCA\xFE\xBA\xBE" idx = data.find(magic) if idx >= 0: # Data after magic is XOR-encrypted encrypted = data[idx + len(magic):] decrypted = bytes(b ^ 0x42 for b in encrypted) print(decrypted.decode(errors='ignore')) ``` **Key insight:** ELF execution requires **program headers** (PT_LOAD segments), NOT section headers. Section headers are metadata for debuggers and analysis tools — they're optional at runtime. Corrupting `e_shoff`, `e_shnum`, or `e_shstrndx` in the ELF header breaks tools but not execution. When tools fail, parse the binary manually or patch the ELF header to zero out section header references before loading in a disassembler. **Recovery approach:** ```bash # Patch section header offset to 0 (removes section table) printf '\x00\x00\x00\x00\x00\x00\x00\x00' | dd of=binary bs=1 seek=40 conv=notrunc # Now Ghidra/IDA can load it using program headers only # Or use readelf -l (program headers only, ignores sections) readelf -l stubborn_elf ``` **When to recognize:** `readelf -S` crashes or shows garbage. `file` command identifies it as ELF. `readelf -l` (lowercase L, program headers) works fine. The binary runs normally despite tool failures. **References:** BSidesSF 2026 "stubborn-elf" --- See also: [patterns-ctf-2.md](patterns-ctf-2.md) for Part 2 (multi-layer self-decrypting binary, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice, decision tree obfuscation, GF(2^8) Gaussian elimination), [patterns-ctf-3.md](patterns-ctf-3.md) for Part 3 (Z3 boolean circuit, sliding window popcount, keyboard LED Morse code, C++ destructor-hidden validation, VM sequential key-chain brute-force, BWT inversion, OpenType font ligature exploitation, GLSL shader VM with self-modifying code).