Phase 13: Tools & Protocols

The Reverse Engineering Landscape: History, Tool Families, and How a Decompiler Is Built

Type: Build

Languages: Python, C, x86-64 assembly, Java (one Ghidra script)

Prerequisites: Phase 13 · 32 (ELF from scratch), Phase 13 · 33 (disassembly and control flow), Phase 13 · 34 (lifting to pseudocode)

Time: ~150 minutes

The Problem

You ask an agent to summarise what three reverse engineering tools say about grade in ledger-x86_64-O1.o. The agent runs each tool, reads three listings, and reports that the function sits at 0x08000050 in one tool, 0x400010 in another, and 0x100010 in the third. It concludes that the tools disagree about where the function is.

The tools agree. The object is relocatable and has no load address, so each tool chooses one. Subtract the address each tool gave .text and every answer is 0x10. The agent compared raw addresses because nothing told it what the address meant in each tool.

Later the same agent reads grade in the -O0 build and reports 3, 8, 8, and 9 basic blocks from four tools. All four numbers are real. One tool stopped at jmp rax. Two tools read the jump table through its relocations. One tool split a block at the jump to place a label. A block count without the reason behind it is a number without a claim.

This lesson is the map you need before you trust any of those numbers. Lessons 32 to 36 built one decoder, one lifter, one evidence record, and one rebuild. The second half of the track puts them next to the tools people use. To do that you need the history of the field, the families the tools fall into, the pipeline every decompiler runs, and a harness that compares what the tools say on objects you compiled yourself.

Fifty-Three Years in Dated Steps

The field is older than most of its tools. Each date below has a primary source in Further Reading.

1973 to 1991: debuggers first, then disassemblers

The first decompilation theses are from 1973. Hollander at Stanford and Housel at Purdue wrote them, and both are known today through citations. The practical tools of the period were debuggers. Maranzano and Bourne wrote the adb tutorial at Bell Labs in 1977, and the debugger shipped with UNIX V7 in 1979. Tim Paterson's DEBUG, with an 8086 disassembler, shipped in PC-DOS 1.00 in August 1981. Richard Stallman's GDB entered distribution in 1986. SoftICE put a kernel debugger under DOS in 1987.

Batch disassemblers came next. Sourcer, from V Communications in 1988, ran multiple passes over a DOS program and wrote a commented listing. GNU binutils began with unnumbered snapshots in 1988, and objdump carries a 1990 copyright. Ilfak Guilfanov wrote the first line of IDA in January 1991 and released IDA 0.1 as shareware on 20 May 1991.

1991 to 2009: interactive disassemblers and the first decompilers

Cristina Cifuentes finished "Reverse Compilation Techniques" at QUT in July 1994. Her decompiler, dcc, took i80286 DOS programs to C, and her 1995 paper with Gough set the phases that every later decompiler kept. REC followed in 1997 and Boomerang in 2002, the first open-source decompiler, under a BSD licence.

IDA became a commercial product at DataRescue in 1996, gained a plugin SDK in 1999, and moved to Hex-Rays, founded in 2005. The Hex-Rays Decompiler 1.0 shipped on 17 September 2007. In the same year Mike Van Emmerik submitted "Static Single Assignment for Decompilation", the thesis that put SSA at the centre of decompiler data flow. Halvar Flake's "Structural Comparison of Executable Objects" (2004) started the diffing family, and BinDiff went on sale in August 2004.

radare started in February 2006 as a forensic hex editor with 64-bit offsets. radare2, the modular rewrite under LGPL-3.0, had its first release on 7 March 2009.

2009 to 2019: frameworks and intermediate representations

LLDB was announced as an LLVM subproject on 8 June 2010. BAP, the Binary Analysis Platform with its BIL intermediate language, appeared at CAV 2011. Google acquired zynamics, the makers of BinDiff, on 1 March 2011. Hopper appeared on the Mac App Store around the end of 2011. Matt Godbolt started Compiler Explorer in 2012 so that a C++ programmer could see the assembly a compiler produced.

angr received its first line of code in August 2013. The Phoenix paper (Schwartz, Lee, Woo, Brumley, USENIX Security 2013) showed semantics-preserving structuring with 40 goto statements across 8,676 functions. DREAM (Yakdan and others, NDSS 2015) removed the goto statements with pattern-independent structuring. Vector 35 was founded on 14 January 2015, and Binary Ninja went on sale on 23 July 2016 with its Low Level IL and a Python API. angr was released publicly in 2015 and placed third at the DARPA Cyber Grand Challenge in August 2016. BinDiff became free of charge in March 2016. Cutter 1.0 gave radare2 a Qt GUI in December 2017. In 2018 Guilfanov presented "Decompiler internals: microcode" at Recon, the first public description of the Hex-Rays IR.

2019 to 2026: open decompilers and the research turn

Ghidra 9.0 was released by the NSA at the RSA Conference on 5 March 2019 under Apache-2.0, with full source on GitHub a month later. angr 8.19.2.4 added a preliminary decompiler the same year. Binary Ninja 2.0 shipped its High Level IL decompiler on 20 May 2020. Rizin was announced as a fork of radare2 on 8 December 2020, and the Cutter team moved to it. IDA Freeware 7.6 added a cloud x64 decompiler in May 2021.

BinDiff 8 was open-sourced under Apache-2.0 on 25 September 2023. SAILR (Basque and others, USENIX Security 2024) became angr's default structurer and argued that only spurious goto statements should be removed, because Linux 6.1 has 3,754 of them in its source. Binary Ninja 4.0 added a Free edition in February 2024, and IDA 9.0 added idalib for headless use. The learned-decompiler papers run from SLaDe (2023) through LLM4Decompile (2024) to DecompileBench (2025) and two 2026 studies on agent-assisted readability and on decompilers that recompile more and preserve less.

The versions this lesson ran are the current ones on 14 October 2026: Ghidra 12.1.4 (21 September 2026), radare2 6.2.4 (5 October 2026), rizin 0.9.1 (29 June 2026), and angr 10.0.3 (10 October 2026). Binary Ninja 6.0 "Krypton" (3 September 2026) and IDA 9.4 (13 July 2026) are the current closed tools. This lesson did not run them.

The Tool Families

A tool belongs to a family by what it produces. The families overlap in one program, and the overlap is where most confusion starts.

Disassemblers

A linear sweep decodes from the first byte of a code section to the last. GNU objdump -d and llvm-objdump -d work this way, and neither builds a control-flow graph. GNU objdump treats symbols as instruction boundaries and sweeps between them. binutils 2.47 (26 July 2026) is GPL-3.0-or-later. LLVM 23.1.3 is Apache-2.0 with the LLVM exception. Apple's Command Line Tools ship llvm-objdump as "Apple LLVM version 17.0.0", and that is the copy this lesson captured.

An interactive disassembler runs recursive descent from known entries, adds heuristics for function starts, and keeps a database. IDA Pro (processor modules, autoanalysis queues, FLIRT signatures), Ghidra (Disassembler plus Auto Analysis analyzers), Binary Ninja (recursive descent, then linear sweep over unknown regions), Hopper, radare2 and rizin (aa, aaa, af), with Cutter and iaito as GUIs, are all in this family. Andriesse and others (USENIX Security 2016) measured nine of them on compiler-generated binaries. Instruction accuracy was close to 100 percent, and the hard primitive was function starts, with false negative rates of 20 percent or more.

Decompilers

Commercial decompilers: the Hex-Rays Decompiler (twelve target families from x86 to RISC-V), Binary Ninja's HLIL with its Pseudo C rendering, and Hopper's pseudo-code view. Open decompilers: the Ghidra decompiler, a C++ engine under Apache-2.0 that r2ghidra and rz-ghidra reuse as a plugin and that pypcode reuses as a lifter; the angr decompiler on AIL with SAILR structuring under BSD-2-Clause; r2dec and jsdec in JavaScript; Boomerang and dcc as history. radare2 also has a built-in pdc, "pseudo disassembler output in C-like syntax". rizin has no built-in decompiler. The command pdc does not exist in rizin 0.9.1, and pdg needs the rz-ghidra plugin.

Debuggers

GDB (1986) with the GEF and pwndbg front-ends, LLDB (2010), WinDbg with Time Travel Debugging (2017), OllyDbg (frozen in 2014), x64dbg (GPL-3.0, 2014), the Ghidra Debugger with its Python connectors, the Binary Ninja debugger, and the built-in debuggers of radare2 and rizin. Lesson 42 uses two of them on your own builds.

Binary analysis frameworks

angr (VEX IR, symbolic execution, CFGFast and CFGEmulated, decompiler), BAP (BIL, OCaml), Dyninst, Jakstab, and ByteWeight are the frameworks that Andriesse named. Ghidra is a library through PyGhidra, Binary Ninja through import binaryninja, and IDA through idalib.

Format and engine libraries

LIEF 1.0.0 (12 July 2026, Apache-2.0) parses ELF, PE, Mach-O, COFF, and the Android formats. CLE, "CLE Loads Everything", is angr's loader. Capstone 5.0.9 (28 May 2026, BSD-3-Clause) decodes; Keystone 0.9.2 (GPL-2.0) assembles; Unicorn 2.1.4 (GPL-2.0) emulates. radare2 6.2.4 bundles Capstone 5, and rizin 0.9.1 vendors Capstone, Zydis, and tree-sitter.

Diffing tools and compiler-side viewers

BinDiff 8 (Apache-2.0) with its BinExport 12 plugin for IDA Pro, Binary Ninja, and Ghidra; Diaphora 3.4.3 (11 October 2026, AGPL-3.0) as an IDA plugin; radiff2 and rz-diff. Compiler Explorer (BSD-2-Clause) shows the assembly a compiler emits for a source fragment, which is the forward direction of everything else in this list. Lesson 44 uses the diffing family.

Versions, licences, and what this lesson ran

The Free edition of Binary Ninja decompiles x86, x86-64, ARMv7, and AArch64 only and has no API. IDA Free has no IDAPython, no SDK, and no idalib, and its decompiler runs in the cloud. Every statement about those two tools in this lesson comes from their documentation and release notes, and the figure marks them as not run.

The Decompiler Pipeline, Stage by Stage

Cifuentes named the phases in 1994: syntax analysis, semantic analysis, intermediate code generation, control-flow graph generation, data-flow analysis, control-flow analysis, and code generation. She split them into a machine-dependent front end, a machine- and language-independent middle, and a language-dependent back end. Every decompiler since keeps that shape and renames the parts. The table below gives the eight stages this lesson uses and the name each tool gives them. The figure shows the same table with the tool columns selectable.

Loader

The loader parses the container, maps sections, and reads symbols and relocations. In a relocatable object nothing has an address yet, so the loader picks one. radare2 and rizin map the file at 0x08000000, which puts .text at 0x08000040 because the section starts at file offset 0x40. CLE places .text at 0x400000. Ghidra's ElfLoader places it at 0x100000. llvm-objdump prints section offsets and no base at all.

The loaders also differ on relocations. Ghidra processes them on import, and CLE applies them. radare2 prints a hint when it opens the object: relocations were not applied, and bin.relocs.apply=true changes that. What a loader does with relocations decides what every later stage can see, and you will meet the effect in the -O0 jump table below.

Disassembly

Lesson 33 built both methods. Every tool in the table except llvm-objdump runs recursive descent and then adds a search for function starts. Binary Ninja's documentation names the phases in order: Recursive Descent, Call Target Analysis, CFG Analysis as "linear disassembly of the unknown regions", and Tail Call Analysis. Ghidra runs its analyzers by priority, with "Disassemble Entry Points" and "Function Start Search" early. IDA keeps autoanalysis queues, each with a priority, and FLIRT signatures for library code. angr's CFGFast documents four sources of function starts: symbols, a prologue scan, call targets, and a complete scan of the remaining bytes. That last source is where angr invents functions at padding.

CFG recovery

Each tool cuts blocks the way lesson 33 did, with one difference you will measure. angr ends a block at a call, because its IRSB is a single-entry, multiple-exit block and a call is an exit. Ghidra, radare2, and rizin run through a call. On grade_label at -O0, which calls grade, angr reports 2 blocks and the other three report 1. The instructions are the same.

Lifting

Binary Ninja defines lifting as "translating from a lower level language to an intermediate language considered higher". The IRs differ in what they are for. P-code and VEX were built for analysis and decompilation. ESIL and RzIL were built for emulation, and neither has an SSA stage. Hex-Rays microcode starts as RISC-like code with 72 opcodes and passes through maturity levels from MMAT_GENERATED to MMAT_LVARS. Binary Ninja lifts to Lifted IL, then folds flags into conditions in LLIL, and MLIL "translates registers and memory accesses into variables". The next section puts four of these IRs side by side on one instruction.

SSA

Van Emmerik's thesis lists what SSA buys a decompiler: expression propagation, parameter and return identification without an ABI assumption, preservation analysis, type analysis as a sparse data-flow problem, and the analysis of indirect jumps and calls. Ghidra calls its SSA construction Heritage and adds two operations that exist only in the graph, MULTIEQUAL for the phi and INDIRECT for side effects. Binary Ninja exposes an SSA form at every IL level. angr has two SSA stages inside Clinic. Hex-Rays does not name an SSA form and does use-def data flow in its global optimisation passes. ESIL and RzIL have none.

Data flow and types

Ghidra's main simplification loop runs "Eliminate Dead Code", "Propagate Local Types", and "Perform Term Rewriting" until nothing changes, with the rewriting done by a pool of Rule objects. Hex-Rays transforms microcode, detects call arguments at MMAT_CALLS, and allocates local variables at MMAT_LVARS. angr runs CallingConventionAnalysis, VariableRecoveryFast, and Typehoon, an implementation of Noonan, Loginov, and Cok's polymorphic type inference. Lesson 41 is about this stage.

Structuring

Ghidra's CollapseStructure searches for sub-graphs that match a structure, collapses them to one node, and when it gets stuck removes an edge and marks it unstructured. Hex-Rays reduces the CFG to if, while, do, and switch and leaves the rest as goto regions. angr identifies single-entry, single-exit regions and structures them with SAILRStructurer by default, with PhoenixStructurer and DreamStructurer as alternatives. Lesson 34 wrote one structurer by hand.

Emission

Ghidra emits C tokens through PrintC with an Oppen pretty-printer after ActionNameVars and ActionSetCasts. Hex-Rays beautifies and prints the ctree. Binary Ninja renders HLIL as Pseudo C, Pseudo Rust, or Pseudo Python. angr returns codegen.text. radare2 prints pdc, pdd, or pdg.

One Instruction in Four IRs

Take 48 83 ec 10, subq $0x10, %rsp. Decode it on paper with the rules from lesson 33.

1. 48 is REX with W = 1. The operand is 64 bits.
2. 83 is the group-1 arithmetic opcode with an 8-bit immediate. The ModRM reg field picks the operation.
3. ec is ModRM 11 101 100. mod = 11 is the register form. reg = 101 is /5, which is sub. r/m = 100 is %rsp.
4. 10 is the immediate, 16.

Four lifters were given those four bytes. The fixtures hold the exact commands and the full text.

ESIL from radare2 6.2.4 is one comma-separated string of 30 tokens:

textCopy16,0x8000000000000000,-,!,16,rsp,-=,63,$o,^,of,:=,63,$s,sf,:=,$z,zf,:=,$p,pf,:=,64,$b,cf,:=,4,$b,af,:=

It is a stack machine. 16,rsp,-= subtracts in place. Every flag is then written with an internal operator: $o overflow, $s sign, $z zero, $p parity, $b borrow.

RzIL from rizin 0.9.1 is an S-expression with 14 set effects:

textCopy(seq (set op1 (var rsp)) (set op2 (bv 64 0x10)) (set sub (- (var op1) (var op2))) (set rsp (var sub))
     (set _result (var sub)) (set pf (! (lsb (let _val (cast 8 false (var _result)) ...))))
     (set zf (is_zero (var _result))) (set sf (msb (var _result))) ... (set cf (|| ...)) (set of (|| ...)) (set af (|| ...)))

Every flag is a boolean expression over bitvectors. The parity flag alone is a four-level let.

P-code from the SLEIGH specification x86:LE:64:default through pypcode 4.0.1 is nine operations after the IMARK:

textCopyCF = RSP < 0x10
OF = sborrow(RSP, 0x10)
RSP = RSP - 0x10
SF = RSP s< 0x0
ZF = RSP == 0x0
unique[58300:8] = RSP & 0xff
unique[58400:1] = popcount(unique[58300:8])
unique[58500:1] = unique[58400:1] & 0x1
PF = unique[58500:1] == 0x0

Register transfers with temporaries in the unique space. The flags are computed at the instruction, eagerly, and CF and OF are computed before the subtraction overwrites RSP.

VEX from pyvex 10.0.3 is seven statements after the IMark:

textCopyt2 = GET:I64(offset=48)
t0 = Sub64(t2,0x0000000000000010)
PUT(offset=144) = 0x0000000000000008
PUT(offset=152) = t2
PUT(offset=160) = 0x0000000000000010
PUT(offset=168) = 0x0000000000000000
PUT(offset=48) = t0

No flag appears. VEX names registers by their offset in the guest state, and offset 48 is rsp. Offsets 144 to 168 are cc_op, cc_dep1, cc_dep2, and cc_ndep. The lifter stores the operation code 8, which is the 64-bit subtraction, and the two operands. A later branch that reads a flag computes it from the thunk. That is why pcode.writesFlags is true and vex.computesFlags is false in the harness report, and why VEX is the shortest of the four.

The same four lifters on 48 63 c7, movslq %edi, %rax, the first instruction of grade at -O1, give 5 ESIL tokens (32,edi,~,rax,=), 1 RzIL set, 1 P-code operation (RAX = sext(EDI)), and 4 VEX statements. No lifter writes a flag, because the instruction has none. Ghidra's own listing of that instruction, captured through the headless script, prints the same operation as (register, 0x0, 8) INT_SEXT (register, 0x38, 4): register offset 0 is RAX, offset 0x38 is RDI, and the sizes are 8 and 4 bytes.

83 f9 04, cmpl $0x4, %ecx, from the bounds check in grade, gives 30 ESIL tokens, 13 RzIL set effects, 10 P-code operations, and 7 VEX statements with cc_op 7, the 32-bit subtraction. Nothing writes %ecx. The two subtraction widths you will meet in VEX are 7 for 32 bits and 8 for 64 bits, and the translator you build carries that table.

Four Address Bases for One Function

Before the harness can compare anything it has to agree with itself about where a function is. Do the arithmetic once by hand for grade in ledger-x86_64-O1.o.

The symbol table says grade is at .text + 0x10 with size 46. Every tool agrees once the base is removed. The harness reads the base from each capture: iSj in radare2 and rizin, the .text section entry in the angr capture, the .text memory block in the Ghidra capture, and zero for the sweep. The test test_every_tool_places_grade_at_offset_0x10_from_its_own_base fixes that fact.

Where the Tools Agree and Where They Split

The harness in code/main.py reads the captures, reduces every address, and compares. These are the numbers it reports, and the tests assert each of them.

The agreement case: ledger at -O1

Four functions, 56 instructions in the sweep, 54 inside the symbols and 2 in padding. Every tool reports the same instruction starts in every function: 9, 14, 13, and 18. The four graph builders report 3, 3, 5, and 4 blocks and 2, 3, 5, and 4 edges, and the edge sets are identical, pair for pair, across radare2, rizin, angr, and Ghidra. The five blocks and five edges of total are the ones lesson 33 drew by hand.

The one difference is at the padding. The 3 bytes between functions (66 90 after grade, 90 after total) are decoded by the sweep, which has no reason to stop, and by angr, which creates sub_40003e and sub_40005f with no symbol. radare2, rizin, and Ghidra claim none of them.

The vectorised case: ledger at -O2

total is 41 instructions in 10 blocks with 13 edges, in every one of the four graph builders. The SSE2 loop and the scalar remainder loop from lesson 33 are cut the same way everywhere. angr again adds two functions at padding.

The jump table: grade at -O0

Here the tools split, and the split is the lesson.

All four decoded the same 33 instructions when they reached them. radare2 stopped at jmp rax and left 36 bytes unclaimed, and the harness names that range as missed code. rizin and Ghidra found the four distinct targets that lesson 33 read from the five relocations in .rela.rodata. angr lists the case blocks inside grade and still has no edge from the jump block to them; its capture records the transfer at 0x62 as leading to an unresolvable target node, and its decompiler printed a computed goto. Ghidra has one block and one edge more than rizin for the same control flow because its switch analysis placed a label on the jmp rax and the block model starts a new block at a label.

The lesson-33 graph had 8 blocks and 11 edges. rizin matches it exactly. The point is to see that 9 and 11 and 7 and 2 are not four measurements of code found. They are four answers to how far a tool followed one indirect jump and where it chose to cut.

Data in code: sweep-trap

lookup is 24 bytes with the 4 table bytes 31 c0 90 48 at 0x10. The sweep reports 8 instruction starts in that range. The harness lists its differences from the majority: extra starts at 0x10, 0x12, and 0x13, and a missing start at 0x14. That is the addq $0x1, %rax from lesson 33 one byte early. radare2, rizin, and Ghidra report 6 starts, 2 blocks, 1 edge, and leave 0x10 to 0x14 unclaimed. angr reports the same 2 blocks and 1 edge for lookup and then invents sub_400010 on the table bytes: 2 instructions in a 4-byte block that ends at 0x14, where the second block of lookup starts, so the REX byte at 0x13 belongs to no instruction. The harness's last finding for this object is that no graph builder decodes 0x10 to 0x14, so those bytes sit inside a function and are data.

Invented and missed, by tool

Across the four objects the harness names 9 functions that angr invents with no symbol, every one at padding or at the trap table. It names one range that radare2 misses, the 36 case bytes of grade at -O0. It names 9 ranges that the linear sweep decodes and no graph builder treats as code: the eight padding gaps plus the trap table. It names no missed function in any tool.

Three Decompilers on grade

Three of the captured tools emit C-like text for grade at -O1. The source is switch (score / 10) with five cases, compiled to a bounds check and a table of return values.

Ghidra 12.1.4:

cCopyuint grade(int param_1)
{
  uint uVar1;
  uint uVar2;

  uVar2 = param_1 / 10 - 6;
  uVar1 = -(param_1 >> 0x1f) ^ (uint)((ulong)((long)param_1 * 0x66666667) >> 0x3f);
  if (uVar2 < 5) {
    uVar1 = *(uint *)(&DAT_001000b4 + (ulong)uVar2 * 4);
  }
  return uVar1;
}

angr 10.0.3:

cCopyextern unsigned int g_4000b4[4];

unsigned int grade(int a0)
{
    unsigned int v1;  // ecx
    unsigned int v2;  // eax

    v1 = (unsigned int)((long long)(a0 * 1717986919) >> 0x22) + (unsigned int)(a0 * 1717986919 >> 63) - 6;
    v2 = 0;
    if (v1 <= 4)
        v2 = g_4000b4[v1];
    return v2;
}

radare2 6.2.4, pdc:

cCopyvoid sym.grade (int64_t arg1) {
    rax = edi
    rax = rax * 0x66666667
    rcx = rax
    rcx >>= 0x22
    rax >>>= 0x3f
    ecx += eax
    ecx += 0xfffffffa
    eax = 0
    v = ecx - 4
    if (((unsigned) v) <= 0) {
        eax = ecx
        rcx = rip
        eax = dword [rcx + rax*4]
    }
    return
}

Read the three against the pipeline. Ghidra's term rewriting folded the multiply by 0x66666667 and the shift by 34 back into param_1 / 10, which is lesson 34's idiom. angr lifted the same bytes and left the arithmetic as a0 * 1717986919 >> 0x22. radare2's pdc is ESIL emulation plus pseudo syntax: register names, no variables, no types, and the bounds check printed as <= 0 after an unsigned subtraction. Ghidra typed the table load through DAT_001000b4; angr declared g_4000b4[4], four elements where the source has five. Neither Ghidra nor angr recovered the enumeration of cases, because at -O1 there is no switch left in the bytes. At -O0, where the jump table exists, Ghidra printed switch(param_1 / 10) with cases 6 to 10 and a default, and angr printed a computed goto.

rizin has no decompiler of its own, and the capture records that pdc does not exist there. Binary Ninja's HLIL and the Hex-Rays ctree are the two outputs this lesson could not capture.

Build It

Run the demo:

bashCopycd phases/13-tools-and-protocols/37-the-reverse-engineering-landscape
python3 code/main.py

The output is one JSON report. It opens with the tool versions recorded at capture time and with the tools present on your machine, which may all be absent. Then for each of the four objects it gives the symbol extents, the padding gaps, the base each tool used, the function starts each tool found or invented, and for each function the instruction-start agreement, block counts, edge counts, and the edges each tool lacks. It ends with byte accounting per tool, the tools' own notes, a list of findings in plain sentences, and the IR section: the translator's output for the three instructions next to the counts measured on the captured lifts.

Expected shape:

jsonCopy{
  "captureTools": {"llvm-objdump": "Apple LLVM version 17.0.0", "radare2": "radare2 6.2.4 ...", "rizin": "rizin 0.9.1 ...", "angr": "10.0.3", "ghidra": "12.1.4"},
  "localTools": {"radare2": "absent", "...": "..."},
  "fixtures": {
    "ledger-x86_64-O0.o": {
      "bases": {"llvm-objdump": "0x0", "radare2": "0x8000040", "angr": "0x400000", "ghidra": "0x100000", "...": "..."},
      "functions": {"grade": {"blocks": {"radare2": 3, "rizin": 8, "angr": 8, "ghidra": 9}, "edges": {"radare2": 2, "rizin": 11, "angr": 7, "ghidra": 12}, "...": "..."}},
      "findings": ["radare2 never reaches bytes 0x76-0x9a inside grade that the other graph builders decode", "..."]
    }
  },
  "ir": {"translator": [{"text": "subq $0x10, %rsp", "flags": {"pcodeLike": "eager", "vexLike": "thunk"}, "...": "..."}], "observed": {"...": "..."}}
}

Read code/main.py in this order:

1. read_text_symbols() reads the section headers and the STT_FUNC symbols of .text. The symbol table is the only ground truth the harness has, and TextSection.gaps() derives the padding from it.
2. parse_objdump(), parse_r2_family(), parse_angr(), and parse_ghidra() each turn one capture into a ToolView: a base, the instructions as offset to length, and one FunctionView per function with blocks, edges, and any decompiled text. Each parser knows one tool's vocabulary and nothing else.
3. majority() and ranges() are the two helpers every comparison uses.
4. function_report() compares instruction starts against the majority of the graph builders, then block counts, then edge sets, and lists what each tool lacks.
5. byte_report() accounts for every byte as code inside a symbol, padding, unclaimed, invented, or missed, and names the bytes inside symbols that no graph builder decodes.
6. findings() turns those into sentences, and compare() assembles one report per object.
7. decode_small(), to_pcode_like(), and to_vex_like() are the translator: three encodings in register form, one eager style, one thunk style.
8. observed_ir() measures the captured lifts, and figure_data() assembles the timeline, the stage table, and the evidence the figure shows.

Three more commands exist. python3 code/main.py compare FIXTURE.o prints one object's report. python3 code/main.py ir 4883ec10 prints the translator's two forms for one instruction. python3 code/main.py figure-data prints the data embedded in the figure.

The captures were made once by code/fixtures/capture_tools.py, which records every command and version in captures.json. The harness and the tests never start a tool. If you have the tools, run the capture script to refresh the files, and read the fixtures README for the environment it needs.

Use It

Use the harness on a new object in this order.

1. Build the object yourself and keep the compiler command. Read its symbol table before you open any tool, so you know how many functions there are and where the padding is.
2. Capture each tool once in headless mode. Store the raw output and the exact command beside the object. A disagreement you cannot re-read is not a finding.
3. Check the base. Pick one function and confirm that every tool places it at the same offset after you subtract its .text address. If one does not, stop and find out what that tool loaded.
4. Compare instruction starts first. If the graph builders disagree here, the rest of the comparison is about decoding and not about analysis.
5. Compare blocks and edges, and read each tool's notes before you judge a count. A switch_op, a COMPUTED_JUMP, or an unresolvable target node explains most differences.
6. Account for bytes. Padding that a tool claims is a heuristic at work. Symbol bytes that a tool leaves are a stopped walk. Symbol bytes that no graph builder claims are data.
7. Write down which tools you did not run. A description from documentation is useful and is a different kind of evidence from a capture.

Interactive Lab

Lab A: one function, four bases

From the code directory, open Python:

bashCopycd phases/13-tools-and-protocols/37-the-reverse-engineering-landscape/code
python3 -q

Run:

pythonCopyfrom main import *
views = load_views("ledger-x86_64-O1.o")
{tool: hex(view.base) for tool, view in views.items()}
{tool: hex(view.functions["grade"].start + view.base) for tool, view in views.items()}

Write the four raw addresses down before you run the second line and check them against the table in this lesson. Then replace grade with grade_label and compute by hand where it sits in Ghidra.

Lab B: the jump table from four sides

pythonCopyreport = compare("ledger-x86_64-O0.o")
report["functions"]["grade"]["blocks"]
report["functions"]["grade"]["edgesMissingPerTool"]["radare2"]
report["notes"]["rizin"]
report["notes"]["ghidra"]
report["bytes"]["radare2"]["misses"]

Count the edges radare2 lacks and match each one to a line of the -O0 listing from lesson 33. Then read the Ghidra note and find the switchD label in the Ghidra capture.

Lab C: eager flags against a thunk

pythonCopytranslate("48 83 ec 10")["pcodeLike"]
translate("48 83 ec 10")["vexLike"]
translate("83 f9 04")["vexLike"]
{row["name"]: (row["pcode"]["ops"], row["vex"]["statements"]) for row in observed_ir()["instructions"]}

Find the cc_op constant in both outputs and explain why one is 8 and the other 7. Then try translate("48 01 c8") and read the TranslateError.

Lab D: the trap under five tools

pythonCopyreport = compare("sweep-trap.o")
report["functions"]["lookup"]["instructionStarts"]
report["bytes"]["dataInsideSymbols"]
report["functionStarts"]["angr"]

Decode the bytes 31 c0 90 48 83 c0 01 on paper as a sweep would, then as a descent would after the jmp. Match your two listings to the extra and missing starts the harness reports.

Practice Lab

Add a fifth capture and extend the harness.

Requirements:

• Pick one tool this lesson did not capture and that you can run on your machine: GNU objdump, LIEF with Capstone, a Binary Ninja Free session exported by hand, or Ghidra with a different analyzer set. Capture its output on all four objects with the exact command and version in captures.json.
• Write one parser that turns the capture into a ToolView with the tool's base, instructions, functions, blocks where the tool has them, and edges where it has them.
• Add the tool to TOOLS and, if it builds a graph, to DESCENT_TOOLS. Run the demo and read what changed in the majority sets.
• Write a test that asserts the tool's block counts on ledger-x86_64-O1.o and a test that asserts what it does with the trap table.
• Write one paragraph for the lesson that says, in the tool's own words, what it calls the loader, the disassembler, and the CFG.

Success means python3 code/main.py reports the new tool in captureTools, every function in ledger-x86_64-O1.o keeps allDescentToolsAgree true, and the findings for sweep-trap.o say what the new tool did with bytes 0x10 to 0x14.

Lesson Artifact

This lesson includes outputs/skill-the-reverse-engineering-landscape.md. Use it to run several tools on one object you built, reduce their addresses to one base, and report, per function, where they agree on instruction starts, blocks, and edges, and which bytes each tool invents or misses. It requires captures with recorded commands and versions, a symbol table read from the file, and a statement of which tools were described from documentation only.

Verify It

Run the demo and the deterministic suite:

bashCopycd phases/13-tools-and-protocols/37-the-reverse-engineering-landscape
python3 code/main.py
python3 -m unittest discover -s code/tests -v

Verification should prove:

• the ELF reader finds four function symbols in ledger-x86_64-O1.o with sizes 16, 46, 31, and 63, and two padding gaps
• the sweep captures hold 86, 56, 84, and 11 instructions for the four objects
• radare2 and rizin report the same instructions, blocks, and edges on ledger-x86_64-O1.o, and only radare2 has pdc text
• angr marks sub_40003e and sub_40005f as functions with no symbol
• Ghidra's blocks for total are the five blocks of lesson 33 and its call flows are not graph edges
• every tool places grade at offset 0x10 after its base is removed
• every tool agrees on every instruction start in every function of ledger-x86_64-O1.o
• the four graph builders report 3, 3, 5, 4 blocks and 2, 3, 5, 4 edges there, with identical edge sets
• total at -O2 has 10 blocks and 13 edges in every graph builder
• grade at -O0 has 3, 8, 8, and 9 blocks and 2, 11, 7, and 12 edges in radare2, rizin, angr, and Ghidra, with the reasons in each tool's notes
• grade_label at -O0 is one block everywhere except angr, which ends a block at the call
• angr invents one function per padding gap plus one on the trap table, and no other tool invents or misses a function
• the sweep gains starts 0x10, 0x12, 0x13 and loses 0x14 on the trap, and no graph builder decodes 0x10 to 0x14
• the findings name the missed range in radare2 and every invented function in angr
• the manifest records a version and a command for all 21 capture files
• three tools decompile grade, and only Ghidra folds the division idiom
• the translator writes CF, OF, SF, ZF, PF eagerly in one style and a cc_op thunk in the other, writes no register for cmp, and no flag for movslq
• the translator refuses memory forms, other opcodes, extra REX bits, and wrong lengths
• the translator's cc_op numbers 8 and 7 equal the ones in the captured VEX
• the captured lifts give 9, 1, 10 P-code operations and 7, 4, 7 VEX statements for the three instructions
• the timeline runs from 1973 to 2026 in order, the pipeline has eight stages and six tools with a term in every cell, and the two closed tools are marked as not run
• the interactive figure embeds the same data the harness produces

Production Failure Modes

Operational Rule

Capture every tool once with its command and version, reduce every address to one base, and compare instruction starts before blocks and blocks before edges. Account for every byte as claimed, padding, data, or missed. Read a tool's own note before you judge its count, and say which tools you only read about.

Further Reading

• Cifuentes, "Reverse Compilation Techniques", PhD thesis, QUT (1994)
• Cifuentes and Gough, "Decompilation of Binary Programs", Software: Practice and Experience (1995)
• Van Emmerik, "Static Single Assignment for Decompilation", PhD thesis, University of Queensland (2007)
• Schwartz, Lee, Woo, Brumley, "Native x86 Decompilation Using Semantics-Preserving Structural Analysis and Iterative Control-Flow Structuring" (USENIX Security 2013)
• Yakdan et al., "No More Gotos" (NDSS 2015)
• Basque et al., "Ahoy SAILR! There is No Need to DREAM of C" (USENIX Security 2024)
• Andriesse et al., "An In-Depth Analysis of Disassembly on Full-Scale x86/x64 Binaries" (USENIX Security 2016)
• Flake, "Structural Comparison of Executable Objects" (DIMVA 2004)
• Maranzano and Bourne, "A Tutorial Introduction to ADB", Bell Labs (1977)
• Stallman, KTH lecture on GDB entering distribution (30 October 1986)
• Hex-Rays, "IDA: celebrating 30 years of binary analysis innovation"
• Guilfanov, "Decompiler internals: microcode", Recon Brussels 2018 (slides)
• Hex-Rays, microcode view and maturity levels
• Hex-Rays, IDA 9.4 release notes
• Hex-Rays, IDA Free
• Ghidra releases (12.1.4)
• Ghidra P-code reference
• Ghidra decompiler source, main work flow and Action classes
• Ghidra analyzeHeadless README
• Binary Ninja, BNIL overview
• Binary Ninja, "Architecture Agnostic Function Detection in Binaries" (2017)
• Binary Ninja, batch processing and headless licences
• Binary Ninja 6.0 "Krypton" release post
• Binary Ninja Free
• radare2 book, ESIL
• radare2 book, decompilers
• radare2 releases (6.2.4)
• rizin, RzIL design notes
• rizin releases (0.9.1)
• Announcing Rizin (8 December 2020)
• Cutter releases (2.5.0)
• angr documentation, CFG recovery
• angr documentation, the decompiler and its analysis passes
• angr documentation, intermediate representation (VEX)
• angr on PyPI (10.0.3)
• pypcode on PyPI
• llvm-objdump command guide
• GNU objdump manual
• LIEF documentation
• Capstone releases
• BinDiff releases (v8, open source)
• Diaphora
• Hopper Disassembler downloads
• Compiler Explorer
• LLDB announced as an LLVM subproject (2010)
• Boomerang decompiler
• Tan et al., "LLM4Decompile" (EMNLP 2024)
• Gao et al., "DecompileBench" (Findings of ACL 2025)
• Liu, Raff, Micinski, "When LLM Decompilers Recompile More and Preserve Less" (2026)

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.