Phase 13: Tools & Protocols

Dynamic Analysis of Your Own Program: Debuggers, Traces, and Confirming Static Claims

Type: Build

Languages: Python, C, arm64 assembly, lldb command language

Prerequisites: Phase 13 · 33 (disassembly and control flow), Phase 13 · 35 (evidence, confidence, and unknowns)

Time: ~130 minutes

The Problem

Lesson 41 read struct account from DWARF and said balance sits at offset 4. Lesson 33 built the control-flow graph of classify and said the first branch goes to the return -1 block when the balance is negative. Lesson 32 said the first two arguments arrive in x0 and x1. Every one of those sentences was derived from bytes. None of them was observed.

An agent then reads the same program and reports "main calls bump, classify, and sum_balances". The symbol table lists all three, so the sentence looks safe. You build the program at -O2, set a breakpoint on each function, and run it. The process prints its four lines and exits. Every breakpoint reports hit count = 0.

The agent was not wrong about the source. It was wrong about the build it was asked about. The compiler inlined all three calls and folded sum_balances(rows, 3) into the constant 315. The only way to know that for this binary on this machine is to run it under a debugger and look.

This lesson builds a harness that does the looking. It compiles account.c, reads the listing, plans one lldb session, runs it, and grades each static claim against what the stops show. The grading has three outcomes. confirmed means a stop showed the claim to be true for this run. refuted means a stop showed it to be false. not-observed means no stop could test it, and the record says why. Lesson 35 then stores each verdict as an observed record with authority controlled-replay.

A Debugger Is Three System Services and a Table

A debugger does four things to a process it does not own the code of. It stops the process, reads and writes its registers, reads and writes its memory, and gets told when the process stops again. The kernel provides each of these as a service, and the services differ by operating system.

On Linux the service is ptrace(2). The manual page says it "provides a means by which one process (the tracer) may observe and control the execution of another process". PTRACE_TRACEME lets a child announce that its parent will trace it, PTRACE_ATTACH takes an unrelated process, PTRACE_PEEKDATA reads "a word at the address addr in the tracee's memory", PTRACE_GETREGS reads the registers, and PTRACE_SINGLESTEP runs one instruction. The Yama module limits attach: /proc/sys/kernel/yama/ptrace_scope at its default of 1 means "the target process must be a descendant of the caller", 2 means only an administrator may attach, and 3 means nobody may.

On macOS ptrace is a smaller interface. Apple's manual lists PT_TRACE_ME, PT_ATTACHEXC, PT_STEP, PT_CONTINUE, and PT_KILL, and says that with PT_ATTACHEXC "Signals from the child go to the parent as Mach exceptions". Memory and registers do not go through ptrace at all. lldb's debugserver calls task_for_pid() to get a Mach task port for the process, then task_set_exception_ports() so that breakpoints arrive as exception messages. The system daemon taskgated decides who may call task_for_pid.

You can see the cost of that design in a transcript this lesson keeps. The session's shell runs under a Seatbelt sandbox. The same plan that produces eleven stops outside the sandbox produces this inside it:

textCopy(lldb) run
error: process exited with status -1 (attach failed ((os/kern) invalid address))

The process was created and then could not be adopted. The error is about the debugger's rights, and it says nothing about the program. The harness records it as a launch failure, with zero stops and no verdicts.

The table is the breakpoint table. The debugger remembers which instruction byte it replaced at which address, so it can put the original back when you continue, and it counts hits per location. breakpoint list prints that table, and the harness reads it at the end of every run.

Breakpoints: One Byte or One Word Replaced

A software breakpoint is an instruction the processor traps on, written over the first bytes of the instruction you want to stop at. On x86-64 that is INT3, opcode CC, one byte long, which the Intel manual describes as "intended for calling the debug exception handler". On arm64 it is BRK #imm16, a four-byte instruction whose exception "records the exception in ESR_ELx, using the EC value 0x3C". The Arm manual adds that the preferred return address is the BRK instruction itself, which is why a stop reports the address of the replaced instruction and not the one after it.

lldb sets breakpoints by name, by file and line, or by address. By name it skips the prologue:

textCopy(lldb) breakpoint set -n bump
Breakpoint 1: where = account-O0`bump + 12 at account.c:13:19, address = 0x00000001000004bc

bump + 12 is the fourth instruction. The three before it are sub sp, sp, #0x10, str x0, [sp, #0x8], and str w1, [sp, #0x4], the frame setup that the DWARF line table attributes to the function's opening line. At the stop, x0 and x1 still hold the arguments, because the prologue only copied them.

By address, lldb needs to know which module the address belongs to. A bare breakpoint set -a 0x100000504 before the process exists is a load address with no section behind it, and in a first test it stayed unresolved through the whole run with hit count = 0. The form the harness uses names the module:

textCopy(lldb) breakpoint set -a 0x100000504 -s account-O0
Breakpoint 4: address = 0x0000000100000504

With -s the number is a file address inside account-O0, lldb adds the load slide, and after the run the location reads resolved, hit count = 2. The lesson 33 listing gives file addresses, so this is the form that connects a static listing to a dynamic stop.

Decode the branch by hand

The first conditional branch in classify at -O0 is the word 36f800a8 at 0x1000004f0, which llvm-objdump prints as tbz w8, #0x1f, 0x100000504. Decode it from the Arm A64 encoding of TBZ.

1. Bit 31 is b5, the high bit of the bit number. It is 0.
2. Bits 30 to 25 are 011011, the test-and-branch group. Bit 24 is 0, which selects TBZ and not TBNZ.
3. Bits 23 to 19 are b40, the low five bits of the bit number: 11111 = 31. With b5 = 0 the tested bit is 31, the sign bit of a 32-bit value.
4. Bits 18 to 5 are imm14: 00000000000101 = 5. The branch offset is imm14 * 4 = 20 = 0x14, so the target is 0x1000004f0 + 0x14 = 0x100000504.
5. Bits 4 to 0 are Rt: 01000 = 8, so the register is w8.

w8 was loaded one instruction earlier by ldr w8, [x8, #0x4], which is a->balance at offset 4. TBZ branches when the bit is zero. The branch is taken when the sign bit is clear, which is balance >= 0. The source wrote if (a->balance < 0) return -1;, so the compiler inverted the condition: the taken edge skips the return -1 block, and the fall-through at 0x1000004f4 enters it. The static claim the harness carries is therefore an edge claim with a predicate: taken when a->balance >= 0, fall-through otherwise. The lift of the second branch works the same way. subs w8, w8, #0xc8 computes balance - 200 and b.le 0x100000524 is taken when that is at most zero, which is balance <= 200.

Watchpoints Are Hardware and Few

A watchpoint stops the process when memory at an address is accessed. It cannot be a replaced instruction, because any instruction anywhere might do the access. Processors therefore carry a small set of address comparators. On x86-64 they are the debug registers: DR0 through DR3 hold "the addresses of up to 4 breakpoints", and DR7 enables each one and sets its condition (execute, write, or read-write) and its length (1, 2, 4, or 8 bytes). On arm64 the pairs are DBGWVR<n>_EL1, which "holds the data virtual address used for watchpoint matching", and DBGWCR<n>_EL1 for the controls, and the Arm manual says "An implementation can include between 2-64 watchpoints".

lldb reports the number it was given:

textCopy(lldb) watchpoint set variable a->balance
Watchpoint created: Watchpoint 1: addr = 0x16fdfe00c size = 4 state = enabled type = m
(lldb) watchpoint list
Number of supported hardware watchpoints: 4

Four on this machine. The watched address 0x16fdfe00c is x0 + 4: the struct pointer at the bump stop plus the DWARF offset of balance. size = 4 is the int. type = m means modify. When the slots run out lldb refuses the next watchpoint, and because a region that is not aligned or not a power of two can take two slots, the number of watchpoints you may set can be smaller than four.

Data watchpoints report after the access. The Intel manual says "an instruction that writes memory overwrites the original data before the debug exception generated by a data breakpoint is generated", and lldb keeps the old value in its own cache to print both:

textCopyWatchpoint 1 hit:
old value: 250
new value: 255
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
    frame #0: 0x00000001000004d0 account-O0`bump(a=0x000000016fdfe008, by=5) at account.c:14:12

The stop is at 0x1000004d0, the instruction after str w8, [x9, #0x4] at 0x1000004cc. One quirk is worth knowing before you parse the output. Right after watchpoint set, lldb prints Watchpoint 1 hit: with an empty old value and new value: 250. That is the initial read, and the harness ignores a hit without an old value.

Stepping, Stop Hooks, and Batch Mode

lldb's command grammar is <noun> <verb> [-options] [arguments]. The stepping verbs are thread step-in (s), thread step-over (n), thread step-inst (si), thread step-inst-over (ni), and thread step-out (finish). thread until 12 runs to a line. thread return <expr> returns early with a value. The harness does not step. It stops at planned addresses and lets a stop hook do the reading:

textCopytarget stop-hook add -o "register read x0 x1 pc" -o "memory read -s4 -fx -c24 $x0" -o "frame variable -P 1"

-o may be repeated, and the commands run in order at every stop. register read names the two argument registers and the program counter. memory read -s4 -fx -c24 $x0 prints 24 four-byte words in hex from the address in x0, which is 96 bytes, three struct account rows. frame variable -P 1 prints the arguments and locals and expands pointers one level, so a shows its five fields.

Batch mode is lldb -b. The manual says --batch runs the commands from -s, -S, -o, and -O and then quits, and that if a command stopped "due to a signal or crash, the debugger will return to the interactive prompt". The harness writes the plan to a file and passes it with -s. One setting matters: settings set interpreter.stop-command-source-on-error false. Without it, the first continue after the process has exited ends the whole file, and breakpoint list never runs. With it, lldb prints error: Process must be launched. and goes on.

Two facts about the output shape decide how the parser works. First, hook output is printed as the hook runs, and the Process N stopped block with its frame #0 line is printed afterwards. The parser therefore collects register, memory, and variable lines into a pending chunk and attaches the chunk to the next frame #0 line it sees. A chunk whose program counter is in dyld at _dyld_start is the entry stop before the program runs, and no frame #0 ever claims it. Second, a breakpoint with --auto-continue does not report its stop at all, and a stop hook does not run for it. A first version of this harness used auto-continue and recorded nothing. The plan uses plain breakpoints and a fixed budget of continue commands instead.

The Plan Comes From the Listing

Plan.commands() builds the command file from the llvm-objdump -d text of the binary it will run. For account-O0:

textCopybreakpoint set -n bump
breakpoint set -n classify
breakpoint set -n sum_balances
breakpoint set -a 0x100000504 -s account-O0
breakpoint set -a 0x1000004f4 -s account-O0
breakpoint set -a 0x100000524 -s account-O0
breakpoint set -a 0x100000514 -s account-O0
target stop-hook add -o "register read x0 x1 pc" -o "memory read -s4 -fx -c24 $x0" -o "frame variable -P 1"
run
watchpoint set variable a->balance
watchpoint list
continue
...
breakpoint list
watchpoint list -v

The three name breakpoints are the function entries. The four address breakpoints are the target and the fall-through of each conditional branch that Listing.conditional_branches() finds in classify: tbz at 0x1000004f0 and b.le at 0x100000510. The watchpoint is set at the first stop, which the plan expects to be bump. Sixteen continue commands follow, more than any run needs, and the leftovers fail harmlessly after exit. The budget is not knowledge about the program. It is a ceiling.

The claims are built from the same inputs by claims_for():

The field offsets come from struct_layout(), which reads DW_TAG_structure_type with DW_AT_byte_size (0x20) and the five DW_TAG_member entries with their DW_AT_data_member_location out of the dSYM dump. The plan never types the number 4. It reads it.

The Run at -O0: Eleven Stops

The first stop is the bump breakpoint. The hook printed this before the stop block:

textCopy      x0 = 0x000000016fdfe008
      x1 = 0x0000000000000005
      pc = 0x00000001000004bc  account-O0`bump + 12 at account.c:13:19
0x16fdfe008: 0x00000073 0x000000fa 0x00000009 0x00000000
0x16fdfe018: 0x00000000 0x3ff40000 0x00000719 0x00000001
0x16fdfe028: 0x0000006c 0xffffffd8 0x00000003 0x00000000
0x16fdfe038: 0x00000000 0x40000000 0x00000721 0x00000001
(account *) a = 0x000000016fdfe008 {
  kind = 's'
  balance = 250
  code = 9
  rate = 1.25
  label = 0x0000000100000719 "savings"
}
(int) by = 5

Read the eight words of the first row against the DWARF offsets. Word 0 at offset 0 is 0x73, which is 's', the kind byte, followed by three padding bytes. Word 1 at offset 4 is 0xfa = 250, balance. Word 2 at offset 8 is 0x9, code, a short in the low half with two padding bytes above it. Word 3 at offset 12 is padding before the 8-byte-aligned double. Words 4 and 5 at offset 16 are 0x3ff4000000000000, which is the IEEE-754 encoding of 1.25. Words 6 and 7 at offset 24 are 0x100000719, the label pointer into __TEXT. The next eight words start with 0x6c = 'l' and 0xffffffd8 = -40: the third row, loan, 32 bytes later. Every number agrees with frame variable, so _check_layout() records field-offsets as confirmed at this stop, and it does so again at each of the eleven stops where x0 holds a struct pointer.

x0 equals a and x1 equals by, so arg-registers is confirmed at this entry. The harness compares only the declared arguments. At the classify stops x1 still reads 0x5, a leftover from bump, and classify has one argument, so nothing is compared against it. A register that no argument lives in is noise, and the lesson 32 claim says nothing about it.

The second stop is the watchpoint, old value 250, new value 255, by = 5. One hit in the whole run, so store-once is confirmed. Then come three activations of classify, each with its entry stop and one or two edge stops:

_check_branches() groups the stops into activations, reads the balance at the entry, and for each edge stop computes the successor the lifted predicate predicts. Five edge stops, five agreements: branch-1 confirmed three times, branch-2 twice. Stop 5 shows the balance as 255, which is the value the watchpoint saw written. The run is one program and the evidence connects.

At stop 10 the hook read 24 words from list. _check_stride() takes word 1, word 9, and word 17, which are offsets 4, 36, and 68: 100, 255, and -40. It compares each with the balance classify saw at the same address, sums them to 315, and compares the sum with the sum 315 line the program printed. Then the process exits with status 0 and breakpoint list shows hit counts of 1, 3, 1 for the names and 2, 1, 1, 1 for the four edges. All eight claims end confirmed, with 25 records behind them, 23 of them tied to a stop and two to the finished run.

The Run at -O2: Zero Stops

The same source at -O2 is a different program. llvm-objdump -d of main has four bl instructions, all to printf. In place of the calls it has mov w8, #-0x1, mov w8, #0x1, a zero through xzr, and mov w8, #0x13b, which is 315. The compiler computed every result at compile time. nm still lists _bump, _classify, and _sum_balances, because they are external functions and the out-of-line copies must exist for other objects to call. The dSYM has DW_TAG_subprogram entries for all four functions and zero DW_TAG_inlined_subroutine entries, so even the debug information does not record the inlining.

The plan for this build has three name breakpoints and no edge breakpoints. classify at -O2 is six instructions, ldr, cmp, cset, cmp, csinv, ret, with no conditional branch in it. The edge claims from the -O0 listing have no address in this binary, and the harness does not carry them over. A claim about an edge belongs to the build whose listing has the edge.

The run:

textCopy(lldb) breakpoint set -n bump
Breakpoint 1: where = account-O2`bump at account.c:13:16, address = 0x0000000100000460
(lldb) run
sizeof 32 align 8
offsets 0 4 8 16 24
class 0 1 -1
sum 315
Process 48444 exited with status = 0 (0x00000000)
(lldb) watchpoint set variable a->balance
error: Command requires a process which is currently stopped.
(lldb) breakpoint list
1: name = 'bump', locations = 1, resolved = 1, hit count = 0

Each breakpoint resolved. The address exists, the symbol exists, and lldb wrote a BRK there. The hit count is zero because no instruction fetch ever reached the address. The verdicts:

Be exact about what the refutation says. The claim was that main calls the three functions in this program. For this build and this run it does not, and the harness checked that the run completed, because a hit count of zero before exit would mean only that the run had not reached the call yet. The claim did not say the functions are dead code, and the run does not say that either: they are exported and another object could call them. The other four claims are neither confirmed nor refuted. The struct still has balance at offset 4 in this build, and the -O2 DWARF says so, but no stop in this run read memory to show it. not-observed keeps that distinction, and lesson 35 keeps the unknown open until a run does observe it.

Core Dumps

A core file is the process's memory and registers written to disk, so the debugger can read a stopped program after the program is gone. Apple's core(5) says "A small number of signals which cause abnormal termination of a process also cause a record of the process's in-core state to be written to disk", into /cores/core.pid, "provided the terminated process had write permission in the directory, and the directory existed". On this machine sysctl reports kern.coredump: 1 and kern.corefile: /cores/core.%P, /cores exists with mode drwxrwxr-t root admin, and the shell's ulimit -c is 0. No kernel core is written until you raise the limit with ulimit -c unlimited, or system-wide with launchctl limit core unlimited.

lldb writes a core on demand without a crash. At the bump breakpoint:

textCopy(lldb) process save-core --style stack account-O0-stack.core
Saving 49152 bytes of data for memory region at 0x16fdf4000

Modified-memory or stack-memory only corefile created.  This corefile may 
not show library/framework/app binaries on a different system, or when 
those binaries have been updated/modified. Copies are not included
in this corefile.  Use --style full to include all process memory.

The file was 53,248 bytes. Loading it needs the binary beside it:

textCopy(lldb) target create "./account-O0" --core "account-O0-stack.core"
(lldb) bt
* thread #1, stop reason = ESR_EC_BRK_AARCH64 (fault address: 0x0)
  * frame #0: 0x00000001000004bc account-O0`bump(a=0x000000016fdfe008, by=5) at account.c:13:19
    frame #1: 0x0000000100000640 account-O0`main at account.c:40:5
    frame #2: 0x000000019a949d54 dyld`start + 7184
(lldb) frame variable *a
(account) *a = (kind = 's', balance = 250, code = 9, rate = 1.25, label = "")
(lldb) frame variable a->label
(const char *) a->label = 0x0000000100000719 ""

The backtrace and the integer fields are intact, because they live on the stack that the core saved. label prints as an empty string with its pointer intact. The pointer 0x100000719 is correct; the bytes it points at are the literal "savings" in the __TEXT segment of the executable, and a stack-style core does not include that segment. The pointer is an observation. The string is not available, and lldb does not invent it. --style full includes all process memory, and a full core of this 34 KB program on the research machine was 2.1 GB, because the process maps the shared cache. The stop reason reads ESR_EC_BRK_AARCH64: the core recorded the BRK exception that the breakpoint raised.

On Linux the kernel writes the core according to /proc/sys/kernel/core_pattern, a template with %p for the PID, %e for the command name, %s for the signal, and %t for the time, and a leading | pipes the dump to a program. systemd distributions use that pipe for systemd-coredump, and coredumpctl list, info, dump, and debug retrieve what it stored. gdb prog core and lldb -c core prog both open the result.

gdb and lldb Side by Side

The two debuggers do the same work with different words. COMMAND_MAP in the harness holds 35 rows, printed by python3 code/main.py commands. The ones this lesson uses:

lldb's p is an alias of dwim-print, which prints a plain variable through the frame variable path and falls back to expression for anything else. v is frame variable. Prefer frame variable in a harness: it reads DWARF locations and never runs code in the debuggee.

Versions and licences on 2026-10-14. lldb carries the LLVM version number, and the current llvm.org release is LLVM 23.1.3 of 2026-10-06, under the Apache License 2.0 with LLVM exceptions. Homebrew ships it as the separate formula lldb. The Command Line Tools on this machine ship lldb-1703.0.236.21, Apple's own numbering, with an embedded Python 3.9.6, and Apple's lldb is already signed to debug. GDB 18.1 was released on 2026-09-25 under GPL-3.0-or-later. Its release notes list Windows, .gdb_index, AArch64 FPMR, and Python gdb.Corefile changes and no Darwin entry. GDB has no native support for Apple silicon processes through 18.1: a 15-patch "Initial AArch64-Darwin support" series was posted in October 2025, and in February 2026 a maintainer set out the route to land it through copyright assignment and the BFD patches first. The Homebrew gdb formula has arm64 bottles and builds with a faked --target=x86_64-apple-darwin20, and the pull request that added them says "it is not possible to debug aarch64-darwin binaries via GDB". Any non-Apple debugger on macOS also needs a self-signed certificate and the com.apple.security.cs.debugger entitlement, or task_for_pid fails with "(os/kern) failure (0x5)". gdb's place in this course is Linux, where Debian stable ships gdb 16.3 for arm64, and where rr lives.

Tracing: What the Program Asked the Kernel For

A debugger shows the program's state at a stop. A tracer shows the program's effects on the system without stopping it for long. The tools are per operating system.

dtruss says in its manual that "Since this uses DTrace, only users with root privileges can run this command". Apple's dtrace(1) adds that "With default System Integrity Protection settings, D programs are not permitted to access kernel address values or kernel memory contents" and that destructive actions are not allowed under SIP. sudo dtruss ./account-O0 on this machine was not run in this lesson, because the harness runs without root, and the lesson records that as unverified. A trace answers a different question from the harness: "did this run issue a write for each printf line" is a trace question, "did x1 hold 5 at bump" is a debugger question. Both are per run.

Record and Replay

rr records a run once and replays it deterministically under gdb, with reverse execution: set a watchpoint on the value you care about and reverse-cont runs backwards to the write. The project describes itself as "You record a failure once, then debug the recording, deterministically, as many times as you want". The current release is 5.9.0 of 2025-02-13, under the MIT licence with a BSD-2-Clause block. It is Linux only. It needs hardware performance counters to count branches, so it runs on Intel Nehalem and later, certain AMD Zen cores, and "certain AArch64 microarchitectures (e.g. ARM Neoverse N1 or the Apple Silicon M-series)" under Linux. Inside a virtual machine the counters must be virtualized, and the project wiki says "No other virtualization on Mac does either", so Docker and Colima on this machine cannot run it. Pernosco is the commercial service built on rr recordings.

gdb has a software recorder of its own. record full records "a log of the process execution" with a default limit of 200,000 instructions, and record btrace uses Intel branch tracing and "does not record data". Neither is available in lldb. On this macOS machine, the harness's saved transcript is the record: the same plan on the same binary produces the same stops in the same order, and the fixture tests pin that order.

What One Run Proves

One run proves one input on one build. The C standard defines undefined behaviour as "behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this document imposes no requirements", so a program that overflows a signed int may legitimately print different results at -O0 and -O2. Lesson 36 met this with the percent_change overflow cells. UBSan turns the question into a stop: -fsanitize=signed-integer-overflow -fsanitize-trap=undefined makes the overflow "execute a trap instruction", which lands in lldb as a stop with a reason.

Optimized code also moves what you want to watch. GDB's manual says "the optimizer has rearranged your code; the debugger shows you what is really there" and that a variable optimized out of existence is one "GDB never sees". GDB prints such a value as <optimized out>, and its source separates that from <unavailable>: optimized out "does not actually exist in the program", unavailable "exists in the program (at the machine level), but GDB has no means to get to it". lldb prints <variable not available> when the program counter is outside every range of the variable's DWARF location list. A watchpoint on a local that lives in a register never fires, because a register is not memory. The -O2 run above shows the extreme case: the whole function was folded away, and the silence of a watchpoint that was never set proves nothing.

The harness therefore writes three kinds of record and never a fourth. A stop that tested the claim and agreed is confirmed. A stop that tested it and disagreed is refuted. A claim no stop could test is not-observed, with the reason, and the breakpoint hit counts are part of that reason. Lesson 35's ledger accepts the first two as observed records with authority controlled-replay, and keeps the third as an open unknown.

Debugging Only Your Own Builds on macOS

On Apple silicon the linker signs for you. Apple's ld(1) says of -adhoc_codesign: "The default for Apple Silicon binaries is to be ad-hoc codesigned." codesign -dvvv account-O0 on the fixture prints:

textCopyFormat=Mach-O thin (arm64)
CodeDirectory v=20400 size=387 flags=0x20002(adhoc,linker-signed) hashes=9+0 location=embedded
Signature=adhoc
TeamIdentifier=not set

0x20002 is CS_ADHOC (0x2) plus CS_LINKER_SIGNED (0x20000), "Automatically signed by the linker" in Apple's code-signing header. codesign -d --entitlements - prints nothing after the executable name: the binary has no entitlements. lldb launched it, hit every breakpoint, set a hardware watchpoint, and wrote a core, with DevToolsSecurity -status reporting "Developer mode is currently disabled" and System Integrity Protection enabled.

Two entitlements control who may debug what, and they sit on opposite sides. com.apple.security.cs.debugger is for the debugger: "A Boolean value that indicates whether the app is a debugger and may attach to other processes or get task ports". Apple's debugserver carries the private form of it, and codesign -d --entitlements - on the Command Line Tools copy lists com.apple.private.cs.debugger, com.apple.private.set-exception-port, com.apple.private.thread-set-state, and com.apple.private.logging.diagnostic. com.apple.security.get-task-allow is for the target. Xcode injects it into debug builds, strips it on export, and notarization rejects a binary that still has it. Apple's engineers state the rule for the refusal: the task port is denied when the target has the hardened runtime enabled and does not have get-task-allow. A plain clang build is not hardened and triggers neither condition. If you ever sign a fixture with codesign --options runtime, re-sign it with codesign -s - -f --entitlements get-task-allow.plist before debugging.

Three more facts close the section. First, you may see the dialog "Developer Tools Access needs to take control of another process for debugging to continue" the first time in a login session; sudo DevToolsSecurity -enable removes the extra password for members of the admin or _developer groups. Second, you do not disable SIP to debug your own code. Apple says "You don't need to disable SIP to run and debug apps from Xcode", and the only SIP component that touches debugging is the one csrutil enable --without debug relaxes, for other people's processes. Third, the Seatbelt sandbox in this session blocks task_for_pid entirely, and the fixture sandbox-O0.lldb.txt shows the result. A run that cannot start is an environment fact, and the harness reports it as one.

Everything in this lesson applies to programs you compiled. The same commands would attach to another process only if its owner opted in, and nothing here teaches how to change that.

Build It

Run the demo:

bashCopycd phases/13-tools-and-protocols/42-dynamic-analysis-of-your-own-program
python3 code/main.py

The output is one JSON report over the saved transcripts. For each build it lists the breakpoints with their resolution and hit counts, the hardware watchpoint count, every stop with its registers and arguments, the exit status, the program's output, the errors lldb printed, and the claims with their verdicts and the evidence records behind them. It adds the core round trip, the Python API results, the sandbox failure, and which external tools are absent on this machine.

Expected shape:

jsonCopy{
  "program": "account.c",
  "variants": {
    "O0": {
      "binary": "account-O0",
      "breakpoints": {"1": {"spec": "name=bump", "address": "0x1000004bc", "resolved": true, "hits": 1}, "...": "..."},
      "hardwareWatchpoints": 4,
      "watchpoints": {"1": {"address": "0x16fdfe00c", "size": 4, "type": "m", "hits": [[250, 255]], "hitCount": 1}},
      "stops": [{"index": 0, "pc": "0x1000004bc", "function": "bump", "kind": "entry", "registers": {"x0": "0x16fdfe008", "x1": "0x5"}, "...": "..."}],
      "exitStatus": 0,
      "claims": [{"id": "arg-registers", "verdict": "confirmed", "confirmed": 5, "refuted": 0, "not-observed": 0, "...": "..."}],
      "evidence": [{"claim": "arg-registers", "verdict": "confirmed", "stop": 0, "observation": "bump: x0 = 0x16fdfe008 and a = 0x000000016fdfe008, x1 = 0x5 and by = 5"}]
    },
    "O2": {"stops": [], "claims": [{"id": "calls", "verdict": "refuted", "...": "..."}], "...": "..."}
  },
  "core": {"savedAt": {"function": "bump", "pc": "0x1000004bc"}, "labelString": "", "...": "..."},
  "types": {"python": "3.9.6", "size": 32, "...": "..."},
  "sandbox": {"stops": 0, "errors": ["process exited with status -1 (attach failed ((os/kern) invalid address))"]},
  "absentTools": ["bpftrace", "gdb", "llvm-objdump", "ltrace", "rr", "strace"]
}

Read code/main.py in this order:

1. Listing parses the llvm-objdump -d text into functions and instructions, and Function.conditional_branches() finds the branches with their targets and fall-throughs.
2. struct_layout() reads the structure and member entries from the llvm-dwarfdump text.
3. lift_condition() turns one conditional branch into a predicate, the lesson 34 step for tbz, cbz, and the compare-and-branch forms.
4. Plan.commands() writes the lldb command file from the listing, and Plan.lldb_command() is the batch invocation.
5. parse_transcript() walks the output in order and builds Stop records with registers, memory, variables, and watchpoint values, plus the breakpoint table and the exit status.
6. claims_for() states the eight claims, and evaluate() grades them stop by stop with the _check_* functions.
7. rollup() reduces the records to one verdict per claim, with refuted taking precedence over confirmed.
8. capture() and capture_extras() produced the fixtures, and live() repeats the whole loop on a fresh build when clang and lldb can run.

Four more commands exist. python3 code/main.py plan O0 prints the command file for a build. python3 code/main.py commands prints the gdb and lldb table. python3 code/main.py tools reports which external tools are on PATH. python3 code/main.py live O0 compiles account.c into a temporary directory, plans from its fresh listing, runs lldb, and grades the claims against that run. Inside a sandbox it reports that lldb could not launch the process, with the error, and exits 1.

Use It

Use the harness on a program of your own in this order.

1. Build it with -g and the optimization level you will ship. Keep the binary, its dSYM or its DWARF, and the llvm-objdump -d text together.
2. Write the claims before you run. Each one names a function, a register or an address or an offset, an expected value or edge, and the lesson or document it came from. A claim you write after seeing the stop describes the stop and tests nothing.
3. Generate the plan from the listing and read it. Every address in it should trace to a line in the listing. If a breakpoint has no such line, remove it.
4. Run once in batch mode and keep the transcript with the command line and the debugger version in its header. The transcript is the evidence. The verdicts are derived from it, and anyone can re-derive them.
5. Grade. Read every not-observed record and its reason before you read the confirmations. The reasons tell you which claims need another input, another build, or another breakpoint.
6. Run the optimized build with the same claims. Expect resolved breakpoints with zero hits where the compiler inlined, and keep those claims open.
7. Feed the confirmed and refuted records into the lesson 35 ledger as observed, authority controlled-replay, one record per stop. Leave the not-observed ones as unknowns with the breakpoint hit counts attached.

Interactive Lab

Lab A: read a stop by hand

From the code directory, open Python:

bashCopycd phases/13-tools-and-protocols/42-dynamic-analysis-of-your-own-program/code
python3 -q

Run:

pythonCopyfrom main import *
t = load_transcript("session-O0.lldb.txt")
s = t.stops[0]
hex(s.pc), s.function, s.reason, s.registers
[hex(w) for w in s.memory[:8]]
s.struct_pointer().fields

Before the second line, write down which word you expect to hold 250 and which two words hold 1.25. Then run struct.unpack("<d", struct.pack("<II", s.memory[4], s.memory[5])) and compare. Then look at t.stops[1].watch_hit and t.stops[1].pc, and find that address in account-O0.objdump.txt: it is the instruction after the store.

Lab B: the branch and its successor

pythonCopyplan = make_plan("O0")
plan.branches()
classify = plan.listing.function("classify")
[lift_condition(classify, b, "a->balance")[0] for b in plan.branches()]
[(hex(s.pc), s.struct_pointer().field_int("balance")) for s in t.stops_in("classify")]

Match each edge stop to its entry stop and decide, before you look at the evidence, which successor the predicate predicts. Then run [e.to_dict() for e in evaluate(plan, t, struct_layout(open("fixtures/account-O0.dwarfdump.txt").read(), "account")) if e.claim.startswith("branch")] and compare.

Lab C: the run that never stopped

pythonCopyo2 = load_transcript("session-O2.lldb.txt")
len(o2.stops), o2.exit_status, o2.hits()
o2.errors[:3]
make_plan("O2").branches()
rollup(evaluate(make_plan("O2"), o2, struct_layout(open("fixtures/account-O2.dwarfdump.txt").read(), "account")))

Explain each not-observed in your own words, then open account-O2.objdump.txt and find the four constants in main that replaced the calls. Then count DW_TAG_inlined_subroutine in account-O2.dwarfdump.txt with count_tags().

Lab D: break the evidence

pythonCopytext = open("fixtures/session-O0.lldb.txt").read().replace("x1 = 0x0000000000000005", "x1 = 0x0000000000000006", 1)
broken = parse_transcript(text)
[e.to_dict() for e in evaluate(plan, broken, struct_layout(open("fixtures/account-O0.dwarfdump.txt").read(), "account")) if e.claim == "arg-registers"][:2]

One edited register turns the first arg-registers record to refuted and the rollup with it, while the four other entries stay confirmed. Then delete the old value: 250 line before the watchpoint stop and watch store-once fall to not-observed. A grader that cannot be made to fail is not grading.

If lldb can launch a process in your shell, finish with python3 main.py live O0 and python3 main.py live O2. The stack addresses will differ from the fixtures. The verdicts should not.

Practice Lab

Extend the harness with a claim about the loop in sum_balances.

Requirements:

• Find the loop back edge of sum_balances in the -O0 listing with conditional_branches() and add a breakpoint at the loop header.
• Add a claim that the header executes count + 1 times for count = 3, which is the lesson 33 back edge plus the exit test.
• Extend the stop hook or the plan so the hook reads the loop counter's stack slot at each header stop, and write a _check_loop() that records the sequence of counter values.
• Re-capture the transcript with python3 main.py capture O0 on a machine where lldb can launch, and re-run the suite. The plan test will fail until the committed command file matches the new plan, which is the point of that test.
• Add a test that edits one counter value in the transcript and asserts the claim turns refuted.

Success means the report shows the header stops in order with their counter values, and the -O2 run keeps the new claim not-observed with the reason that the loop was vectorised or folded away.

Lesson Artifact

This lesson includes outputs/skill-dynamic-analysis-of-your-own-program.md. Use it to turn a program you built, its listing, and a list of static claims into one saved debugger transcript and one verdict per claim, with the stop that produced each verdict and the reason for every claim that stayed not-observed. It applies to Claude, Cursor, Codex, OpenClaw, Hermes, or any agent that reads SKILL.md.

Verify It

Run the demo and the deterministic suite:

bashCopycd phases/13-tools-and-protocols/42-dynamic-analysis-of-your-own-program
python3 code/main.py
python3 -m unittest discover -s code/tests -v

The suite runs without lldb. It reads the transcripts captured once into code/fixtures/. Verification should prove:

• the listing parser finds the four functions with their ranges and the two conditional branches of classify at -O0 with target and fall-through
• the lift of tbz w8, #0x1f is a->balance >= 0 and the lift of subs plus b.le is a->balance <= 200
• main at -O0 calls bump, classify three times, and sum_balances, and main at -O2 calls only printf
• x86-64 j<cc> mnemonics are conditional and jmp and callq are not
• the DWARF dump yields a 32-byte struct with members at 0, 4, 8, 16, and 24
• the -O2 DWARF has four subprograms and zero inlined-subroutine entries, and nm still lists the three functions
• the plan equals the committed command file for both builds, and the transcript header records the lldb command that ran it
• the -O0 transcript has eleven stops in the expected functions, exit status 0, and the four output lines
• hook output is attached to the stop that follows it, with x1 = 5, memory starting at x0, and the label string
• the breakpoint table carries resolution and hit counts 1, 3, 1, 2, 1, 1, 1
• the watchpoint is 4 bytes of type modify at x0 + 4, hit once with old value 250 and new value 255, and four hardware slots are reported
• the -O2 transcript has zero stops, three resolved breakpoints with zero hits, no watchpoint, and one error per unused continue
• the sandboxed launch fails before any stop with the attach error
• the core round trip keeps the backtrace and balance = 250 and loses the string behind label
• the Python API reads the five fields, the size 32, and the function ranges without a process
• a hand-written x86-64 transcript parses with rdi, rsi, and rip
• every claim is confirmed at -O0 with 5, 11, 3, 2, 1, 1, 1, 1 records
• the branch evidence names the value, the condition, and the edge
• the field-offset evidence decodes the words through the DWARF layout
• the store and stride evidence carry the old and new value and the three row balances
• the -O2 run refutes the call edges and leaves the other claims not-observed
• an edited register value is refuted, a deleted watchpoint hit is not-observed, and refuted wins the rollup
• the command map has gdb and lldb forms for every row, and the hardware table names four x86 debug registers
• the demo lists absent tools instead of failing, and live mode reports the missing tool by name
• the figure data comes from both sessions and the site module embeds the same data

Production Failure Modes

Operational Rule

Plan the stops from the listing, run once in batch mode, keep the transcript, and grade every static claim at every stop that can test it. Write confirmed or refuted only from a stop, write not-observed with the hit counts and the reason, and never let a resolved breakpoint stand in for a run. Debug only the builds you made.

Further Reading

• LLDB Tutorial: command grammar, breakpoints, watchpoints, stop hooks
• lldb(1) manual page: --batch, --one-line, --source, --core
• GDB to LLDB Command Map
• LLDB Python Reference: SBTarget, SBFrame, SBValue, and the embedded interpreter
• LLDB Caveats: the Python build lldb links against
• LLVM 23.1.3 release
• GDB 18.1 release announcement, 2026-09-25
• GDB manual: Process Record and Replay
• GDB manual: Debugging Optimized Code
• GDB wiki: Permissions on Darwin, code signing a debugger
• gdb-patches: Initial AArch64-Darwin support series, October 2025
• Homebrew gdb formula pull request adding arm64 bottles
• ptrace(2), Linux manual page
• lldb debugserver MachTask.mm: task_for_pid and task_set_exception_ports
• Intel SDM Volume 2: INT n, INTO, INT3, INT1
• Intel SDM Volume 3: debug registers DR0 to DR7 (Debug, Branch Profile, TSC, and Resource Monitoring Features)
• Arm A64 ISA: BRK, Breakpoint instruction
• Arm A64 ISA: TBZ, Test bit and branch if zero
• Arm Architecture Reference Manual, D2.9 Watchpoint exceptions
• Apple ld(1): -adhoc_codesign
• Apple ld64 LinkEdit.hpp: CS_LINKER_SIGNED
• Apple: Debugging Tool Entitlement, com.apple.security.cs.debugger
• Apple: Hardened Runtime
• Apple: Resolving common notarization issues, the get-task-allow entitlement
• Apple: Disabling and Enabling System Integrity Protection
• Apple Developer Forums: debugserver may not get the taskport of a hardened process
• Apple DevToolsSecurity(8)
• Apple core(5)
• Apple Technical Note TN2124: Mac OS X Debugging Magic, core dumps
• lldb process save-core help text
• core(5), Linux manual page
• coredumpctl(1)
• Apple dtrace(1)
• Apple dtruss(1m)
• strace releases
• ltrace
• bpftrace CHANGELOG
• rr project
• rr 5.9.0 release
• rr wiki: Building And Installing, hardware and VM requirements
• Clang UndefinedBehaviorSanitizer
• ISO/IEC 9899:2024 working draft N3220, clause 3.5 undefined behavior
• DWARF Debugging Information Format Version 5
• Procedure Call Standard for the Arm 64-bit Architecture (AAPCS64)
• System V Application Binary Interface, AMD64 Architecture Processor Supplement

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.