A continuation from the previous post, on writing a debugger in Rust for x86_64 GNU/Linux…
Going back to the first part in this series of blog post, we implemented a base set of 7 commands (including quit and help) that showcased very trivial utilities available to a debugger. The great thing about those few commands were that they were very simple and straightforward to implement, requiring a single ptrace system call each! The reader might be almost mislead to believe that a breakpoint command might follow the same path, but alas, this is not so. There is no single ptrace message that you can send to your tracee that will implement such a faculty.
Looking at what a breakpoint does on a piece of paper, it is unreasonable to think that such a complex feature could be boiled down to one ptrace system call. It takes in a line, function of source code that a process is running, and somehow translates that to somewhere in the running executable, and forces that tracee to stop there so we can peek at its state at that point. This is not very trivial at all! Rather than relying on the OS system calls exclusively, a breakpoint needs debug information and additional tracer state. It will also require platform architecture specific understanding in order to implement.
int 3, Seizing Control
So the question is how does one stop a running program in its tracks at an arbitrary place in program state? Well, this question is often inextricably tied to platform-dependent details, so suffice to say, it often comes with a series of follow-up inquiries. For the case of our target platform, x86_64 GNU/Linux, our journey begins with Intel interrupts.
Interrupts are signals which are handled in a very dynamic way by a CPU. Many of them are tied to hardware itself, triggered by things like clock cycles, external hardware. Without interrupts, our OS would not be able to evict a running process to run others with a fair schedule. Without interrupts, this paragraph I type out on this keyboard on right now would be heartlessly ignored and reduced to a series of audible mechanical clicks.
Interrupts are also originate in software itself, usually allow us to signal to an OS that we need it to do something on behalf of us (system call). This is accomplished through the int opcode for Intel systems. Think back to the assembly we were using in the first post on debuggers. It uses an interrupt to make a system call to write to STDOUT the message provided below, context switching to OS to write words to screen.
.global _start
.data
msg:
.ascii "John Dorman here!\n"
len = . - msg
.text
_start:
# we want to write msg to screen, or STDOUT, to do this, need to pass args
# to a system call, syscall write is 4
# arguments for that are (file_descriptor, msg, len)
mov $4,%eax # eax register gets syscall code
mov $1,%ebx # ebx gets 1, or STDOUT file descriptor
mov $msg,%ecx # the message pointer goes here
mov $len,%edx # the length of message goes here
int $0x80 # interrupt triggered here, arguments passed to OS, write
# to screen happensHere’s the where int comes into play for our debugger. There is a interrupt specifically for stopping, or trapping a running process so that one may look at that process at that state. int 3, when triggered, will generate a software interrupt to trap a process, ceasing execution for a debugger to handle it. Now, we can freely write these into our code by just inserting them manually into disassembly, or as inlined assembly in Rust:
xor %eax, %eax
movq $1, %eax
int 3 # I want my program to stop here for my debugger!
addq $1, %eaxuse core::arch::asm;
// This macro will insert inlined assembly into Rust code
// more on that here https://doc.rust-lang.org/core/arch/macro.asm.html
let var = 2 + 2;
asm!("int 3", ...);
var -= 1;However, this is a very manual and exhausting approaching to laying traps for a process for debugging. A debugger is able to do that in a running process with the breakpoint command, translating a line of source code to an address for an instruction, overwriting that instruction with the int 3 opcode. What’s more, it keeps the original instruction it overwrites to be later written back to the process so that, when a developer continues running after the breakpoint, the program counter of CPU is rolled back, the original instruction is replace and the CPU continues running the original code.
So, we know how to trap a process at random places in code, the next problem is the translation of arbitrary places in source code into that of the address to the associated instruction of a running process. This requires debug information.
DWARF, Line Programs: How Source Code to Machine Code Address Resolution Works
DWARF is a debugging information standard that has been widely utilized within GNU/Linux spaces, especially in the context of ELF object formats. It has extensive support across different compilers and associated debuggers. If you have not heard of it, well, now you have. Writing a debugger precipitates reading debug information like DWARF, and DWARF is too common not to support!
Source line to address translation, in particular, is supported by a specific DWARF section called .debug_lines. The DWARF5 standard document describes this section in great detail. They first cite a trivial approach by providing a mapping a manner one might do if given a large CSV file. With every row an instruction, and columns to identify fields like source line, source column, source file, basic block status, so on and so forth. They then posit…
Such a matrix, however, would be impractically large. We shrink it with two techniques. First, we delete from the matrix each row whose file, line, source column and discriminator is identical with that of its predecessors. Any deleted row would never be the beginning of a source statement. Second, we design a byte-coded language for a state machine and store a stream of bytes in the object file instead of the matrix. This language
DWARF5 Standard Documentation
can be much more compact than the matrix. To the line number information a consumer must “run” the state machine to generate the matrix for each compilation unit of interest. The concept of an encoded matrix also leaves room for expansion. In the future, columns
can be added to the matrix to encode other things that are related to individual instruction addresses.
This solution then becomes one of creating a program within DWARF in acts upon virtual registers of a machine to calculate a given mapping. Thankfully, another Rust third-party crate gives us a means to do just that.
Accessing DWARF within the Debugger with the Gimli Crate
To join the naming scheme of fantastical creatures of the likes of elves and dwarves, the Rust third party ecosystem produced a crate call gimli, christened in honor of Tolkien’s hearty warrior of Middle Earth. With this Rust crate, in combination with the object crate for reading ELF, we can begin reading DWARF line programs.
Gimli provides an optimized copy-on-write model for inspecting DWARF data. Operating on references alone, until you write, you can use read features of gimli with little overhead. Facilities for specifically for gathering line programs from .debug_lines are provided within.
Using gimli begins by finding the reference to the start of the debug information. Passed a reference to ELF file loaded into memory, attempt to get the debug sections. I’ll spare you the gorier details, as the library is pretty thorough. Suffice to say, this code will allow me to retrieve the address for a given file, line in source code.
use gimli::{self, Dwarf};
use object::{Object, ObjectSection};
use std::borrow;
use std::cell::Ref;
use std::error::Error;
pub fn load_dwarf_data(f_buf: &[u8])
-> Result<Dwarf<borrow::Cow<'_, [u8]>>, Box<dyn Error>> {
let elf_obj = object::File::parse(&*f_buf)?;
let section_loader = |section: gimli::SectionId| -> Result<borrow::Cow<[u8]>, gimli::Error> {
match elf_obj.section_by_name(section.name()) {
Some(ref section) => Ok(section
.uncompressed_data()
.unwrap_or(borrow::Cow::Borrowed(&[][..]))),
None => Ok(borrow::Cow::Borrowed(&[][..])),
}
};
let dwarf_cow = gimli::Dwarf::load(& section_loader)?;
Ok(dwarf_cow)
}
pub fn src_line_to_addr(
dwarf_cow: Ref<'_, Dwarf<borrow::Cow<'_, [u8]>>>,
filename: &str,
line_num: u64,
) -> Result<u64, Box<dyn Error>> {
let dwarf = dwarf_cow
.borrow(|section| gimli::EndianSlice::new(&*section, gimli::RunTimeEndian::Little));
let mut iter = dwarf.units();
while let Some(header) = iter.next()? {
let unit = dwarf.unit(header)?;
// Iterate over the Debugging Information Entries (DIEs) in the unit.
let mut entries = unit.entries();
if let Some((_depth, top_die)) = entries.next_dfs()? {
if let (Ok(Some(_comp_dir_atval)), Ok(Some(name_atval))) = (
top_die.attr_value(gimli::DwAt(0x1b)),
top_die.attr_value(gimli::DwAt(0x03)),
) {
let cu_name = dwarf.attr_string(&unit, name_atval)?.to_string()?;
if cu_name == filename {
let line_prog = unit.line_program.unwrap();
let mut rows = line_prog.rows();
while let Ok(Some((_header, row))) = rows.next_row() {
if let Some(l) = row.line().map(u64::from) {
if l == line_num {
return Ok(row.address());
}
}
}
}
}
}
}
Err(Box::new(gimli::Error::InvalidAddressRange))
}
Thus we have a means of resolving source lines to instruction addresses run by CPU. But we are not done yet with address resolution. An address value derived from DWARF data could represent two different things depending on the ELF object file type.
Handling Position Independency and ASLR
For any given ELF object file in GNU/Linux, there is an associated ELF type. There are several types, but the two that play a key part in DWARF address values are ET_EXEC and ET_DYN. With a type of ET_EXEC, the address derived from DWARF will be just that–the address to that instruction. If that file is ET_DYN however, it becomes the offset from the base section in memory.
Historically, ET_EXEC served as the type of an executable object file, i.e. your program your run ./myprogram. ET_DYN has been referred to as the shared object format, or shared library format, which are the library files with affixed extension .so. These cannot be executed, but serve your executable as libraries for existing code. What’s more is that these can be loaded and placed anywhere in memory dynamically, and shared across process space.
Nowadays though, on most GNU/Linux systems, when you compile your program, you will recognize that even executables are of type ET_DYN. Below is an example ELF executable produced with clang test.c -o test.
traceedb$ readelf -h test
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x1050
Start of program headers: 64 (bytes into file)
Start of section headers: 14600 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 38
Section header string table index: 37The reason for this is security. Modern operating systems enforce address space layout randomization, or ASLR, by default. An executable can take advantage of this if it is dynamically locatable with ET_DYN. Compilers default to this as well if the option is available, forcing you to use flags to specify otherwise.
The first order of business for actually insuring the address we obtain from DWARF is indeed the correct address is 1) turning randomized addresses of for the child, and 2) determining ELF type, and if dynamic, determining base segment address to add to address value. Number one is simple, and requires an additional cargo feature and two lines before running the tracee execv system call.
// In run_target(), needs personality feature of nix, so added that to cargo
let tracee_persona = personality::get()
.expect("Critical Err: cannot get process persona!");
personality::set(tracee_persona | personality::Persona::ADDR_NO_RANDOMIZE)
.expect("Critical Err: cannot set tracee process personality!");
The second, however, is not so straightforward. Determining base segment for code is something that can only be done at runtime, when the process is delegated to a virtual memory region. Luckily, just about every resource in GNU/Linux is a file, and this includes memory maps. We can inspect /etc/proc and derive segment data for our running processes by looking up process ID.
nextdb$ cat /proc/145087/maps
555555554000-555555555000 r--p 00000000 08:20 6124 /home/jdorman/projects/nextdb/test
555555555000-555555556000 r-xp 00001000 08:20 6124 /home/jdorman/projects/nextdb/test
555555556000-555555557000 r--p 00002000 08:20 6124 /home/jdorman/projects/nextdb/test
555555557000-555555559000 rw-p 00002000 08:20 6124 /home/jdorman/projects/nextdb/test
7ffff7fbd000-7ffff7fc1000 r--p 00000000 00:00 0 [vvar]
7ffff7fc1000-7ffff7fc3000 r-xp 00000000 00:00 0 [vdso]
7ffff7fc3000-7ffff7fc5000 r--p 00000000 08:20 250145 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffff7fc5000-7ffff7fef000 r-xp 00002000 08:20 250145 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffff7fef000-7ffff7ffa000 r--p 0002c000 08:20 250145 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffff7ffb000-7ffff7fff000 rw-p 00037000 08:20 250145 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0 [stack]This can only be determined at runtime programmatically by yet another third party crate, procmapsrs, to parse running process segment addresses found in /proc/. This is only for the position independent case. If the ELF header was ET_EXEC, then the DWARF address would be the address and not the offset.
// Only needed for PIC code, where DWARF addresses become offsets from base
// segment. Need to determine at runtime where the base is by parsing with
// the procmaps crate and finding address.
fn get_segment_base_addr(pid: Pid) -> Result<usize, &'static str> {
Mappings::from_pid(pid.into())
.map_err(|_| "Failed to find segment base")?
.iter()
.nth(0)
.map(|mem_region| mem_region.base)
.ok_or("Failed to find segment base")
}Tying Everything Together
The breakpoint command can be implemented in the same manner as the previous commands, that is, via a struct and an implementation block for the Execute interface. This time, however, the command will actually have to deal with internal state. Breakpoints are recorded down in debuggers like gdb/lldb. All commands up to this point can be considered stateless as they do not modify the TraceeDbg struct in any way.
// command.rs
pub struct Breakpoint(pub u64);
impl Execute for Breakpoint {
fn execute(&self, pid: Pid, is_et_dyn: bool)
-> Result<TargetStat, &'static str> {
let mut text_addr = self.0;
// If this ELF is ET_DYN -> DWARF address needs offset
if is_et_dyn {
text_addr += get_segment_base_addr(pid)? as u64
}
let brkptrec = BrkptRecord::new(pid, text_addr as *mut c_void);
Ok(TargetStat::BreakpointAdded(brkptrec))
}
}
define_help!(
Breakpoint,
"b/breakpoint <file:line> = a standard breakpoint"
);
// breakpoint.rs
use nix::{errno::Errno, libc, sys::ptrace, unistd::Pid};
use std::ffi::{c_uint, c_void};
use std::ptr;
#[derive(PartialEq, Eq, Debug)]
pub struct BrkptRecord {
// I am lazy and made these all public for now
pub pid: Pid,
pub pc_addr: *mut c_void,
pub original_insn: i64,
}
impl BrkptRecord {
pub fn new(pid: Pid, text_addr: *mut c_void) -> Self {
let original_insn =
ptrace::read(pid, text_addr)
.expect("Failed to read text region for breakpoint!");
Self {
pid,
pc_addr: text_addr,
original_insn,
}
}
pub fn activate(&self) {
// 0xCC is the bytecode for int 3
let trap = ((self.original_insn & 0xFFFFFF00) | 0xCC) as *mut c_void;
unsafe {
ptrace::write(self.pid, self.pc_addr, trap).unwrap();
}
}
pub fn recover_from_trap(&self) {
println!("Recovering from trap!");
// Write the original instruction back, overwriting trap
unsafe {
ptrace::write(
self.pid, self.pc_addr, self.original_insn as *mut c_void
)
.expect("FATAL: failed to write with PTRACE_POKEDATA");
}
// Be sure you reset program counter
// so we point to original instruction!
let mut regs = ptrace::getregs(self.pid)
.expect("FATAL: Failed to send PTRACE_GETREGS");
regs.rip -= 1;
ptrace::setregs(self.pid, regs)
.expect("FATAL: Failed to send message PTRACE_SETREGS");
}
}The breakpoint command is what produces the associated BrkptRecord that keeps track of the breakpoint for the tracer. It has a small interface that can be used to activate the breakpoint, recover in the event we do hit it when running the process. You could theoretically continue to add to this interface, but I will leave it as is for now.
Parsing the breakpoint command was interesting, but was made exceedingly easy with the help of functional Rust and common results and options. I made it accept a string argument of form <file>:<line_number>. Parsing that, it then needs to pass the file and line number to the resolution function for running line programs to get the address to the right instruction. It also requires that the debug symbols are there from gimli. Otherwise, it reports an error.
// To parse for the argument passed to breakpoint <file>:<line_number>
fn parse_file_and_lineno(string: &str) -> Result<(&str, u64), &'static str> {
let vec: Vec<&str> = string.split(':').collect();
if vec.len() != 2 {
return Err("Failed to parse, please supply in format of file:lineno");
}
if let Ok(lineno) = vec.index(1).parse::<u64>() {
return Ok((vec[0], lineno));
} else {
return Err("Failed to parse a line number from supplied argument!");
}
}
// dbg.rs TraceeDbg method, prompt_user_cmd
"b" | "breakpoint" => {
if let Some(ref symref) = self.symbols {
let res = args_iter
.next()
.as_deref()
.ok_or("Insufficient arguments for command!")
.and_then(|arg| parse_file_and_lineno(arg))
.and_then(|(fname, lno)| {
src_line_to_addr(symref.borrow(), fname, lno)
.map_err(|_| "Failed to resolve address!")
});
match res {
Ok(addr) => Ok(Box::new(Breakpoint(addr))),
Err(msg) => Err(msg),
}
} else {
Err("Cannot resolve source lines without debug symbols!")
}
}In addition, extra state was needed to keep track of breakpoints. For every time the process stops, we need to check to see if we have landed on the program counter address (value of %rip) that was equivalent to a breakpoint address. I accomplished this through a Rust standard library hash map, std::collections::HashMap, that contained program counter keys and breakpoint record values.
#[derive(Debug)]
pub struct TraceeDbg<'dwarf> {
program: Option<String>,
breakpoints: RefCell<HashMap<u64, BrkptRecord>>,
symbols: Option<RefCell<Dwarf<borrow::Cow<'dwarf, [u8]>>>>,
position_ind_p: bool,
}
// ...in debugger loop
'await_process: loop {
let wait_status = waitpid(target_pid, None);
'await_user: loop {
match wait_status {
Ok(WaitStatus::Stopped(_, Signal::SIGTRAP))
| Ok(WaitStatus::Stopped(_, Signal::SIGSTOP)) => {
// Everytime we stop, we need to check if we stopped
// in place due to breakpoint, search for active breakpoint
// in hashmap with associated program counter, recover from
// trap, i.e. write back
let regs = ptrace::getregs(target_pid)
.expect("FATAL: failed to send PTRACE_REGS");
// attempt to get breakpoint with %rip value key, if one exists
// run the closure in map(), which recovers
// from the breakpoint.
self.breakpoints
.borrow()
.get(regs.rip)
.map(|brkpt| brkpt.recover_from_trap());
match self.prompt_user_cmd()
.and_then(|cmd| cmd.execute(target_pid, )) //...
In addition, the builder we talked about must incorporate all of this new state, from DWARF references to PIC flags in main.
fn main() {
println!("TRACEEDB DEBUGGER\nType \"help\" for command list!");
let mut args = env::args().skip(1).take(2);
let elf_buf: Vec<u8>;
let mut builder = TraceeDbg::builder();
if let Some(prog) = args.next() {
elf_buf = fs::read(prog.as_str())
.expect("Given program not found, exiting");
let file = object::File::parse(&*elf_buf)
.expect("Failed to parse program as ELF, exiting");
let is_et_dyn = match file.kind() {
ObjectKind::Executable => false,
ObjectKind::Dynamic => true,
_ => panic!(
"Please provide an ELF executable of type ET_DYN or ET_EXEC!"
),
};
builder = builder.program(prog)
.is_position_independent(is_et_dyn)
.dwarf_symbols(&elf_buf.as_slice());
}
builder.build().run();
}Testing the Breakpoint Command
Here I tested against both ELF types, using the C code below compiled with Clang. ET_EXEC uses -no-pie flag, while the ET_DYN case uses compiler default. Note how the %rip register lands on my desired address, translated from DWARF given the line and file number. Compile with DWARF debug information (ex. clang -g3 test.c -o test) to actually use breakpoints. Error message prints out in the case where no debug information was found.
#include <unistd.h>
#include <stdio.h>
int main(void) {
signed long a = 1;
signed long b = 2;
signed long c = 3;
signed long d = a + b + c;
printf("sum is %ld\n", d);
return 0;
}
# The ET_DYN case
Running `target/debug/traceedb test`
TRACEEDB DEBUGGER
Type "help" for command list!
Spawned child process 151527
Entering debugging loop...
Running traceable target program "test"
> breakpoint test.c:11
Breakpoint added, activating: 0x555555555141
> c
Recovering from trap!
> reg
%RIP: 0x555555555141
%RAX: 0x555555555140
%RBX: 0x0
%RCX: 0x555555557df0
%RDX: 0x7fffffffd7a8
%RBP: 0x1
%RSP: 0x7fffffffd680
%RSI: 0x7fffffffd798
%RDI: 0x1
# The ET_EXEC case
Running `target/debug/traceedb test`
TRACEEDB DEBUGGER
Type "help" for command list!
Spawned child process 152801
Entering debugging loop...
Running traceable target program "test"
> b test.c:11
Breakpoint added, activating: 0x401131
> c
Recovering from trap!
> reg
%RIP: 0x401131
%RAX: 0x401130
%RBX: 0x0
%RCX: 0x403e18
%RDX: 0x7fffffffd7a8
%RBP: 0x1
%RSP: 0x7fffffffd680
%RSI: 0x7fffffffd798
%RDI: 0x1
The Results
While this breakpoint command does work, there a few things I failed to do. They are single-use breakpoints that do not recover after they have been triggered. So if you place one in a loop, well, you are only triggering it once rather than multiple times. Other breakpoint implementations in popular debuggers default to laying the trap back down, keeping the breakpoint “activated.” I find that this is a great big juggling game to keep the breakpoint active while being sure to execute the instruction it replaces.
There are probably a few other edge cases I did not handle. Some with DWARF, some with using breakpoints in combination with other commands. Readers are free to pick this apart.
The code overall has very poor data flow. I was forever hesitant to add global variables to the code, instead electing to bury them in structs for debugger. There are a lot of fields passed into functions that probably are best fed there implicitly through other means. There is also a need to unify error handling and reporting, create test suites using native Rust test framework.
Overall, you can see just how much thought goes into a debugger command most take for granted. Final code for this project is viewable here at https://github.com/acolite-d/tracee-db . People are free to do with it what they will.
Leave a Reply