Phillip Tennen

axle’s red rectangle of doom

Reading time: 4 minutes


Over the course of developing an operating system, things are going to crash, and they’re going to crash a lot.

One way to make crashes slightly less annoying is to isolate them: the rest of the system continues running, and you can poke around further after an unexpected condition arises in one process. Perhaps surprisingly, it took me several years before this became feasible!

Previously, when any code path, in any process, encountered an error condition (such as a page fault, or an explicit assert()), the kernel would spew some debug information over the serial port, then lock up to prevent further shenanigans.

This is pretty overzealous and often unnecessary. Even for exceptions triggered in kernel code, it’s often perfectly acceptable for just the responsible process to be terminated, and to allow the rest of the system to continue on its way.

In late 2021, I added a crash reporter application that reports failures in most processes. The mechanism works for both CPU-level faults (which trap to kernel-space✱) and voluntary aborts (which technically trap too, since they happen via a syscall).

✱ Note
More correctly, faults fault to kernel space. Faults return execution to the faulting instruction once the kernel’s fault handler completes, whereas traps proceed from the instruction following whichever one caused the trap. The former is useful, for example, if the kernel page faults because some memory is backed by lazily loaded data. After loading the data, the instruction that tried to access the memory should try running again. The int instruction, which is the mechanism used by syscalls, triggers a software-initiated fault, and continue execution after the int instruction (i.e. after the syscall returns).

Not all failures can be routed to the crash reporter, though. Three services are vital to the crash reporting flow. If any of them crash, the system must lock up.

  • com.axle.crash_reporter
    • If the crash reporter itself crashes, we definitely don’t have recourse for displaying a crash report.
  • com.axle.awm (axle window manager)
    • We need the window manager to display the crash reporter’s window.
  • com.axle.file_server
    • The file server is required to launch the crash reporter.

If one of these critical services dies, you get the axle red rectangle of doom (patent pending).

Crash reporting

Each kernel code path that handles an implicit task failure, such as the page fault handler, ends in a call to task_assert(), which includes a user-facing description of the error.

void _handle_page_fault(const register_state_t* regs)
kernel/kernel/vmm/vmm.c.x86_64.arch_specific 

char desc[512];
snprintf(desc, sizeof(desc), "Page fault: %s at 0x%p", reason, faulting_address);
task_assert(false, desc, regs);

The syscall for a voluntary abort, called by libc::assert(), is just another wrapper over task_assert().

kernel/kernel/syscall/sysfuncs.c 

static void task_assert_wrapper(register_state_x86_64_t* regs, const char* msg) {
  task_assert(false, msg, regs);
}

The machinery will then decide whether the crashing process is important enough to warrant the fuss of a full lockup, or if we can show the interactive crash reporter instead.

bool _can_send_crash_report(void)
kernel/kernel/assert.c 

amc_service_t* s = amc_service_of_active_task();
if (!strncmp(s->name, FILE_SERVER_SERVICE_NAME, AMC_MAX_SERVICE_NAME_LEN) ||
    !strncmp(s->name, CRASH_REPORTER_SERVICE_NAME, AMC_MAX_SERVICE_NAME_LEN) ||
    !strncmp(s->name, AWM_SERVICE_NAME, AMC_MAX_SERVICE_NAME_LEN)) {
    printf("Cannot generate crash report because the died process is critical to crash-reporting: %s\n", s->name);
    return false;
}
return true;

If we’re able to use the interactive crash reporter, we first ask the file server to launch it if it’s not already running. We then build up a crash report consisting of the registers at the time the fault was raised✱.

✱ Note
This is one neat benefit from the fact that this code path is shared by voluntary syscalls to assert(). Since syscalls are built upon traps (another type of interrupt), the interrupt handling mechanism will capture register state as though this code was invoked through a fault handler.

void task_build_and_send_crash_report_then_exit(const char*, const register_state_t*)
kernel/kernel/assert.c 

// Launch the crash reporter if it's not active
if (!amc_service_is_active(CRASH_REPORTER_SERVICE_NAME)) {
    file_server_launch_program_t req = {0};
    req.event = FILE_SERVER_LAUNCH_PROGRAM;
    snprintf(req.path, sizeof(req.path), "/usr/applications/crash_reporter");
    amc_message_send(FILE_SERVER_SERVICE_NAME, &req, sizeof(file_server_launch_program_t));
}

void task_build_and_send_crash_report_then_exit(const char*, const register_state_t*)
kernel/kernel/assert.c 

if (!append(
    &crash_report_ptr, 
    &buf_size, 
    "\nRegisters:\n")) goto finish_fmt;
if (!append(
    &crash_report_ptr, 
    &buf_size, 
    "rip 0x%p  rsp 0x%p\n", 
    regs->return_rip, 
    regs->return_rsp)) goto finish_fmt;
if (!append(
    &crash_report_ptr, 
    &buf_size, 
    "rax 0x%p  rbx 0x%p  rcx 0x%p rdx 0x%p\n", 
    regs->rax, 
    regs->rbx, 
    regs->rcx, 
    regs->rdx)) goto finish_fmt;
// ...

The crash reporter needs the dead process to host an amc service to do its work, so if the process isn’t hosting one at the time of death, the crash reporter will anoint it with an auto-generated service name like com.axle.corpse_service_PID_{id}_time_{ms}, as a sort of quiet eulogy.

bool _can_send_crash_report(void)
kernel/kernel/assert.c 

if (!amc_service_of_active_task()) {
    // Register an AMC service so we can send a crash report
    char buf[AMC_MAX_SERVICE_NAME_LEN];
    snprintf(buf, sizeof(buf), "com.axle.corpse_service_PID_%d_time_%d", getpid(), ms_since_boot());
    amc_register_service(buf);
}

We then symbolicate the return addresses we see on the stack. Each time the kernel launches an ELF, the kernel stores its symbol table. The kernel also stores its own symbol table, so that we can symbolicate both kernel and user-binary symbols when reporting crashes.

bool symbolicate_and_append(int, uintptr_t*, char**, int32_t*)
kernel/kernel/assert.c 

// Is the frame mapped within the kernel address space?
if (frame_addr >= VAS_KERNEL_CODE_BASE) {
    const char* kernel_symbol = elf_sym_lookup(
        &boot_info_get()->kernel_elf_symbol_table, 
        (uintptr_t)frame_addr
    );
    snprintf(symbol, sizeof(symbol), "[Kernel] %s", kernel_symbol ?: "-");
    // Note: Root function of exec() in core amc commands
    // Note: Root function of fs_server launch in the kernel
    if (kernel_symbol != NULL && 
        (!strncmp(kernel_symbol, AMC_EXEC_TRAMPOLINE_NAME_STR, 32) || 
         !strncmp(kernel_symbol, FS_SERVER_EXEC_TRAMPOLINE_NAME_STR, 32))) {
        found_program_start = true;
    }
}
else {
    task_small_t* current_task = tasking_get_current_task();
    const char* program_symbol = elf_sym_lookup(
        &current_task->elf_symbol_table, 
        (uintptr_t)frame_addr
    );
    snprintf(symbol, sizeof(symbol), "[%s] %s", current_task->name, program_symbol);
    if (!strncmp(program_symbol, "_start", 7)) {
        found_program_start = true;
    }
}

Completing the kernel’s work, the crash report is sent to the crash reporter application.

void task_build_and_send_crash_report_then_exit(const char*, const register_state_t*)
kernel/kernel/assert.c 

crash_reporter_inform_assert_t* inform = kmalloc(crash_report_msg_len);
inform->event = CRASH_REPORTER_INFORM_ASSERT;
inform->crash_report_length = crash_report_len;
memcpy(&inform->crash_report, crash_report_buf, crash_report_len);
amc_message_send(CRASH_REPORTER_SERVICE_NAME, inform, crash_report_msg_len);

The crash reporter itself is pretty straightforward: it listens for crash reports, and renders them to a text view (in green text, since we’re 😎hacking😎). Like all services, it defines structured message types that the kernel needs to conform to.

userspace/crash_reporter/crash_reporter_messages.h 

#define CRASH_REPORTER_INFORM_ASSERT 100
typedef struct crash_reporter_inform_assert {
    uint32_t event; // CRASH_REPORTER_INFORM_ASSERT
    uint32_t crash_report_length;
    char crash_report[];
} crash_reporter_inform_assert_t;

void _amc_message_received(amc_message_t*)
userspace/crash_reporter/crash_reporter.c 

char buf[512];
snprintf(buf, sizeof(buf), "Crash report for %s:\n", msg->source);
crash_reporter_inform_assert_t* assert_event = (crash_reporter_inform_assert_t*)&msg->body;
gui_text_view_puts(
    _g_text_view, 
    buf, 
    color_white()
);
gui_text_view_nputs(
    _g_text_view, 
    assert_event->crash_report_length, 
    assert_event->crash_report, 
    color_green()
);
gui_text_view_puts(_g_text_view, "\n", color_white());

Newsletter

Put your email in this funny little box, and I'll send you a message when I post new stuff.