DDB(4) Device Drivers Manual DDB(4)

ddb — kernel debugger

The ddb debugger provides a means for debugging the kernel, and analysing the kernel after a system crash ("panic"), with a gdb(1)-like syntax.

ddb is invoked upon a kernel panic when the sysctl(8) ddb.panic is set to 1. It may be invoked from the console when the sysctl ddb.console is set to 1, using any of the following methods:

  • Using the key sequence Ctrl-Alt-Esc.
  • Sending a BREAK when using a serial console.
  • Writing to the sysctl ddb.trigger.
  • For i386 and amd64 architectures, using the key sequence Ctrl-Alt-Delete when the sysctl machdep.kbdreset is set to 2.

ddb prompts for commands on the console with:

ddb>

The general syntax of a ddb command is:

command [/modifiers] [address][,count]

To save typing, ddb makes use of a context inferred from previous commands. In this context, the current location is called dot. The examine, search, show struct, and write commands update dot to be that of the last address examined or the last location modified, and have intuitive effects on next and prev. All the other commands do not change dot, and set next to be the same. (See VARIABLES.)

An expression can be used in place of address (see EXPRESSIONS). Omitting address in a command uses the last value of dot. A missing count is taken to be 1 for printing commands or infinity for stack traces. Entering a blank line causes the last command to be repeated using next in place of address, a count of 1, and no modifiers.

ddb has a feature like more(1) for the output. If the number of lines output in response to one command exceeds the number set in the $lines variable, it displays the message ‘--db_more--’ and waits for a response.

The valid responses are:

⟨space⟩
One more page.
⟨return⟩
One more line.
Abort the current command, and return to the command input mode.

The following command line editing keys are provided:

back one character
forward one character
beginning of line
end of line
erase word back
| ⟨del⟩
erase previous character
erase next character
delete to end of line
delete line
previous in command history
next in command history
redraw line
exchange the two characters to the left of the cursor

The following commands may be typed at the ‘ddb>’ prompt. Some commands consist of more than one word, and if only the first word or words are entered, the possible alternatives to complete the command are displayed and no other action is performed.

List the available commands.
[e]x[amine] [/bhlqaAxzodurcsmiI] [addr][,count]
Display the contents at address addr according to the formats in the modifier. If no format is specified, the last formats specified for this command are used.

The format characters are:

look at by bytes (8 bits)
look at by half words (16 bits)
look at by long words (32 bits) (default)
look at by long longs (64 bits) (only available on 64-bit platforms)
print the location being displayed
print the location with a line number if possible
display in unsigned hex
display in signed hex
display in unsigned octal
display in signed decimal
display in unsigned decimal
display in current radix, signed
display low 8 bits as a character. Non-printing characters are displayed as an octal escape code (e.g., '\000').
display the null-terminated string at the location. Non-printing characters are displayed as octal escapes.
display in unsigned hex with character dump at the end of each line. The location is also displayed in hex at the beginning of each line.
display as an instruction
display as an alternate format instruction depending on the machine:

alpha
Print affected register contents for every instruction.
amd64,
 
i386
Do not skip padding to the next long word boundary for unconditional jumps.
m88k
Decode instructions for the opposite CPU model (e.g. m88110 when running on an m88100 processor).

The value of next is set to the addr plus the size of the data examined.

[rint] [/axzodurc] [addr]
Print addr according to the modifier character. The valid modifiers are a subset of those from the examine command, and act as described there. If no modifier is specified, the last one specified in a previous use of print is used.

For example,

print/x $eax

will print something like this:

xxxxxx
[rint] [addr]
Pretty-print addr using CTF debug symbols included in the kernel binary image. The CTF section is normally added by running ctfstrip(1) as part of building a new kernel.
[rite] [/bhl] [addr] expr [expr ...]
Write the value of each expr expression at succeeding locations start at addr. The write unit size can be specified using one of the modifiers:

byte (8 bits)
half word (16 bits)
long word (32 bits) (default)

The value of next is set to addr plus the size of values written.

Warning: since there is no delimiter between expressions, the command may not parse as you expect. It is best to enclose each expression in parentheses.

$name [=] expr
Set the named variable or register with the value of expr. For valid variable names, see VARIABLES.
how
Reboot the machine depending on how:

boot sync
Sync disks and reboot.
boot crash
Dump core and reboot.
boot dump
Sync disks, dump core and reboot.
boot halt
Just halt.
boot reboot
Just reboot.
boot poweroff
Power down the machine whenever possible; if it fails, just halt.
[addr][,count]
Set a break point at addr. If count is supplied, ddb allows the breakpoint to be silently hit (count - 1) times before stopping at the break point.
[elete] [addr]
Delete the break point set with the break command.
[tep] [/p] [,count]
Single step count times. If the /p modifier is specified, print each instruction at each step. Otherwise, only print the last instruction.

Warning: depending on machine type, it may not be possible to single-step through some low-level code paths. On machines with software-emulated single-stepping (e.g., alpha), stepping through code executed by interrupt handlers will probably do the wrong thing.

name(expr [, expr ...] )
Call the function named by name with the argument(s) listed in parentheses. Parentheses may be omitted if the function takes no arguments. The number of arguments is currently limited to 10.
[ontinue] [/c]
Continue execution until a breakpoint or watchpoint. If the /c modifier is given, instructions are counted while executing.

Warning: when counting with /c, ddb is really silently single-stepping. This means that single-stepping on low-level code may cause strange behavior.

addr [,size]
Set a watchpoint for the region starting at addr. Execution stops and control returns to ddb when an attempt is made to modify a watched region. The size argument defaults to 4.

If you specify a wrong space address, the request is rejected with an error message.

Warning: attempts to watch wired kernel memory may cause an unrecoverable error on some systems (e.g., i386).

addr
Delete the watchpoint at address addr that was previously set with a watch command.
[/s[0-9]]
This is a tiny and handy tool for random kernel hangs analysis, of which its depth is controlled by the optional argument of the default value of five. It uses some sophisticated heuristics to spot the global symbol that caused the hang. Since the discovering algorithm is a probabilistic one, you may spend substantial time to figure the exact symbol name. This smart thing requires a little of your attention, the input it accepts is mostly of the same format as that of the famous hangman(6) game, to which it, apparently, is obliged by the name. Hint: the nm(1) utility might help.
[/p]
Stop at the next "call" or "return" instruction. If the /p modifier is specified, ddb prints the call nesting depth and the cumulative instruction count at each call or return. Otherwise, it stays silent until the matching return is hit.
[/p]
Stop at the next matching return instruction. If the /p modifier is specified, ddb prints the call nesting depth and the cumulative instruction count at each call or return. Otherwise, it remains mostly quiet.
[/p]
The next command is a synonym for match.
pid
Send an uncatchable SIGABRT signal to the process specified by the pid argument.
[/tu] [frameaddr][,count]
Show the stack trace. The /t modifier interprets the frameaddr argument as the TID of a process and shows the stack trace of that process. frameaddr is subject to the radix; use the 0t prefix to enter a decimal TID. The /t modifier is not supported on all platforms. The /u modifier shows the stack trace of user space; if omitted, the kernel stack is traced instead. The count argument is the limit on the number of frames to be followed. If count is omitted, all frames are printed.

Warning: user space stack trace is valid only if the machine dependent code supports it.

Search memory for a value beginning at addr. This command might fail in interesting ways if it doesn't find the searched-for value. This is because ddb doesn't always recover from touching bad memory. The optional count argument limits the search. The modifiers are the same as those of the write command.

The next address is set to the address where value is found, or just after where the search area finishes.

Shortcut for boot reboot
what
Displays various things, depending on what:
show bcstats
Prints the buffer cache statistics.
show breaks
Prints a list of all breakpoints that have been set with the break command.
show buf addr
Prints the struct buf at addr.
show extents
Prints a detailed list of all extents.
show locks [addr]
Prints the list of locks held by a thread. If an optional address is not specified, curproc is assumed. The option WITNESS is required for this command to be available.
show malloc [addr]
Prints malloc debugging information if available. If an optional address is specified, only information about that address is printed.
show map [/f] addr
Prints the vm_map at addr. If the /f modifier is specified, the complete map is printed.
show mbuf [/cp] addr
Prints the struct mbuf header at addr. Depending on the mbuf flags struct pkthdr and struct m_ext are printed as well. If the /c modifier is specified, print the mbuf chain linked with the m_next pointer. /p does the same using m_nextpkt. Both can be combined.
show mount [/f] addr
Prints the struct mount at addr. If the /f modifier is specified, prints out all vnodes (see also show vnode) and also all bufs (see also show buf) on all those vnodes.
show nfsnode [/f] addr
Prints the struct nfsnode at addr. If the /f modifier is specified, prints out additional information as well.
show nfsreq [/f] addr
Prints the struct nfsreq at addr. If the /f modifier is specified, prints out additional information as well.
show object [/f] addr
Prints the vm_object at addr. If the /f modifier is specified, the complete object is printed.
show page [/f] addr
Prints the vm_page at addr. If the /f modifier is specified, the complete page is printed.
show panic
Prints the panic string.
show pool [/p] addr
Prints the pool at addr. Valid modifiers:
Print the pagelist for this pool.
show proc [/t] [addr]
Prints the struct proc at addr. If an optional address is not specified, curproc is assumed. The /t modifier interprets the addr argument as the TID of a proc.
show registers [/u]
Display the register set. If the /u modifier is specified, it displays user registers (or the currently saved registers) instead of the kernel's. Note: The /u modifier is not supported on every machine, in which case incorrect information may be displayed.
show route addr
Prints the struct rtentry at addr.
show socket addr
Prints the struct socket at addr. If the socket is spliced, the struct sosplice associated with the socket is printed as well.
show struct name [addr]
Prints the content of the memory at addr as a struct name. Nested structures and bit fields are not printed. Character arrays are printed as bytes.
show swap
Prints a detailed list of all swaps.
show tdb [/f] addr
Prints the struct tdb at addr. If the /f modifier is specified, prints out all fields of this IPsec SA.
show uvmexp
Displays a selection of uvm counters and statistics.
show vnode [/f] addr
Prints the struct vnode at addr. If the /f modifier is specified, prints out all bufs (see also show buf) currently attached to this vnode.
show watches
Displays all watchpoints set with the watch command.
show witness [/b]
Prints the current order list. If the /b modifier is specified, the list of found lock order violations is printed instead. The option WITNESS is required for this command to be available.
show all procs [/anow]
Display information on all processes.

(Default) Show process information in a ps(1)-like format. Information printed includes process ID, thread ID, parent process ID, UID, process status, process flags, process wait channel message and process command name.
Shows the kernel virtual addresses of each process' proc structure, u-area, and vmspace structure. The vmspace address is also the address of the process' vm_map structure and can be used in the show map command.
Shows non-idle threads that were on CPU when ddb was entered. Information printed includes thread ID, process ID, UID, process flags, thread flags, current CPU, and command name.
Shows each thread's ID, command, process group, wait channel address, and wait channel message.
show all bufs [/f]
Display information about all buffers in the system.

For each buffer, print a more detailed output. See the show buf command for more information.
show all callout
Display the contents of the callout table.
show all pools [/a]
Display information about all system pools in a format similar to vmstat(8).

Displays “interesting” address information.
show all locks
Prints the list of locks held by all threads in the system. The option WITNESS is required for this command to be available.
show all mounts [/f]
Display information on all mounted filesystems.

For each filesystem, list all its struct vnode addresses. These addresses can be used in the show vnode command.
show all nfsnodes [/f]
Display information about all nfsnodes in the system.

For each nfsnode, print a more detailed output. See the show nfsnode command for more information.
show all nfsreqs [/f]
Display information for all outstanding NFS requests.

For each NFS requests, print a more detailed output. See the show nfsreq command for more information.
show all routes [/iI] [rtableid] [,count]
Show internet routing tables. Default for rtableid is 0 and count is 1.

Restrict to AF_INET.
Restrict to AF_INET6.
show all tdbs [/f]
Display information about all IPsec SAs in the system.

For each tdb, print a more detailed output. See the show tdb command for more information.
show all vnodes [/f]
Display information about all vnodes in the system.

For each vnode, print a more detailed output. See the show vnode command for more information.
A synonym for the show all callout command.
[/anow]
A synonym for show all procs.
[hine] subcommand [arg ...]
Perform a platform-specific command.

The following commands are supported by multiprocessor kernels on these platforms: amd64, i386, macppc, mips64, and sparc64.

Display the state of each CPU.
N
Stop the current CPU and start handling ddb on the selected CPU.
[N]
Resume normal processing on the selected CPU, or all CPUs if none is specified.
[N]
Stop normal processing on the selected CPU, or all CPUs (except the one handling ddb) if none is specified.

Other platform-specific commands:

arm:

addr
Display the trapframe at addr.

i386:

Display the contents of the privileged registers: IDTR, GDTR, LDTR, TR, CR0, CR2, CR3, and CR4.

m88k:

N
Stop the current CPU and start handling ddb on the selected CPU.
addr
Display the trapframe at addr.
Display the registers from when ddb was entered.
[addr]
Search for and display stack exception frames, starting from addr if given, else the address in register r31, and stopping at the next 8k boundary.
Display where the current CPU was stopped.

mips64:

[/p asid] [/c] [tlb]
 
??
 

sh:

[addr]
Display the cache, starting from addr, defaulting to 0.
Display the switch and trap frames.
Display the TLB.

sparc64:

Display the context addresses for all threads.

ddb denotes registers and variables by $name. Register names can be found with the show registers command.

Built-in debugger variables currently supported are:

$radix
Input and output radix.
$maxoff
Addresses are printed as symbol+offset unless offset is greater than $maxoff.
$maxwidth
The width of the displayed lines.
$lines
The number of lines to page. This is used by the “more” feature.
$tabstops
Tab stop width.
$log
Controls whether the output of ddb will also appear in the system message buffer.

These variables can also be controlled outside ddb via the ‘ddb’ sysctl(8) hierarchy.

Almost all expression operators in C are supported except for ‘~’, ‘^’, and unary ‘&’. Special rules for expressions in ddb are:

identifier
The name of a symbol. It is translated to the address (or value) of the symbol. ‘.’ and ‘:’ can be used in the identifier. The following can be accepted as an identifier, if supported by an object format dependent routine:
  • [filename:]func[:linenumber]
  • [filename:]variable
  • filename[:linenumber]
The symbol may be prefixed with ‘symboltablename::’ (e.g., ‘emulator::mach_msg_trap’) to specify other than kernel symbols.
number
The radix is determined by the first two letters: ‘0x’: hex, ‘0o’: octal, ‘0t’: decimal, otherwise, the value of $radix is used.
dot: the current address.
next: the next address.
The address of the start of the last line examined. Unlike dot or next, this is only changed by the examine or write command.
The last address explicitly specified.
variable
The value of a register or variable.
expr # expr
A binary operator which rounds up the left hand side to the next multiple of right hand side.
expr
Indirection. It may be followed by a ':' as described above with identifier.

ctfstrip(1), gdb(1), nm(1), witness(4), sysctl.conf(5), hangman(6), crash(8), sysctl(8), extent(9), pool(9), uvm_init(9)

This kernel facility first appeared in the Mach 3.0 operating system developed by CMU. Hangman (which stands for "hangs maniacal analyzer") first appeared in OpenBSD 1.2.

OpenBSD 7.8 December 21, 2024 DDB(4)