In the most cases writing extensive logs and running profilers is sufficient for debugging. But in the most complex cases it may be useful to do interactive debugging with GDB. Additionally, GDB is the only option for debugging coredumps. 🐙 userver provides the capability to perform such debugging.
First of all, you can use GDB on your service based on userver just like any other binary (debug symbols are included by default in all build types).
Userver-specific debug features
To use userver-specific debug features, you need to allow execution of debug scripts, linked into your binary. It can be done by adding the following line to your ~/.gdbinit file
add-auto-load-safe-path <path-to-your-binary>
Alternatively, if you trust all the files you are debugging:
The simplest extentions for GDB, that userver provides, are pretty-printers for certain data stuctures. Below is an example comparing the output for a formats::json::Value with and without pretty-printers:
In addition, the output has a hierarchical structure that is displayed correctly when debugging from the IDE.
Coroutines exploration
🐙 userver provides GDB command utask, which mimics thread command and allows you to explore all coroutines (userver tasks), including running and suspended ones, in a manner similar to threads.
Commands
utask list: Lists all tasks with their names (corresponding span names) and statuses. Example:
(gdb) utask list
Task State Span
0x10f27fc40800 Suspended task_3
0x10f27fc3f000 Suspended task_2
0x10f27fc3d800 Suspended task_1
0x10f27fc3c000 Suspended task_0
0x10f27fc38000 Suspended span
0x10f27fc42000 Running task_4
utask apply <task> <cmd...>: Executes <cmd...> in the context of selected <task>. The <task> may be specified by its ID ("Task") or name ("Span") (as shown in utask list), or set to "all" to apply the command to all tasks. <cmd...> can be any GDB command, including Python scripts.
Examples:
Print "Hello world!" for all tasks
(gdb) utask apply all print "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc40800
$1 = "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc3f000
$2 = "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc3d800
$3 = "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc3c000
$4 = "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc38000
$5 = "Hello world!"
Executing command `print "Hello world!"` for task 0x10f27fc42000
$6 = "Hello world!"
Get backtrace of the suspended task_1
(gdb) utask apply task_1 backtrace
Executing command `backtrace` for task 0x10f27fc3d800
#1 0x00000000006131d8 in boost::context::fiber::resume() && (this=0x7fffefae6e38)
at /usr/include/boost/context/include/boost/context/fiber_fcontext.hpp:377
Executing command `python print('Hello from python!', 'current frame:', gdb.selected_frame().function())` for task 0x10f27fc3d800
Hello from python! current frame: boost::context::fiber::resume() &&
For now utask commands are implemented for only linux x86 platforms, but can be easily extended for other platforms.
In addition, all of the above functionality works for debugging both a live process and coredumps.
GDB complains: received signal ?, Unknown signal
This is a side effect of stack usage monitor interferring with GDB. In unit tests you can set the environment variable USERVER_GTEST_ENABLE_STACK_USAGE_MONITOR=0 to disable the monitor, in other binaries you can either disable it via static config option coro_pool.stack_usage_monitor_enabled in components::ManagerControllerComponent or by disabling it at all at build time of the framework via USERVER_FEATURE_STACK_USAGE_MONITOR (see Build options).
Adding new pretty-printers and commands
If you need a new pretty-printer or a GDB command, you can always implement it yourself in userver/scripts/gdb and bring us a PR!
