Core Dump and GDB

Introduction

GDB can be used for debugging computer program bugs, such as segmentation faults. The key file used for debugging is the core dump file, which is generated when a program crashes. The core dump file contains a snapshot of the program’s memory at the time of the crash, allowing developers to analyze the state of the program and identify the cause of the crash.

In this blog post, I would like to discuss how to generate a core dump file from a Docker container and how to use GDB to debug quickly.

Core Dump and GDB

Core Dump File Size Limit

Make sure the core dump file size limit is set to unlimited. It can be checked using the ulimit command in the terminal.

$ ulimit -c
unlimited

It can be set to unlimited, if it is not, using the ulimit command as well.

$ ulimit -c unlimited

Core Dump File Name Pattern

To save a core dump file in a Docker container, we have to set the core_pattern, i.e., the file name pattern for the core dump files. This has to be set from the host computer before Docker container is launched. We could set the core_pattern to /tmp/core.%e.%p, which will save the core dump files in the /tmp directory with the format core.<executable_name>.<pid>.

# The /proc/sys/kernel/core_pattern will be reset after reboot.
$ echo '/tmp/core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern

The /proc/sys/kernel/core_pattern will be reset to default after rebooting the host computer.

Launch Docker Container

We could use the Docker container prepared for debugging from the previous article “Debug C/C++ Programs In Docker Container”.

$ docker run -it --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $(pwd):/mnt -w /mnt cpp-debug:0.0.1

Run Computer Programs

The problematic C++ program from the previous article “Illegal Memory Access and Segmentation Fault” was used to demonstrate how to generate a core dump file.

In this case, we build the C++ program with and without debugging symbols. When we run the computer programs, the program will crash with segmentation faults and generate a core dump file in the /tmp directory following the core_pattern we set earlier.

$ g++ oob.cpp -o oob
$ g++ -g oob.cpp -o oob_debug
$ ls /tmp/
tmpk1oekgd5
$ ./oob
Vector Size: 32
Accessing Out-Of-Bounds Element At Index: 32
Accessing Out-Of-Bounds Element At Index: 41
Accessing Out-Of-Bounds Element At Index: 131
Accessing Out-Of-Bounds Element At Index: 1031
Accessing Out-Of-Bounds Element At Index: 10031
Accessing Out-Of-Bounds Element At Index: 100031
Segmentation fault (core dumped)
$ ls /tmp/
core.oob.22  tmpk1oekgd5
$ ./oob_debug
Vector Size: 32
Accessing Out-Of-Bounds Element At Index: 32
Accessing Out-Of-Bounds Element At Index: 41
Accessing Out-Of-Bounds Element At Index: 131
Accessing Out-Of-Bounds Element At Index: 1031
Accessing Out-Of-Bounds Element At Index: 10031
Accessing Out-Of-Bounds Element At Index: 100031
Segmentation fault (core dumped)
$ ls /tmp/
core.oob.22  core.oob_debug.24  tmpk1oekgd5

Examine Core Dump File Using GDB

GDB can be used to examine the core dump files generated by computer programs built with or without debugging symbols.

$ gdb oob /tmp/core.oob.22
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at---Type <return> to continue, or q <return> to quit---
:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from oob...(no debugging symbols found)...done.
[New LWP 22]
Core was generated by `./oob'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005f3388800bd4 in access_out_of_bounds_element(std::vector<int, std::allocator<int> > const&, unsigned long) ()
(gdb) q

$ gdb oob_debug /tmp/core.oob_debug.24
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at---Type <return> to continue, or q <return> to quit---
:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from oob_debug...done.
[New LWP 24]
Core was generated by `./oob_debug'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000594f9ca00bd4 in access_out_of_bounds_element (
    v=..., oob_offset=100000) at oob.cpp:10
10          int out_of_bounds_element = v[v.size() - 1 + oob_offset];
(gdb) q

In both cases, GDB informs the user that the program terminated at the function call of access_out_of_bounds_element. In practice, there can be multiple calls of the function from the program, and we would like to know which exact call caused the segmentation fault. The core dump file from the program built with debugging symbols provides such additional information. Therefore, to debug more effectively, it is recommended to build the program with debugging symbols for debugging.

GDB Is Necessary

Since GDB is necessary to examine the core dump file and the executable is required for tracing, sometimes it is not quite necessary to save the core dump file. GDB can be used to run the computer program and the dump file will be recorded by GDB automatically.

Conclusions

The core dump can hardly be used alone without GDB.

References

• GDB - Core Dumps
• Debug C/C++ Programs In Docker Container
• Stack Trace From Core File Without Executable