KasperskyOS Community Edition 1.3

Analyzing stack backtrace data when a process abnormally terminates

When a process is terminated due to an unhandled exception, the KasperskyOS kernel prints stack backtrace data in the following format:

Call Trace (User Mode): [<0000000004068498>] (__assert_func+0x78) from [<000000000405f478>] (_vfs_backend_create+0xe8) [<000000000405f478>] (_vfs_backend_create+0xe8) from [<00000000040bdab0>] (_vfs_init+0xd0) [<00000000040bdab0>] (_vfs_init+0xd0) from [<00000000040111a8>] (init_vfs_helper+0xc8) [<00000000040111a8>] (init_vfs_helper+0xc8) from [<00000000040117e0>] (__eprol+0x88) [<00000000040117e0>] (__eprol+0x88)

Here you can clearly see the sequence of function calls that led to the unexpected termination of the process. At the end of the sequence, the _vfs_backend_create() function code residing in the process memory at the address 0x000000000405f478 called the __assert_func() function, and the __assert_func() function code residing in the process memory at the address 0x0000000004068498 invoked the exception. In other words, the exception occurred during verification of the conditions in the _vfs_backend_create() function. (An entry in the format <function name>+0x<offset> indicates an address within the function code. For example, the entry _vfs_backend_create+0xe8 indicates an address within the _vfs_backend_create() function code offset by 0xe8 bytes relative to the address of the start of this function's code. The entry _vfs_backend_create+0xe8 corresponds to the address 0x000000000405f478.)

The names of all or some functions in the sequence of calls may fail to appear, which makes it more difficult to determine the cause of the abnormal process termination.

Example stack backtrace data containing only memory addresses:

Call Trace (User Mode): [<0000000004068498>] from [<000000000405f478>] [<000000000405f478>] from [<00000000040bdab0>] [<00000000040bdab0>] from [<00000000040111a8>] [<00000000040111a8>] from [<00000000040117e0>] [<00000000040117e0>]

The reason for the missing names of functions in the stack backtrace data is as follows. To print stack backtrace data, the kernel gets the names of functions from the symbol table .symtab and the string table .strtab that are loaded into the memory of the abnormally terminated process from the debug version of the executable file. These tables may be incomplete or entirely missing from the memory of the abnormally terminated process. In addition, the kernel cannot print the names of functions that were imported from dynamic libraries.

The symbol table .symtab and string table .strtab are absent from process memory in the following cases:

  • The process was created from the normal release version of the executable file. (The executable file was built as the normal release version, and the normal release versions of static libraries were included in it during the build.)

    In contrast to the debug version, the normal release version of the executable file does not contain the symbol table .symtab and string table .strtab.

  • The process was created from the debug version of an executable file from which the symbol table .symtab and string table .strtab were deleted before the file was added to the solution image. For instance, they could have been deleted by the strip utility.
  • The process was created from the debug version of an executable file from which the symbol table .symtab and string table .strtab were not deleted before the file was added to the solution image, but these tables were not loaded into the process memory.

    The high-level API for process management and the low-level API for process management let you create a process with optional loading of the symbol table .symtab and string table .strtab.

The symbol table .symtab and string table .strtab are incomplete in the following cases:

  • If the executable file was built as a debug version, the normal release versions of static libraries were included in the file during the build, and the symbol table .symtab and string table .strtab will not contain information about the functions implemented in these libraries.
  • If the executable file was built as a normal release version, the debug versions of static libraries were included in the file during the build, and the symbol table .symtab and string table .strtab will contain information only about the functions that are implemented in these libraries.

For some memory addresses specified in stack backtrace data, information about functions can be obtained from the executable file. If a memory address is present in the symbol table .symtab of the executable file, you can get the code line number and the name of the function corresponding to this memory address. If a memory address is missing from the symbol table .symtab but is present in the symbol table .dynsym of the executable file, you can get the name of the function corresponding to this memory address. (When using ASLR, the memory addresses specified in the stack backtrace data are increased by the Relocation base value for the ELF image load offset that is printed together with the stack backtrace data.) Otherwise, information about functions cannot be received from the executable file. For instance, this applies to functions imported from dynamic libraries and to functions implemented in normal release versions of static libraries.

An executable file in KasperskyOS contains a set of exportable functions (but is not a dynamic library). For example, the functions of the libc library included in executable files are exportable. The symbol table .dynsym contains information about exportable functions. This table is present in the release version of the executable file, and remains in the debug version of the executable file after the debug symbols are deleted from it. (The symbol table .symtab and string table .strtab are part of the debug symbols.)

To get information about functions corresponding to the memory addresses specified in backtrace data from an executable file, use the addr2line utility provided in KasperskyOS Community Edition (the toolchain/bin/aarch64-kos-addr2line executable file). Format of the command for starting the addr2line utility:

aarch64-kos-addr2line [--functions] --exe=<path to the executable file> 0x<address 1> [0x<address 2>]...

In this command, you need to specify the memory addresses from the stack backtrace data reduced by the Relocation base value. The --functions parameter requires output of the names of functions in addition to the numbers of source code lines.

Example:

$ aarch64-kos-addr2line --functions --exe=./build/hello/Hello 0x30090 main /opt/KasperskyOS-Community-Edition-<version>/examples/hello/hello/src/hello.c:6