Comprehensive system analysis by SystemTap

As system administrators, we’d better know SystemTap and be able to assemble its existed recipes for comprehensive system analysis. When I am on my way to make a collection of  systemtap scripts, I realize that I should give a brief introduction to this powerful design first, although it has been usable for years. From its official website,

SystemTap provides free software infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running system.Current project members include Red Hat, IBM, Intel, Hitachi, and Oracle.

To use a full-featured SystemTap, you need the debugging information (“dwarf”) that accompanies your kernel and modules. To probe userspace applications, UTRACE needs to be supported by the running kernel (RHEL has it, but Debian).  Without dwarf, you can still do quite a bit by enabling probing of entry & exit points of kernel & module functions. This is achieved by deferring symbol resolution to runtime & utilizing kprobes to do the same.

What works without Dwarf?

The stap command supports a new probe family, called “kprobes.function” for dwarfless tracing. Following constructs are supported:

  • kprobe.function(FUNCTION)
  • kprobe.function(FUNCTION).return
  • kprobe.module(NAME).function(FUNCTION)
  • kprobe.module(NAME).function(FUNCTION).return
  • kprobe.statement.(ADDRESS).absolute

What Doesn’t Work without Dwarf?

Without debugging information, SystemTap can’t support the following types of language constructs:

  • probe local variables of a function.
  • probe specifications that refer to source files or line numbers.
  • probe specifications that refer to inline functions
  • statements that refer to $target variables
  • tapset-defined variables defined using any of the above constructs.

How to install debugging info for your running kernel?

For Debian, you can just run this command: apt-get install systemtap linux-image-`uname -r`-dbg linux-headers-`uname -r`

On a RHEL 5 system that has a connection to RHN, SystemTap can be installed with the following commands:

yum install systemtap kernel-devel
yum --enablerepo=rhel-debuginfo install kernel-debuginfo

A running Example

We use this script to print the top 10 syscalls called in last 5 seconds:

#!/usr/bin/env stap
# display the top 10 syscalls called in last 5 seconds
global syscalls
function print_top () {
        log ("SYSCALL\t\t\t\tCOUNT")
        foreach ([name] in syscalls-) {
                printf("%-20s\t\t%5d\n",name, syscalls[name])
                if (cnt++ == 10)
        delete syscalls
probe syscall.* {
probe {
        print_top ()

And, here is the result,

root@deb:~# ./system-calls.stp  -v
Pass 1: parsed user script and 68 library script(s) using 49976virt/20908res/1768shr kb, in 90usr/10sys/110real ms.
Pass 2: analyzed script: 395 probe(s), 6 function(s), 20 embed(s), 1 global(s) using 178556virt/125588res/62820shr kb, in 1060usr/50sys/1120real ms.
Pass 3: translated to C into "/tmp/stapzEtPWP/stap_af6ac7a61984ec48e2424ae5b3d40bbc_150444.c" using 178556virt/125960res/63168shr kb, in 420usr/30sys/450real ms.
Pass 4: compiled C into "stap_af6ac7a61984ec48e2424ae5b3d40bbc_150444.ko" in 2020usr/2450sys/4719real ms.
Pass 5: starting run.
sys_futex           		  227
sys_times           		  108
sys_read            		   79
sys_poll            		   57
sys_ppoll           		   50
sys_ioctl           		   37
sys_close           		   35
sys_socket          		   24
sys_select          		   12
sys_newfstat        		   12
sys_open            		   11
sys_futex           		  214
sys_times           		  114
sys_poll            		   59
sys_read            		   53
sys_ppoll           		   50
sys_select          		   14
sys_rt_sigprocmask  		    4
sys_newstat         		    3
sys_write           		    2
sys_socket          		    1
sys_ioctl           		    1
sys_futex           		  224
sys_times           		  114
sys_read            		   61
sys_poll            		   58
sys_ppoll           		   50

SystemTap has officially made a collection of SystemTap scripts, which can be found here. With the help of these scripts, you can dig further when issue comes. Have fun!


Share Button

Leave a comment

Your email address will not be published. Required fields are marked *