strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.

The operation of strace is made possible by the kernel feature known as ptrace.

Some of the features

Attach to an already running process.

$ strace -p 26380
strace: Process 26380 attached
...

Print paths and more info associated with file descriptors.

$ strace -yy cat /dev/null
...
openat(AT_FDCWD, "/dev/null", O_RDONLY) = 3</dev/null<char 1:3>>
fstat(3</dev/null<char 1:3>>, {st_mode=S_IFCHR|0666, st_rdev=makedev(0x1, 0x3),
...}) = 0
fadvise64(3</dev/null<char 1:3>>, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3</dev/null<char 1:3>>, "", 131072) = 0
...

Filter by type of syscall.

-e trace=%clock    Trace all system calls that read or modify system clocks.
         %creds    Trace all system calls that read or modify user and group identifiers or capability sets.
         %desc     Trace all file descriptor related system calls.
         %file     Trace all system calls which take a file name as an argument.
         %fstat    Trace fstat and fstatat syscall variants.
         %fstatfs  Trace fstatfs, fstatfs64, fstatvfs, osf_fstatfs, and osf_fstatfs64 system calls.
         %ipc      Trace all IPC related system calls.
         %lstat    Trace lstat syscall variants.
         %memory   Trace all memory mapping related system calls.
         %network  Trace all the network related system calls.
         %process  Trace all system calls which involve process management.
         %pure     Trace syscalls that always succeed and have no arguments.
         %signal   Trace all signal related system calls.
         %stat     Trace stat syscall variants.
         %statfs   Trace statfs, statfs64, statvfs, osf_statfs, and osf_statfs64 system calls.
         %%stat    Trace syscalls used for requesting file status.
         %%statfs  Trace syscalls related to file system statistics.

Trace only system calls accessing given path.

$ strace -P /etc/ld.so.cache ls /var/empty
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=22446, ...}) = 0
mmap(NULL, 22446, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b7ac2ba9000
close(3) = 0
+++ exited with 0 +++

Perform a full hexadecimal and ASCII dump of all the data read from/written to file descriptors.

$ strace -ewrite=1 -e trace=sendmsg ./recvmsg > /dev/null
sendmsg(1, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="012", iov_len=3},
{iov_base="34567", iov_len=5}, {iov_base="89abcde", iov_len=7}], msg_iovlen=3,
msg_controllen=0, msg_flags=0}, 0) = 15
 * 3 bytes in buffer 0
 | 00000  30 31 32                                          012              |
 * 5 bytes in buffer 1
 | 00000  33 34 35 36 37                                    34567            |
 * 7 bytes in buffer 2
 | 00000  38 39 61 62 63 64 65                              89abcde          |
+++ exited with 0 +++

Perform a syscall fault injection.

$ strace -e trace=open -e fault=open cat
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOSYS (Function not implemented) (INJECTED)
open("/lib/x86_64-linux-gnu/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOSYS (Function not implemented) (INJECTED)
open("/lib/x86_64-linux-gnu/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOSYS (Function not implemented) (INJECTED)
open("/lib/x86_64-linux-gnu/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOSYS (Function not implemented) (INJECTED)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOSYS (Function not implemented) (INJECTED)
cat: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 38
+++ exited with 127 +++

Count time, calls, and errors for each system call.

$ strace -c ls > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 89.76    0.008016           4      1912           getdents
  8.71    0.000778           0     11778           lstat
  0.81    0.000072           0      8894           write
  0.60    0.000054           0       943           open
  0.11    0.000010           0       942           close
  0.00    0.000000           0         1           read
  0.00    0.000000           0       944           fstat
  0.00    0.000000           0         8           mmap
  0.00    0.000000           0         4           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         7           brk
  0.00    0.000000           0         3         3 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           sysinfo
  0.00    0.000000           0         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.008930                 25440         3 total