DTrace_notes.txt

Notes for non-trivial things I had to dig for to find.

1. To print a global symbol value in a DTrace script, put a ` in front of it.

  For instance,

  dtrace -qn 'dnlc_lookup:return{@[(arg1)?"Hit":"Miss"]=count()} tick-10sec{printf("%Y ",walltimestamp);printa("%s: %@d ",@);printf("DNLC entries: %d\n",`dnlc_nentries);trunc(@)}'

One-liners

Detect zeroed blocks being written to disk (such as sdelete or dd if=/dev/zero from the client):

dtrace -qn 'zio_compress_data:return/args[1]==0/{@["zero blocks"]=count()} tick-1sec{printf("%Y",walltimestamp);printa(@);clear(@);}'

Find how much time a userland process spends on-cpu to determine how effective its threading model is (using rpcbind as example) - credit to Saso Kiselkov:

dtrace -n 'BEGIN{total=0}sched:::on-cpu/pid=='$(pgrep rpcbind)'/{self->start=walltimestamp} sched:::off-cpu/pid=='$(pgrep rpcbind)'/{total += walltimestamp - self->start} tick-1s{printf("total: %lli.%09lli", (long long) total/1000000000, (long long) total % 1000000000); total=0}'

Detect whether ixgbe is tripping on an unsupported SFP:

dtrace -n 'ixgbe_identify_sfp_module_generic:return/arg1==-19/{trace("IXGBE_ERR_SFP_NOT_SUPPORTED")}'

Find out who is unmounting ZFS snapshots (stolen from https://www.illumos.org/issues/5273):

dtrace -n 'fbt:zfs:zfsvfs_teardown:entry /strstr(stringof(((struct zfsvfs *)arg0)->z_vfs->vfs_mntpt->rs_string), "snapshot") != 0/ { printf("pid %d, execname %s, unmounting %d, %s, %Y", pid, execname != 0 ? execname : "NULL", arg1, stringof(((struct zfsvfs *)arg0)->z_vfs->vfs_mntpt->rs_string), walltimestamp ); stack(); ustack(); }'