The following are extended examples of
iosnoop, a Solaris 10 DTrace program to
snoop disk I/O activity live.
Default Output contains most of the useful information - PID, SIZE, COMM and PATHNAME.
# iosnoop UID PID D BLOCK SIZE COMM PATHNAME 100 15795 R 3808 8192 tar /usr/bin/eject 100 15795 R 35904 6144 tar /usr/bin/eject 100 15795 R 39828 6144 tar /usr/bin/env 100 15795 R 3872 8192 tar /usr/bin/expr 100 15795 R 21120 7168 tar /usr/bin/expr 100 15795 R 43680 6144 tar /usr/bin/false 100 15795 R 44176 6144 tar /usr/bin/fdetach 100 15795 R 3920 8192 tar /usr/bin/fdformat 100 15795 R 3936 8192 tar /usr/bin/fdformat 100 15795 R 4080 8192 tar /usr/bin/fdformat 100 15795 R 9680 3072 tar /usr/bin/fdformat 100 15795 R 4096 8192 tar /usr/bin/fgrep 100 15795 R 46896 6144 tar /usr/bin/fgrep 100 15795 R 4112 8192 tar /usr/bin/file [...] |
Usage the following is the usage message from ver 0.95.
# iosnoop -h USAGE: iosnoop [-a|-A|-Degstv] [-d device] [-f filename] [-m mount_point] iosnoop # default output -a # print all data -A # dump all data, space delimited -D # include time delta, us -e # include device name -g # include command arguments -s # include start time, us -t # include completion time, us -v # include completion time, string -d device # instance name to snoop -f filename # snoop this file only -m mount_point # this FS only |
Timestamp Output using the -v switch displays timestamps in a human readable format.
# iosnoop -v STRTIME UID PID D BLOCK SIZE COMM PATHNAME 2005 Jan 6 19:33:31 0 7471 R 57136 8192 bash /usr/sbin/df 2005 Jan 6 19:33:31 0 7471 R 57152 8192 bash /usr/sbin/df 2005 Jan 6 19:33:40 0 7472 R 23544 4096 uptime /usr/bin/sparcv9/uptime 2005 Jan 6 19:33:40 0 7472 R 4618080 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:40 0 7472 R 4618144 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:40 0 7472 R 4618160 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:40 0 7472 R 4618096 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:41 0 7472 R 4617456 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:41 0 7472 R 4618624 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:41 0 7472 R 4617408 8192 uptime /lib/sparcv9/libc.so.1 2005 Jan 6 19:33:49 0 7473 R 23544 4096 uptime /usr/bin/sparcv9/uptime 2005 Jan 6 19:33:53 0 3 W 8305 2560 fsflush <none> [...] |
Numeric Timestamp Output the io event start time (-s), end time (-t), and delta time (-D) can be included; which may be especially useful for analysis by another program (eg, a plotter). All times are in microseconds (us).
# iosnoop -stD STIME TIME DELTA UID PID D BLOCK SIZE COMM PATHNAME 64725795838 64725810400 14562 0 376 R 4614800 8192 vold /lib/libnsl.so.1 64727705209 64727719525 14315 0 7482 R 68444 1024 ls /etc/dcopy 64727721005 64727726839 5833 0 7482 R 68636 1024 ls /etc/ff 64727727626 64727728269 643 0 7482 R 68638 1024 ls /etc/fmthard 64727729060 64727738694 9634 0 7482 R 58932 1024 ls /etc/format 64727739553 64727740310 757 0 7482 R 58940 1024 ls /etc/fsck 64727741164 64727741692 527 0 7482 R 58942 1024 ls /etc/fsdb 64727742511 64727743175 663 0 7482 R 58964 1024 ls /etc/fstyp 64727743956 64727744567 611 0 7482 R 58966 1024 ls /etc/getty 64727745336 64727745969 633 0 7482 R 58968 1024 ls /etc/grpck 64727746963 64727747604 641 0 7482 R 58970 1024 ls /etc/halt 64727748490 64727749222 731 0 7482 R 58996 1024 ls /etc/inetd.conf 64727749992 64727750841 848 0 7482 R 58998 1024 ls /etc/init 64727751870 64727752582 711 0 7482 R 59002 1024 ls /etc/install 64727753388 64727753946 558 0 7482 R 59004 1024 ls /etc/killall 64727754721 64727755235 514 0 7482 R 59006 1024 ls /etc/labelit [...] |
Viewing Devices the -e switch will display the device instance names, which is often important to understand which activity is occuring on which disk. The -d option can be used to specify a device.
# iosnoop -e DEVICE UID PID D BLOCK SIZE COMM PATHNAME dad1 0 7489 W 8480 2560 sync <none> dad0 0 7489 W 2430 3584 sync <none> dad1 0 7490 R 57152 8192 bash /usr/sbin/df dad0 0 0 R 64 8192 sched <none> dad0 0 0 R 128 8192 sched <none> dad0 0 0 R 144 8192 sched <none> dad0 0 0 R 160 8192 sched <none> dad0 0 0 W 64 8192 sched <none> dad1 0 0 W 74 512 sched <none> dad1 0 0 W 91 512 sched <none> [...] # iosnoop -e -d dad1 DEVICE UID PID D BLOCK SIZE COMM PATHNAME dad1 0 7498 W 8515 44544 sync <none> dad1 0 3 W 8602 512 fsflush <none> dad1 0 0 R 266448 8192 sched <none> dad1 0 0 R 266528 8192 sched <none> dad1 0 0 W 64 3584 sched <none> dad1 0 0 W 72 512 sched <none> dad1 0 0 W 74 512 sched <none> [...] |
Dumping All Data -a will print all fields suitable to view, -A will dump them all in space delimited format.
# iosnoop -a STRTIME DEVICE UID PID D BLOCK SIZE PATHNAME ARGS 2005 Jan 6 20:09:29 dad1 0 7532 R 61392 8192 /usr/sbin/tar tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 61408 8192 /usr/sbin/tar tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 70160 1024 /etc/cron.d/.proto tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 68442 1024 /etc/crypto/crls tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 75136 1024 /etc/crypto/pkcs11.conf tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 16074 2048 /etc/crypto/kcf.conf tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 73904 1024 /etc/default/devfsadm tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 16578 2048 /etc/default/kbd tar cvf /dev/null /etc 2005 Jan 6 20:09:29 dad1 0 7532 R 73936 1024 /etc/default/keyserv tar cvf /dev/null /etc [...] # iosnoop -A TIME STIME DELTA DEVICE UID PID PPID D BLOCK SIZE MOUNT FILE PATH COMM ARGS 66726457131 66726456519 612 dad1 0 7553 2427 W 28380 1024 / motd /etc/motd vi vi /etc/motd 66726459750 66726458093 1657 dad1 0 7553 2427 W 9162 20480 <none> <none> <none> vi vi /etc/motd 66731487818 66731484492 3325 dad0 0 7554 2427 W 4416 1024 /a test /a/test vi vi /a/test 66731495744 66731488647 7096 dad0 0 7554 2427 W 2545 3584 <none> <none> <none> vi vi /a/test 66735824258 66735822766 1491 dad1 0 7555 2427 W 9202 16384 <none> <none> <none> sync sync 66735825204 66735824724 480 dad0 0 7555 2427 W 2552 1536 <none> <none> <none> sync sync 66739142859 66739142231 627 dad1 0 3 0 W 9234 512 <none> <none> <none> fsflush fsflush 66744333448 66744330277 3170 dad0 0 0 0 W 16 2048 <none> <none> <none> sched sched [...] |
Other Filters include -m to restrict iosnoop to a particular mount point, and -f to restrict iosnoop to a filename.
# iosnoop -m /export/home UID PID D BLOCK SIZE COMM PATHNAME 0 2427 R 1040 1024 bash /export/home 0 2427 R 1046 1024 bash /export/home/brendan 0 7590 R 1078 1024 grep /export/home/brendan/Dev 0 7590 R 3648 8192 grep /export/home/brendan/Dev/iosnoop 0 7591 R 1062 1024 vi /export/home/brendan/.bashrc 0 7591 W 1062 1024 vi /export/home/brendan/.bashrc [...] # iosnoop -f /var/sadm/install/contents UID PID D BLOCK SIZE COMM PATHNAME 0 7602 R 1985792 98304 pkgchk /var/sadm/install/contents 0 7602 R 8082528 131072 pkgchk /var/sadm/install/contents 0 7602 R 8082784 131072 pkgchk /var/sadm/install/contents 0 7602 R 8083040 131072 pkgchk /var/sadm/install/contents 0 7602 R 8083296 131072 pkgchk /var/sadm/install/contents [...] |
Plotting Default Output is possible by redirecting the output to a file, then importing this in StarOffice. Disk head location is red and transfer size blue. Click for larger versions,
Analysis 1 - A quiet system In this example iosnoop is run on a quiet test system for 5 minutes. Very little disk activity was observed.
# iosnoop -v STRTIME UID PID D BLOCK SIZE COMM PATHNAME 2005 Jan 6 21:50:34 0 0 R 1216 8192 sched <none> 2005 Jan 6 21:50:34 0 0 W 74 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 91 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 97 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 613 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 839 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 870 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 907 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 940 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 1113 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 1211 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 1223 4608 sched <none> 2005 Jan 6 21:50:34 0 0 W 1235 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 1315 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 89861 1536 sched <none> 2005 Jan 6 21:50:34 0 0 W 89865 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 89869 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 622418 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 3105061 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 1504 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 89009 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 89026 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 355150 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 355152 1024 sched <none> 2005 Jan 6 21:50:34 0 0 W 355157 512 sched <none> 2005 Jan 6 21:50:34 0 0 W 1504 1024 sched <none> 2005 Jan 6 21:50:53 0 3 W 9970 2048 fsflush <none> 2005 Jan 6 21:50:59 0 0 R 1216 8192 sched <none> 2005 Jan 6 21:50:59 0 0 W 74 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 91 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 97 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 133 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 613 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 745 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 839 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 870 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 940 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 1113 1024 sched <none> 2005 Jan 6 21:50:59 0 0 W 1211 512 sched <none> 2005 Jan 6 21:50:59 0 0 W 1223 4608 sched <none> 2005 Jan 6 21:51:00 0 0 W 1235 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 89861 1536 sched <none> 2005 Jan 6 21:51:00 0 0 W 89865 1024 sched <none> 2005 Jan 6 21:51:00 0 0 W 89869 1024 sched <none> 2005 Jan 6 21:51:00 0 0 W 622418 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 3105046 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 1504 1024 sched <none> 2005 Jan 6 21:51:00 0 0 W 1315 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 89009 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 89026 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 355150 1024 sched <none> 2005 Jan 6 21:51:00 0 0 W 355152 1024 sched <none> 2005 Jan 6 21:51:00 0 0 W 355157 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 3105061 512 sched <none> 2005 Jan 6 21:51:00 0 0 W 1504 1024 sched <none> 2005 Jan 6 21:51:23 0 3 W 9974 512 fsflush <none> 2005 Jan 6 21:51:30 0 0 W 907 512 sched <none> 2005 Jan 6 21:51:30 0 0 W 1504 1024 sched <none> ^C |
Analysis 2 - A busy system In this example iosnoop is run on a busy system for a few seconds. A lot of activity can be seen, as well as the process responsible.
# iosnoop -v STRTIME UID PID D BLOCK SIZE COMM PATHNAME 2005 Jan 6 22:03:21 0 7695 R 16136 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16522 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16524 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16138 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16140 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 624 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 16142 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16336 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16338 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16340 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 16342 1024 tar <unknown> 2005 Jan 6 22:03:21 0 7695 R 144 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 16346 1024 tar /usr/bin/sparcv7 2005 Jan 6 22:03:21 0 7695 R 160 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 22944 8192 tar /usr/bin/sparcv7/savecore 2005 Jan 6 22:03:21 0 7695 R 22960 8192 tar /usr/bin/sparcv7/savecore 2005 Jan 6 22:03:21 0 7695 R 22932 3072 tar /usr/bin/sparcv7/savecore 2005 Jan 6 22:03:21 0 7695 R 287104 8192 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 287120 8192 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 287136 81920 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3725600 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 3725616 16384 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3725680 16384 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3725728 24576 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3725856 24576 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3725920 24576 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3726208 8192 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3727648 131072 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3727904 131072 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 3728160 32768 tar /usr/bin/sparcv7/mdb 2005 Jan 6 22:03:21 0 7695 R 1168 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 281184 8192 tar /usr/bin/sparcv7/gcore 2005 Jan 6 22:03:21 0 7695 R 297820 2048 tar /usr/bin/sparcv7/gcore 2005 Jan 6 22:03:21 0 7695 R 281200 8192 tar /usr/bin/sparcv7/truss 2005 Jan 6 22:03:21 0 7695 R 281216 8192 tar /usr/bin/sparcv7/truss 2005 Jan 6 22:03:21 0 7695 R 281232 81920 tar /usr/bin/sparcv7/truss 2005 Jan 6 22:03:21 0 7695 R 3996464 8192 tar <none> 2005 Jan 6 22:03:21 0 7695 R 3996480 98304 tar /usr/bin/sparcv7/truss 2005 Jan 6 22:03:21 0 7695 R 16348 1024 tar /usr/bin/sparcv9 2005 Jan 6 22:03:21 0 7695 R 22976 8192 tar /usr/bin/sparcv9/amt 2005 Jan 6 22:03:21 0 7695 R 22992 8192 tar /usr/bin/sparcv9/amt 2005 Jan 6 22:03:21 0 7695 R 23008 4096 tar /usr/bin/sparcv9/amt 2005 Jan 6 22:03:21 0 7695 R 23024 8192 tar /usr/bin/sparcv9/ls 2005 Jan 6 22:03:21 0 7695 R 23040 8192 tar /usr/bin/sparcv9/ls 2005 Jan 6 22:03:21 0 7695 R 23056 13312 tar /usr/bin/sparcv9/ls 2005 Jan 6 22:03:21 0 7695 R 23088 8192 tar /usr/bin/sparcv9/moe 2005 Jan 6 22:03:21 0 7695 R 23016 4096 tar /usr/bin/sparcv9/moe 2005 Jan 6 22:03:21 0 7695 R 23104 8192 tar /usr/bin/sparcv9/newtask 2005 Jan 6 22:03:21 0 7695 R 23120 8192 tar /usr/bin/sparcv9/newtask 2005 Jan 6 22:03:21 0 7695 R 23136 5120 tar /usr/bin/sparcv9/newtask 2005 Jan 6 22:03:21 0 7695 R 23152 8192 tar /usr/bin/sparcv9/nohup 2005 Jan 6 22:03:21 0 7695 R 23168 8192 tar /usr/bin/sparcv9/nohup 2005 Jan 6 22:03:21 0 7695 R 23184 6144 tar /usr/bin/sparcv9/nohup 2005 Jan 6 22:03:21 0 7695 R 23200 8192 tar /usr/bin/sparcv9/prctl 2005 Jan 6 22:03:21 0 7695 R 23216 8192 tar /usr/bin/sparcv9/prctl 2005 Jan 6 22:03:21 0 7695 R 23232 30720 tar /usr/bin/sparcv9/prctl ^C |
Back to Brendan Gregg's Homepage