The K9Toolkit is a miscellaneous collection of open source tools written using the KStat library on a Solaris 9 server, mostly in Perl. They actually work on Solaris 8 or newer. This toolkit may help locate performance bottlenecks.
Note: This page was last updated in 2006 and is no longer maintained. It is now part of my Crypt.
These programs require Solaris 8 or newer, and run on sparc or x86. The KStat perl library is shipped with Solaris.
Screenshots and Downloads
nicstat prints statistics for the network interfaces such as Kbytes/sec read and written, Packets/sec read and written, Average packet size, estimated %Utilisation and Saturation. Full example. Tim Cook has been developing the C version, which is available for both Solaris and Linux - nicstat.c. The C version is newer and recommended.
$ nicstat 1
Time Int rKb/s wKb/s rPk/s wPk/s rAvs wAvs %Util Sat
12:33:04 hme0 1.51 4.84 7.26 10.32 213.03 480.04 0.05 0.00
12:33:05 hme0 0.20 0.26 3.00 3.00 68.67 90.00 0.00 0.00
12:33:06 hme0 0.14 0.26 2.00 3.00 73.00 90.00 0.00 0.00
12:33:07 hme0 0.14 0.52 2.00 6.00 73.00 88.00 0.01 0.00
12:33:08 hme0 0.24 0.36 3.00 4.00 81.33 92.00 0.00 0.00
12:33:09 hme0 2.20 1.77 16.00 18.00 140.62 100.72 0.03 0.00
12:33:10 hme0 0.49 0.58 8.00 9.00 63.25 66.00 0.01 0.00
12:33:11 hme0 12.16 1830.38 185.06 1326.42 67.26 1413.06 15.09 0.00
12:33:12 hme0 19.03 3094.19 292.88 2229.11 66.53 1421.40 25.50 0.00
12:33:13 hme0 19.55 3151.87 301.00 2270.98 66.50 1421.20 25.98 0.00
12:33:14 hme0 11.99 1471.67 161.07 1081.45 76.25 1393.49 12.15 0.00
12:33:15 hme0 0.14 0.26 2.00 3.00 73.00 90.00 0.00 0.00
12:33:16 hme0 0.14 0.26 2.00 3.00 73.00 90.00 0.00 0.00
12:33:17 hme0 0.14 0.26 2.00 3.00 73.00 90.00 0.00 0.00
[...]
|
checkcable is a tool to print NIC status: duplex, speed, and auto-neg. checkcable can check if a cable is unplugged. It was originally a Kstat tool for this toolkit, then became a ndd tool for accuracy, and now is a Kstat/ndd hybrid (Kstat where possible). example.
# checkcable Interface Link Duplex Speed AutoNEG hme0 UP FULL 100 ON # checkcable Interface Link Duplex Speed AutoNEG hme0 DOWN FULL 100 ON |
sysperfstat displays utilisation and saturation for CPU, memory, disk and network, all on one line. The saturation values are tuned so that they are similar to the system load averages, but for saturation only. This provides a quick summary of system performance statistics. In the example below, several large files were sent over the network - %Net, %CPU and %Disk utilisation levels can be seen to rise. Full example here.
$ sysperfstat 1
------ Utilisation ------ ------ Saturation ------
Time %CPU %Mem %Disk %Net CPU Mem Disk Net
23:41:00 0.85 44.11 2.40 0.19 0.01 0.00 0.00 0.00
23:41:01 3.00 79.41 0.00 0.00 0.00 0.00 0.00 0.00
23:41:02 32.00 79.69 1.29 8.40 0.00 0.00 0.00 0.00
23:41:03 55.00 80.12 25.15 65.40 0.00 0.00 0.01 0.00
23:41:04 63.00 80.42 22.02 82.53 0.00 0.00 0.01 0.00
23:41:05 82.00 80.64 44.74 84.22 0.00 0.00 0.07 0.00
23:41:06 88.00 80.85 41.31 86.41 2.00 0.00 0.05 0.00
23:41:07 74.00 81.04 48.37 86.16 0.00 0.00 0.05 0.00
23:41:08 75.00 81.06 74.18 84.34 3.00 0.00 0.24 0.00
23:41:09 47.00 80.78 18.59 82.96 0.00 0.00 0.00 0.00
23:41:10 13.00 80.64 0.00 18.79 1.00 0.00 0.00 0.00
23:41:11 2.00 80.64 0.00 0.00 0.00 0.00 0.00 0.00
23:41:13 3.00 80.64 0.00 0.00 0.00 0.00 0.00 0.00
23:41:14 1.50 80.64 0.00 0.00 0.00 0.00 0.00 0.00
[...]
|
prtdevs is a quick tool to print out device instance names for all network interfaces, whether they are plumbed or not, plus a few more general details. Full example here.
$ prtdevs cpu: 0 2 mem: 1024 Mb disk: fd0 lofi1 md0 sd0 sd1 sd6 network: hme0 qfe0 qfe1 qfe2 qfe3 qfe4 qfe5 qfe6 qfe7 |
uptime This is a Perl version of the /usr/bin/uptime command, written as a demonstration of Perl Kstat programming. The second command in the following output is the Perl version.
$ /usr/bin/uptime 2:07am up 294 day(s), 7:38, 8 users, load average: 0.06, 0.07, 0.07 $ $ ./uptime Sun Mar 19 02:07:40 2006, up 294.32 days, load averages: 0.06, 0.07, 0.07 |
swapinfo prints virtual memory (swap) statistics, RAM and disk usage. The values are calculated from the data available in Kstat and "swap -l" (this does not require root access). Full example here.
$ swapinfo RAM _____Total 384.0 Mb RAM Unusable 21.2 Mb RAM Kernel 64.7 Mb RAM Locked 3.8 Mb RAM Used 173.8 Mb RAM Avail 120.5 Mb Disk _____Total 512.4 Mb Disk Alloc 104.4 Mb Disk Free 407.9 Mb Swap _____Total 760.7 Mb Swap Alloc 265.2 Mb Swap Unalloc 13.0 Mb Swap Avail 482.5 Mb Swap (MinFree) 46.0 Mb |
For reference, the following diagram forms the basis for the calculations used by swapinfo. This documents what appears to be the relationship between the different kernel variables used (there dosen't seem to be much documentation for these on the Internet). NOTE: swapinfo and this diagram are "under construction"! Sometimes swapinfo prints obviously false values - eg, negative amounts of RAM. Check for updates.
Swap
Disk RAM
--- --->a +----------------+----------------a --- --- --- ---
| | |1 "avail" |4 "avail" | | | | |
| swap_avail | | | | | | |
| | | | | freemem | | |
| | | A B | A B | | | | |
| --->b +----------------+----------------b --- | | |
| | |2 "unallocated" |5 "unallocated" | | | |
| | | | | availrmem | |
| | | A C | A C | | | |
| swap_resv +----------------+----------------+ | | |
| | |3 "allocated" |6 "allocated" | | pagestotal |
| | | | | | | |
| | | | C D | | | |
ani_max | | +----------------+ --- | |
| | | C D |7 "locked" | | | |
| | +----------------+----------------c | | |
| | |8 "kernel" |9 "kernel" | pageslocked | |
| | | E | | | | |
--- --->c +----------------| | | | |
| E | | | |
A swap_free +----------------+ --- --- |
B swap_avail |10 "unusable" | |
C swap_resv | | pages installed
D swap_alloc | | |
E pp_kernel +----------------+ ---
swapfs_minfree = physmem / 8
|
kstat_walk This is a simple tool used to grep Kstat values.
$ kstat_walk | grep cpu_info cpu_info:0:cpu_info0:state:on-line cpu_info:0:cpu_info0:fpu_type:sparcv9 cpu_info:0:cpu_info0:clock_MHz:333 cpu_info:0:cpu_info0:snaptime:11420375.285353 cpu_info:0:cpu_info0:state_begin:1081220233 cpu_info:0:cpu_info0:crtime:15.935192724 cpu_info:0:cpu_info0:class:misc cpu_info:0:cpu_info0:chip_id:0 cpu_info:0:cpu_info0:cpu_type:sparcv9 cpu_info:0:cpu_info0:implementation:UltraSPARC-IIi |
lbolt This is a simple tool that prints uptime based on the lbolt value read from Kstat. Full example here.
$ ./lbolt Last booted at: Tue Apr 6 13:07:42 2004 up: 132.21 days |
Update: I've deprecated bottleneck eg, loadcpu eg, loadmem eg, loaddisk eg, loadnet eg. They may return after I rethink the algorithms used.
Back to Brendan Gregg's Homepage