CacheKit is a collection of open source perl and shell programs to report on cache activity on a Solaris 8 sparc server. Tools for older Solaris and Solaris x86 are also included in the kit, as well as some SE Toolkit programs and extra Solaris 10 DTrace programs. The caches the kit reports on are: I$, D$, E$, DNLC, inode cache, ufs buffer cache, segmap cache and segvn cache. This kit assists performance tuning.

Note: This page was last updated in 2011 and is no longer maintained. It is now part of my Crypt.

UPDATE 03-Dec-2011: Unfortunately, this toolkit requires regular maintenance to match the performance instrumentation counters (PICs) available on new processors (and PAPI only helps so far), and changes to the Solaris caches. Since it's become out of date, I've moved it to the Crypt. One day I'd like bring this toolkit back up to date, but it will require a lot of work - and I suspect, kernel changes to make certain cache information available and accurate.

download version 0.96, 21-Aug-2005

Contents

• fcachestat - prints statistics from filesystem caches, text version.
• gcachestat - prints statistics from filesystem caches, GUI version.
• ccachestat - prints statistics from CPU caches.
• showsize - prints the maximum cache sizes for various caches.
• dnlcstat - DNLC (Dircectory Name Lookup Cache) hit statistics. This cache contains the pathname for a file and the associated vnode.
• inodestat - inode cache hit statistics. inodes contain the metadata for files ("ls -lis" info), and are used heavily during UFS activity (eg, permission checks).
• ufsbufstat - old UFS buffer cache hit statistics. This cache contains UFS structure blocks - cylinder group blocks, blocks of inodes and their indirects.
• segmapstat - seg_map cache hit statistics. This cache contains pages of a file's contents (read/write). This is used when programs read or write data from the filesystem.
• segvnstat - seg_vn cache hit statistics. This cache contains pages of a file's contents (mmap). The value printed is a crude approximation.
• icache - I$, Instruction cache hit statistics (L1 cache).
• dcache - D$, Data cache hit statistics (L1 cache).
• ecache - E$, External cache hit statistics (L2 cache).
• kstat_walk - Prints out a tree or list view of the Kstat structure using the Sun::Solaris::Kstat library (Solaris 8).
• dnlcsnoop.d - Watch DNLC events as they happen, including PID details. (Solaris 10).
• dnlcps.d - DNLC statistics per process. (Solaris 10).

Operating Systems

These programs have been written for a Solaris 8 (or newer) sparc server. Also included in the kit are programs for older Solaris, Solaris x86, and Solaris 10.

Screenshots

The following demonstrates the operation of each of the programs,

fcachestat - Prints filesystem cache statistics. The first line is historic, the next lines are 1 second samples. By the fourth line, a "find /" was executed. The most dramatic result was the DNLC was exhausted after 10 seconds, pushing more work to the other caches.


$ fcachestat 1 20 --- dnlc --- -- inode --- -- ufsbuf -- -- segmap -- -- segvn --- %hit total %hit total %hit total %hit total %hit total 98.42 130.9M 30.04 2.8M 99.07 19.4M 86.86 27.1M 79.27 13.5M 0.00 0 0.00 0 0.00 0 0.00 0 100.00 1 0.00 0 0.00 0 100.00 2 0.00 0 0.00 0 0.00 0 0.00 0 100.00 2 100.00 5 0.00 0 97.78 1849 56.52 46 100.00 26 100.00 266 100.00 6 96.73 5110 51.05 190 98.57 70 100.00 293 0.00 0 77.05 5055 20.29 1449 99.54 866 100.00 1660 0.00 0 44.64 4433 15.06 3114 99.95 1985 100.00 1537 0.00 0 44.74 4146 10.85 2948 99.75 1974 100.00 1723 0.00 0 50.85 3355 18.51 2069 99.62 1323 99.67 1836 40.00 5 51.30 4314 31.48 2535 100.00 1303 100.00 1933 0.00 0 39.37 3825 9.65 2995 99.95 2030 100.00 2041 0.00 0 15.03 2954 2.26 3322 99.75 2435 100.00 2607 0.00 0 15.98 3341 14.87 3564 99.43 2288 99.51 1631 75.00 8 6.10 3916 2.20 4866 99.69 3570 100.00 3088 0.00 0 10.39 3022 4.44 3556 99.69 2550 99.88 2414 0.00 3 18.25 2646 6.70 2820 99.50 1984 99.87 2307 0.00 2 21.75 2023 8.13 2055 97.38 1448 99.03 1851 0.00 18 24.77 432 15.01 413 84.03 263 87.87 437 0.00 53 12.50 960 5.55 1099 98.20 779 93.74 959 0.00 60

ccachestat - Prints CPU cache statistics. measures I$, D$, E$ hit statistics one by one and prints out summaries.

# ccachestat 5 5 --- I-Cache --- --- D-Cache --- --- E-Cache --- total miss %hit total miss %hit total miss %hit 7424k 107k 98.55 2476k 39k 98.39 879k 200k 77.20 7941k 107k 98.65 3502k 53k 98.46 860k 211k 75.43 39082k 504k 98.71 15243k 293k 98.08 3420k 487k 85.75 19926k 248k 98.75 4985k 103k 97.92 1531k 277k 81.89 11028k 146k 98.67 5340k 95k 98.21 1856k 326k 82.42

gcachestat.se - Displays statistics from the same caches as with "fcachestat", in a GUI. (SE Toolkit only).


showsize - This prints the size of various caches.


# showsize RAM_Total 2048.0 Mb icache 16.0 Kb (linesize 32 bytes) dcache 16.0 Kb (linesize 32 bytes) ecache 8.0 Mb (linesize 64 bytes) DNLC 129461 paths inode 129461 inode ufsbuf 39.3 Mb segvn_max 1703.9 Mb

dnlcstat - The first line is historic, which shows a 98% hit rate since reboot. The next 4 lines are 1 second samples. As the command ran, I executed a "find /" in another window - this has generated the activity in the last two lines.


$ dnlcstat 1 5 dnlc %hit hit miss 98.691 9849315 130597 0.000 0 0 0.000 0 0 94.432 1730 102 92.448 1371 112

inodestat - The first line is historic, which shows a 27% hit rate since reboot. The next 4 lines are 1 second samples. As the command ran, I executed a "find /" in another window - this has generated the activity in the last two lines.

$ inodestat 1 5 inode %hit hit miss 27.300 45475 121103 0.000 0 0 0.000 0 0 100.000 6 0 17.857 15 69

ufsbufstat - The first line is historic, which shows a 99% hit rate since reboot. The next 4 lines are 1 second samples. As the command ran, I executed a "find /" in another window - this has generated a little activity.


$ ufsbufstat 1 5 buf %hit hit miss 99.212 1664229 13224 0.000 0 0 0.000 0 0 100.000 3 0 0.000 0 0

segmapstat - The first line is historic, which shows a 86% hit rate since reboot. The next 4 lines are 1 second samples. As the command ran, I executed a "find /" in another window - this has generated the activity in the last two lines.


$ segmapstat 1 5 buf %hit hit miss 86.199 1815031 290606 100.000 2 0 0.000 0 0 100.000 202 0 97.826 45 1

segvnstat - The first line is historic, which shows a 79% hit rate since reboot. As segvnstat ran, a GUI command was executed during the third line scoring an 87% hit rate. It was then run again during the fifth line, this time scoring a 100% hit rate.


$ segvnstat 5 6 segv %hit hit miss 79.306 10741338 2802874 100.000 1 0 87.654 71 10 100.000 2 0 100.000 77 0 0.000 0 0

icache - This prints 1 second samples of the Instruction cache activity. The last line is a total for these samples (5 seconds).


# icache 1 5 I$ time cpu event total hit %hit 1.010 0 tick 935066 844740 90.34 2.009 0 tick 814208 735138 90.29 3.009 0 tick 617258 545385 88.36 4.009 0 tick 676246 599725 88.68 5.009 0 tick 636541 562476 88.36 5.009 1 total 3679319 3287464 89.35

dcache - This prints 1 second samples of the Data cache activity. The last line is a total for these samples (5 seconds).


# dcache 1 5 D$ time cpu event total hit %hit 1.007 0 tick 1044004 973314 93.23 2.007 0 tick 260763 220060 84.39 3.007 0 tick 383584 339143 88.41 4.007 0 tick 272251 228701 84.00 5.007 0 tick 271442 231759 85.38 5.007 1 total 2232044 1992977 89.29

ecache - This prints 1 second samples of the External cache activity. The last line is a total for these samples (5 seconds).


# ecache 1 5 E$ time cpu event total hit %hit 1.012 0 tick 243463 233505 95.91 2.012 0 tick 222128 213697 96.20 3.012 0 tick 217151 210389 96.89 4.012 0 tick 222873 213473 95.78 5.012 0 tick 263595 252975 95.97 5.012 1 total 1169210 1124039 96.14

kstat_walk - The default output of kstat_walk is to print a tree of the Kstat structure down to the values.


$ kstat_walk hme 0 hme0 phy_failures 0 tx_parity_error 0 macxmt_errors 0 crtime 20.624055158 xcvr_vendor 524311 toolong_errors 0 inits 16 txinits 0 ex_collisions 0 [...]


This demonstrates printing to level 1 only,


$ kstat_walk 1 hme daderror conskbd vmem cpu_info sderr lgrp ufs cpu [...]


This is an example of the "-g" switch, which prints the Kstat structure in a grep'able format. Here we ran grep with "dnlc", printing out lines that match,


$ kstat_walk -g | grep dnlc unix:0:dnlcstats:dir_reclaim_last:2 unix:0:dnlcstats:dir_misses:145983 unix:0:dnlcstats:dir_cached_current:9 unix:0:dnlcstats:hits:9945984 unix:0:dnlcstats:pick_free:0 unix:0:dnlcstats:purge_total_entries:10515 unix:0:dnlcstats:dir_cached_total:22 [...]

dnlcsnoop.d - dnlcsnoop.d is a Solaris 10 (DTrace) program that can print DNLC events as they happen.


# dnlcsnoop.d UID PID PPID VNODE INODE HIT CMD 100 1873 1446 df06ce40 2 Y -bash 100 1873 1446 df315dc0 8 Y -bash 100 1873 1446 df3e0b80 77 Y -bash 0 252 1 df06ce40 2 Y /usr/sbin/nscd 0 252 1 df2e50c0 1975 Y /usr/sbin/nscd 0 252 1 df06ce40 2 Y /usr/sbin/nscd 0 252 1 df2e50c0 1975 Y /usr/sbin/nscd 0 252 1 df06ce40 2 Y /usr/sbin/nscd 0 252 1 df2e50c0 1975 Y /usr/sbin/nscd 100 1446 1442 df06ce40 2 Y -bash 100 1446 1442 df315dc0 8 Y -bash 100 1446 1442 df3e0b80 77 Y -bash 100 1446 1442 df06ce40 2 Y -bash 100 1446 1442 df315dc0 8 Y -bash [...]

CacheKit can report on,

* I$ - Instruction Cache
* D$ - Data Cache
* E$ - External Cache
* DNLC - Directory Name Lookup Cache
* inode cache - Inode cache
* ufs buffer cache - old ufs buffer cache
* segmap - seg_map cache
* segvn - seg_vn cache

For more performance tuning on Solaris see,

• Sun Performance Sun's website that links to performance papers and articles.
• SE Toolkit The SE Toolkit is a well developed suite of tools for the performance professional.
• Solaris Internals This is a website to support the "Solaris Internals" book by Richard McDougall & Jim Mauro. (see "har").
• Books Recommended reading, Unix Internals.

Back to Brendan Gregg's Homepage