Scripts from a thread

Here are some interesting scripts, one-liners, and command line fu extracted/borrowed from a recent zfs-discuss mailing list thread for those that don’t follow the mailing list and want to see how the Solaris masters solve nfs/zfs problems. Note all ownership of scripts and comments belong to its creators I’m just stripping out the fluff and combining all in one place so more people can learn and benefit from it. Of course no entry on Solaris performance monitoring and tuning could be complete without mentioning the bible on the subject Solaris(TM) Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris (Solaris Series) which should be on Every Solaris sysadm's book shelf if it isn't lying open on his desk being read. For those that want to learn the raw facts on how Solaris works I highly recommend Solaris(TM) Internals: Solaris 10 and OpenSolaris Kernel Architecture (2nd Edition) (Solaris Series).
Looking at the congestion in the run queue:

sar -q 5 100

Interestingly, using prstat -mL to monitor thread latency, we see that a handful of threads are the culprates for consuming mass CPU:

prstat –mL

Running zvop_times.d (http://blogs.sun.com/erickustarz/resource/zvop_times.d) I get an idea of what ZFS is doing:

dtrace -s  zvop_times.d

dtrace: script 'zvop_times.d' matched 66 probes

time dtrace -n 'syscall:::entry /execname == "nfsd"/ { @[probefunc] = count(); }'

dtrace: description 'syscall:::entry ' matched 225 probes
^C
Of course they also used many of Brendan G’s DTraceToolkit.
Count and time syscalls from a process:

truss -c -p 22643

Whois doing IO:

dtrace -n 'io:genunix::start { @[pid, execname] = count(); }'

dtrace: description 'io:genunix::start ' matched 3 probes
Who is spinning on locks and mutexes

lockstat -P sleep 10

Adaptive mutex spin: 20406 events in 10.026 seconds (2035 events/sec)
show the stack of active kernel code paths.

dtrace -n 'profile-997hz / arg0 != 0 / { @s[stack()]=count(); }'

Watch zfs file creates:

dtrace -n 'zfs_create:entry { printf("ZFS Create: %s\n",
stringof(args[0]->v_path)); }'

prstat -mL to monitor thread latency, we see that a handful of threads are the culprates for consuming mass CPU:
How much data and how long is nfs taking to write it?
http://unixconsult.org/dtxg.d grab the script, make +x and then run it.
Spencer Shepler @ sun submitted this to the thread:
For future reference, there are two commands that you may want to
make use of in observing the behavior of the NFS server and individual
filesystems.
There is the trusty, nfsstat command. In this case, you would have been
able to do something like:

nfsstat -s -v3 60

This will provide all of the server side NFSv3 statistics on 60 second
intervals.
Then there is a new command fsstat that will provide vnode level
activity on a per filesystem basis. Therefore, if the NFS server
has multiple filesystems active and you want ot look at just one
something like this can be helpful:

fsstat /export/foo 60

Fsstat has a 'full' option that will list all of the vnode operations
or just certain types. It also will watch a filesystem type (e.g. zfs, nfs).
Very useful.