Weak Focus

We’ve been doing a migration to Solaris 10 at work; originally to take advantage of network stack improvements. We’re a Tibco/RV and Wombat shop doing a boatload of multicast, and so the improved efficiency was a big win for us. We’ll put aside the nasty multicast bug we ran into for the moment since I really want to talk about what’s arguably the best thing about Solaris 10: DTrace. Sun’s built an incredibly versatile tool that allows you to look at the system in ways that aren’t available anywhere else without custom kernel insertions. More impressively it allows you to do so in a straightforward manner and with little to no performance overhead. DTrace is magic.

By way of example, we recently had a situation where we noticed some of our servers experiencing really high levels of system time usage. A little digging with prstat and it’s clear that the number of system calls is out of whack. Normally we’d be somewhat stuck at this point, and finding out what’s generating the syscalls would involve a lot of truss, trial and error, and generally a big pain. With Dtrace it was a simple matter of:

dtrace -n ’syscall:::entry { @[probefunc] = count(); }’

This is a dtrace one-liner that dumps a list of all the system calls during the period the script runs, sorted by name. In our particular case, pollsys was the runaway leader of the pack, which made sense given that the application was reading heavily from the network for Tib messages, but we wanted to make sure, so:

dtrace -n ’syscall::pollsys:entry { @[execname] = count(); }’

Which gave us only the count of pollsys calls, sorted by the process that executed them. At this point we knew our app was the culprit, and an investigation of the source lead us to discover some inefficiencies which while not being simple to fix in this case can definitely be improved upon. More importantly, this is barely even scratching the surface of what DTrace can do. Performance monitoring, application profiling, it’s a brave new world out there.