Systemtap's complexity continues to grow.

In the beginning there was dtrace, and everyone said it was good, then Linux needed an answer so they created Systemtap, they said we don't like the limits that Dtrace imposes. We want to be able to track every byte of kernel code, we want to use loops painlessly in our scripts. We want to to be able to call any kernel function from our scripts. We want our probes to work faster and be lighter weight than dtrace's. And the list goes on. The latest additions to the list is we want an IDE, and now they are talking about moving to xml output.

It has been about a year since I started looking at systemtap, back then it was very much a alpha project, with very little working in it. Not to mention stable wasn't really part of the equation. And simple had very little do with any part of it. Yet they say were going to make it simple but as of today were still waiting.

One year later, its still not simple, though it may be getting stable for the simple stuff. Userland probes are still a ways off, they are still working on getting the support into the lower levels of systemtap called kprobes and/or jprobes.

I made a request on the mailing list and on a well read blog asking for stories about people successfully solving real problems with Systemtap. Its been over a month without a single reply, yesterday was the first entry that came close to fulfilling the request of success stories.

Monday I went to lunch with Gavin Romig-Koch. Gavin works in Red Hat support. While we were talking he mentioned trying to solve a customer problem that appeared to be caused by futex. An attempt was made to use GDB to track futex operations from userspace. However, that really slowed down the machine and never got the to reported problem. Gavin thought that SystemTap might be useful in diagnosing this problem. This morning I wrote a quick script to print out futex information while the system is running. I don't know whether this would identify the specific problem Gavin mentioned.

http://sources.redhat.com/ml/systemtap/2006-q2/msg00418.html

Well its something, this level of problem is being solved everyday with dtrace and has been for a couple of years now. Dtrace doesn't need all the bells and whistles that systemtap is insisting on providing to Linux users. Dtrace even has a set of scripts that were developed outside of Sun that provides information to some common questions, its Called Dtrace toolkit and is availible at http://www.opensolaris.org/os/community/dtrace/dtracetoolkit/ this goes a long way in providing a head start for the normal sysadmin access to dtrace's power.

Well features by them self are not bad, when they become the primary goal of the project they do the project mortal harm. The most graphic example of rampant feature-ism and its damage to an opensource project is the wine project, I began watching it years ago, back when win3.1 was current at the time Microsoft was releasing a torrent of upgrades, multimedia extensions, window for workgroups, NT, 95, 98, ME, 2000, 2k3. Wine seemed like it wasn't getting anywhere for years because it was focused on a moving target and it was. Microsoft was moving a 1000 changes a month. Luckily for the wine project, Microsoft tripped over its own pace of development and Wine finally caught up because windows basically was stagnant for 3 years while they rewrote most of its code to deal with stability and security problems.

Systemtap has a double problem, they are tracking two moving targets, the first being the Linux kernel without a stable API or ABI by design, so as new kernel releases are made the systemtap developers have to track changes and add more complexity in order to work with the current version of the kernel plus the stable ones that most users have.

The second moving target that Systemtap is trying to track is an internal one, they keep adding features not implemented in there initial project. I think they are now working on there 4^th or 5^th type of probe implementation. Each surely better than the last, but each one having to be tested and fixed to get to the level of stability the last one was at. To further complicate things, they get to deal with changes in there tool set elfutils that has to be the latest version to work with there latest probe type. They also based a lot of there features on gcc's DWARF data that gives hints at where variables and line numbers of various code segment are at in the executable, turns out that when you enable certain optimizations in certain versions of gcc it stores incorrect DWARF information. so you may not even be probing what you thought you were. They are still adding more complexity and working out rough spots that they failed to document in the start of the project, and people are coming up with new ideas and new language features, such as dealing with probes that may or may not exist in a given script.

To answer all this complexity they now are working on adding an IDE, to help build scripts where dtrace starts out at being a tool designed to be used on the spot by ordinary people that don't know much about the kernel, systemtap seems to want the user to know more and more or resort to using an IDE to develop scripts. Currently to enable an entry probe that fires, when ever the kernel enters a function or syscall you have to seek out its header file to see what the argument you want is called, not very friendly if you ask me. I have wrote up dtrace tip sheet on how dtrace makes it simple without searching header files and without a IDE. They are also working on adding a GUI and XML output to help decipher all the information that the new faster/lighter probes are dumping out. Dtrace provides the tools to summarize and filter information easily so the need for a GUI is much less. I'm starting to get the impression that it may be easier to resort to old debugging method of adding printf's(well the kernel's version kprintf) to code than to use all the excess baggage that systemtap seems to be adding to the solution. Not to mention that you need to recompile the kernel with approved optimizations and debugging information enabled.

Will anyone use a solution that is more complex than the problem it is trying to solve? They may also want to think about stabilizing on a single kernel version until they get it working and then upgrade to the next stable kernel and keep moving until they meet the goals they originally stated.

Here is a simple example of dtrace's simplicity, in this example I want to see the top 5 syscalls “ls -Rl /” is calling and how many of each in a one second. No IDE needed here, just a couple simple functions and an aggregation to monitor 229 probes and print the result.

dtrace -qn 'syscall:::entry /uid==1000/ {@[probefunc]= count();} tick-1s { trunc(@a,5); printa(@); clear(@a)}'

pathconf 336

stat64 336

lstat64 600

write 664

acl 672

pathconf 473

stat64 473

lstat64 787

write 925

acl 945

I don't even want to think what it takes in systemtap to do the same thing. If systemtap moves to xml to display this type of information it basically turns what should be easily readable information into a machine only readable solution that forces the user to break out a GUI to decipher it. Its time for systemtap to stop creating new features and solve the basic problems they set out to do when they started.