Code coverage tool for SBCL

Posted on 2007-05-03 in Lisp
5 comments

SBCL 1.0.5.28 includes an experimental code coverage tool (sb-cover) as a new contrib module. Basically you just need to compile your code with a special optimize proclamation, load it, run some tests, and then run a reporting utility. The reporting utility will produce some html files. One will contain an aggregate coverage report of your whole system, the others will show your source code transformed into angry fruit salad:

For a more substantial example, here's the coverage output for the cl-ppcre test suite.

There are still some places where the coverage output won't be what most people would intuitively expect. Some, like the handling of inlined functions, would be simple to solve. It's just not yet clear to me what the right solution would be. For example in the case of inlined functions the right solution might be suppressing inlining when compiling with coverage instrumentation, or it might be to say "don't do that, then" to the users. Others are fundamentally unsolvable, due to the impossibility of reliably mapping the forms that the compiler sees back to the exact character position in the source file. Hopefully this'll still turn out to be useful in its current state.

If you have any suggestions for improvements, I'd love to hear them.

Next ICFP 2007 (2007-07-25)
Previous ILC 2007 Summary (2007-04-11)

Comments

By Ondrej Svitek on 2007-05-03

That's a neat contrib!

What about adding support for measuring hotness of code? Hotspots would naturally emerge as islands of bright red in the sea of cold blue (or similar suggestive color mapping). It could give one a 'feel' of program execution, as we humans excel at decoding and comprehending visual information.

Of course, it wouldn't be meant as a replacement for real code profiling. Just a fancy tool for quick visual identification of code portions under heavy load, which require further attention.

I believe you have all the necessary bits and pieces to make something like this possible.

Anyway, thanks for the great work, keep it up!

By Juho on 2007-05-03

A couple of years ago I once wrote an sb-sprof extension that could annotate the original source code forms with the corresponding amount of profiler ticks:

http://jsnell.iki.fi/blog/archive/2004-10-30.html

Updating (or more likely rewriting) that code to work with the prettier sb-cover source annotation stuff shouldn't be too hard, and is probably something that I'll do some rainy day.

By anon on 2007-05-03

That's very impressive!

(Secret wish: if all the sbcl contrib code have decent documentation that'll be awesome)

BTW, I've come across this coverage tool for C

http://labs.musecurity.com/2007/03/26/code-coverage-and-fuzzing/

http://labs.musecurity.com/wp-content/uploads/2007/03/code-coverage.png http://labs.musecurity.com/wp-content/uploads/2007/03/ctags-lookup.png

I think the lesser languages benefit a lot more from this kind of tool than lisp but it's good to know that we have it if we need it.

Thanks!

By Thom on 2007-05-10

anon,

As someone who spends a lot of time in a large, organically-grown Lisp code base, I think this might actually be as or more useful to me than the code coverage I used to run on Java systems. Of course, it was my request (among others) that indirectly led Juho to do this, I think, so I've obviously got my bias.

By Paul Dietz on 2007-06-25

This is really cool, thanks.

I've been fiddling with Waters' old COVER package, which doesn't have the spiffy HTML interface. Not sure how hard that would be to add. You might want to look at how it handles annotating/not annotating code, particularly in macro expansions.

One thing I've wanted to do with COVER is make it more extensible, so, for example, it could be used with an automated test generator to evolve tests that cover untested parts of the code. It would be nice to make SB-COVER user extensible too.

Name
Message

As an antispam measure, you need to write a super-secret password below. Today's password is "xyzzy" (without the quotes).

Password

Formatting guidelines for comments: No tags, two newlines can be used for separating paragraphs, http://... is automatically turned into a link.