Re: [csmith-dev] Fwd: Re: csmith test harness help

[John Regehr <regehr@cs.utah.edu>]

Hi Andy,

While we get our githubbing in order (our SVN tree is currently a bit 
messy, with Xuejun having some code on a branch-- all should be well 
once he merges this onto the trunk), allow me to discuss some of the 
directions we'd like to go with Csmith's supporting scripts and drivers. 
 It'll take some work to get there and hopefully somewhere in here 
there's a sub-project or two that might be interesting and fun for you 
to work on.

The goal, as I see it, is for Csmith to be part of a turnkey compiler 
bugfinding system.  A compiler team should be able to allocate a fast 
machine somewhere in the cloud, throw this software onto it, and then 
forget about it until it starts to submit bug reports.  Besides Csmith 
itself, this kind of system has several aspects:

- it should take advantage of as many cores as are available, and 
perhaps also distribute work across multiple nodes

- it needs to be able to pull and build the latest version of the 
compiler under test every 24 hours or whatever

- instead of running executables directly -- as our little driver now 
does -- it needs to be able to run arbitrary emulators or simulators to 
test cross-compilers (we have internal scripts that do this but they are 
horrible)

- it should support fully automated minimization of failure-inducing 
test cases; this is something we're actively working on and it's kind of 
a separate project in itself -- I'm happy to provide more details if 
that would be useful

- it should be able to do a binary search over the compiler revision 
history to try to find out when some bug was introduced

- it should support other ways of determining whether a newly discovered 
failure-inducing test case corresponds to an already reported bug, or if 
it's new.  This came up on the Csmith list just today and it's very 
hard.  Some special cases, however, are easy, such as internal compiler 
errors that can be identified by a string such as "register allocator 
exploded at line 700".

- it should support minimization of the failure-inducing compiler 
options, and perhaps try to figure out the actual pass that caused the 
problem

- it should interface nicely with the existing bug workflow of the 
compiler under test.  Maybe it mails nicely-formatted bug reports to a 
key developer, maybe it drops C programs into the test suite, maybe it 
interfaces directly to the bugzilla.

Basically I want a system that incurs no overhead in terms of developer 
time when no bugs are found, but rapidly brings bugs to people's 
attention when they are introduced.  We're working on some parts of this 
but could definitely use help.  As you can imagine, basically all of 
this non-Csmith infrastructure wants to be in Perl.

John