[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [csmith-dev] Fwd: Re: csmith test harness help

Hi Andy,

While we get our githubbing in order (our SVN tree is currently a bit messy, with Xuejun having some code on a branch-- all should be well once he merges this onto the trunk), allow me to discuss some of the directions we'd like to go with Csmith's supporting scripts and drivers. It'll take some work to get there and hopefully somewhere in here there's a sub-project or two that might be interesting and fun for you to work on.

The goal, as I see it, is for Csmith to be part of a turnkey compiler bugfinding system. A compiler team should be able to allocate a fast machine somewhere in the cloud, throw this software onto it, and then forget about it until it starts to submit bug reports. Besides Csmith itself, this kind of system has several aspects:

- it should take advantage of as many cores as are available, and perhaps also distribute work across multiple nodes

- it needs to be able to pull and build the latest version of the compiler under test every 24 hours or whatever

- instead of running executables directly -- as our little driver now does -- it needs to be able to run arbitrary emulators or simulators to test cross-compilers (we have internal scripts that do this but they are horrible)

- it should support fully automated minimization of failure-inducing test cases; this is something we're actively working on and it's kind of a separate project in itself -- I'm happy to provide more details if that would be useful

- it should be able to do a binary search over the compiler revision history to try to find out when some bug was introduced

- it should support other ways of determining whether a newly discovered failure-inducing test case corresponds to an already reported bug, or if it's new. This came up on the Csmith list just today and it's very hard. Some special cases, however, are easy, such as internal compiler errors that can be identified by a string such as "register allocator exploded at line 700".

- it should support minimization of the failure-inducing compiler options, and perhaps try to figure out the actual pass that caused the problem

- it should interface nicely with the existing bug workflow of the compiler under test. Maybe it mails nicely-formatted bug reports to a key developer, maybe it drops C programs into the test suite, maybe it interfaces directly to the bugzilla.

Basically I want a system that incurs no overhead in terms of developer time when no bugs are found, but rapidly brings bugs to people's attention when they are introduced. We're working on some parts of this but could definitely use help. As you can imagine, basically all of this non-Csmith infrastructure wants to be in Perl.