[csmith-dev] manual test case reduction

[Andreas Zwinkau <zwinkau@kit.edu>]

The next step after finding a failing program with csmith is to minimize 
the program. Regehr just posted a blog entry about delta debugging. Here 
is my manual algorithm, which i ended up with after fixing some libfirm 
bugs.

"Try" means remove one by one and check, if the program still fails.

1. Instead of test.c use the preprocessed file to get rid of
   dependencies (runtime/csmith.h).
2. Identify the function (e.g. func_10) where compiler fails (via gdb)
3. Try to set all other func_XX and main to { return 0; }
   So far i only care about compilation failures. If the generated
   program does the wrong stuff, you probably want to keep the main
   function with all the checksum printing.
4. Try to remove calls from func_10
   I do not remove the arguments, so "func_1(x,y)" => "(x,y)".
5. Try to reduce the control flow graph
   1. Remove "else", "continue", "break", "goto XX"
   2. Remove "if (...)", "for (...)", "while (...)"
6. Try to remove calculations (assignments)
7. Try to simplify the control flow graph
   Shorten conditions in remaining if/for/while statements
8. Try to remove variables
   Start from the end of the function, since the removal of a
   declaration could enable the removal of another one.
   Also, try to reuse variables to enable removals.
9. Try to use basic data types. "uint64" => "long long" => "int".
   Ideally, only ints are left afterwards. No "volatile", no "const".
10.Now remove stuff, which is not used by func_10 anymore, which
   usually includes all stuff, which was included, and all func_XX
   functions.

Now the program should only consist of a main, which does nothing, and 
func_10, which is hopefully small. Here is an example, which now lives 
in our testsuite:
http://pp.info.uni-karlsruhe.de/cgi-bin/cgit.cgi/firm-testsuite/plain/opt/ldst.c

Maybe this helps with your work on test case reduction? Maybe someone 
has a suggestion to improve my workflow?

--
Andreas Zwinkau

 Karlsruhe Institute of Technology (KIT)
 Institut für Programmstrukturen und Datenorganisation (IPD)
 Lehrstuhl Prof. Snelting
 Adenauerring 20a
 76131 Karlsruhe

 Phone:  +49 721 608 48351
 Fax:    +49 721 608 48457
 Email:  zwinkau@kit.edu
 Web:    http://pp.info.uni-karlsruhe.de/person.php?id=107

 KIT – University of the State of Baden-Wuerttemberg and
 National Research Center of the Helmholtz Association