[creduce-dev] creduce with yarpgen

[Alessandro Mantovani <mantovanialessandro1994@gmail.com>]

Dear all,

I am using C-reduce to reduce some C testcases produced by yarpgen, that result in miscompilations, i.e., my target compiler (clang based) produces two binaries, respectively with "O0" and "O3", that, when executed, output a different result of the checksum. 

I have written an interestingness test that looks quite complete to me. Basically, I started with the one suggested on the C-reduce webpage (https://embed.cs.utah.edu/creduce/using/wrong1/test1.sh) and then added a compilation stage with asan, ubsan and msan (for both the "O0" and "O3" optimization levels). I also have tried by enabling the flag "-Werror=uninitialized", even if it could be redundant, as I have msan.

The command line that I use then is simply:

creduce ./test_interestingness.sh driver.c func.c init.h

where driver.c, func.c and init.h are the files generated by yarpgen.

That said, the problem I face is that when C-reduce ends its job I obtain two reduced driver.c and func.c files where the test() function invocation is messed up. Usually in the driver.c it is declared with the following signature:

void test()

and then invoked in the driver.c itself with a certain number N of parameters. Surprisingly, at least for me, in the func.c file, the same test() function is invoked with a number M of parameters where N != M.   

I have to underline that I am testing the arm backend of the compiler and thus, the elf binaries I obtain need to run on an Aarch64 emulator, while the sanitized versions of the same code (asan,ubsan,msan) run on x86. I attach in the current email the interestingness script and the reduced testcases I obtain. If you want to generate the original yarpgen testcases, the yarpgen version is :

yarpgen version 2.0 (build fc8851a on 2022:10:30)

invoked with the --std=c flag and the seed is 2149690884. Unfortunately I cannot share the target compiler as it is private, sorry for that. Is there any additional check or flag I can add to my interestingness test to enforce the fact that the test() function should be declared and invoked coherently ? 

Best Regards,

Alessandro

Attachment: driver.c
Description: Binary data

Attachment: func.c
Description: Binary data

#!/bin/bash

rm -f out*.txt
rm -f ubsan.txt
rm -f msan.txt
rm -f ./small_O0 ./small_O3 ./small_ubsan_O0 ./small_ubsan_O3 ./small_msan_O0 ./small_msan_O3

COMPILER="" #target compiler
EMULATOR="/usr/bin/qemu-aarch64"
INCLUDE="-I./"

TESTCASE="./driver.c ./func.c "
ulimit -t 60

if 
  $COMPILER $INCLUDE -pedantic -Wall -Werror=uninitialized -O0 $TESTCASE -o ./small_O0 > /dev/null 2>&1 &&\
  test -f "./small_O0" &&\
  $EMULATOR ./small_O0 > out_O0.txt &&\
  $COMPILER $INCLUDE -pedantic -Wall -Werror=uninitialized -O3 $TESTCASE -o ./small_O3  >out.txt 2>&1 &&\
  test -f "./small_O3" &&\
  ! grep 'undefined behavior' out.txt &&\
  ! grep 'conversions than data arguments' out.txt &&\
  ! grep 'incompatible redeclaration' out.txt &&\
  ! grep 'ordered comparison between pointer' out.txt &&\
  ! grep 'eliding middle term' out.txt &&\
  ! grep 'end of non-void function' out.txt &&\
  ! grep 'invalid in C99' out.txt &&\
  ! grep 'specifies type' out.txt &&\
  ! grep 'should return a value' out.txt &&\
  ! grep 'uninitialized' out.txt &&\
  ! grep 'incompatible pointer to' out.txt &&\
  ! grep 'incompatible integer to' out.txt &&\
  ! grep 'incompatible pointer types' out.txt &&\
  ! grep 'type specifier missing' out.txt &&\
  ! grep 'tentative array definition' out.txt &&\
  ! grep 'implicit conversion' out.txt &&\
  ! grep 'omitting the parameter' out.txt &&\
  $EMULATOR ./small_O3 >out_O3.txt 2>&1 &&\
  ! diff out_O0.txt out_O3.txt &&\
  /usr/bin/clang-17 -I./ -fsanitize=undefined,address -O0 $TESTCASE -o ./small_ubsan_O0 > /dev/null 2>&1 &&\
  /usr/bin/clang-17 -I./ -fsanitize=memory -O0 $TESTCASE -o ./small_msan_O0 > /dev/null 2>&1 &&\
  /usr/bin/clang-17 -I./ -fsanitize=undefined,address -O3 $TESTCASE -o ./small_ubsan_O3 > /dev/null 2>&1 &&\
  /usr/bin/clang-17 -I./ -fsanitize=memory -O3 $TESTCASE -o ./small_msan_O3 > /dev/null 2>&1
then
  if 
    ./small_ubsan_O0 > ubsan.txt 2>&1 &&\
    ! grep 'error' ubsan.txt && \
    ./small_ubsan_O3 > ubsan.txt 2>&1 &&\
    ! grep 'error' ubsan.txt && \
    ./small_msan_O0 > msan.txt 2>&1 && \
    ! grep -i 'memorysanitizer' msan.txt &&\
    ./small_msan_O3 > msan.txt 2>&1 && \
    ! grep -i 'memorysanitizer' msan.txt
  then
    exit 0
  else
    exit 1
  fi
else
  exit 1
fi