Flux Research Group / School of Computing
Xsmith logo

Xsmith

This project is developing new techniques for the creation of highly effective fuzz testers, also known as “fuzzers,” for programming language compilers and interpreters. Fuzz testing is an automatic and low-cost technique for finding defects in software systems. A fuzzer randomly creates test inputs for a software system; a fuzzer is effective if it can continually create test cases that reveal defects throughout the system under test. It is difficult to create effective fuzzers for programming language compilers and interpreters because these systems have highly structured inputs, but it is important that such fuzzers exist. Programming language implementations are critical software infrastructure: defects in compilers and interpreters can potentially have great costs in terms of software correctness and reliability, human productivity, and computer security. This project seeks to reduce the time and human effort needed to create sophisticated fuzzers for programming language implementations. In so doing, this project is expected to advance the state of the art in random software testing, improve the quality of several programming language implementations selected for study, and produce new open-source software tools that programmers can use to develop new and more effective fuzz testers.

The techniques developed by this project are being embodied in a new generator of fuzz testers, called Xsmith. Xsmith generates language fuzzers from specifications and thus reduces the time and effort required to create fuzzers. More importantly, Xsmith injects sophisticated program-generation techniques into the language fuzzers it creates. This project is investigating three techniques in particular. This first is generation-time analysis, intended to allow Xsmith-derived fuzzers to create output that is both complex and meaningful. The second is feature subsetting, intended to increase the likelihood that Xsmith-derived fuzzers will output bug-triggering test programs. The third is iterative refinement, intended to further diversify the outputs from Xsmith-derived fuzzers.

The project participants are using Xsmith to create fuzz testers for a varied set of programming languages. Where possible, the bug-finding power of Xsmith-derived fuzzers will be compared to that of existing fuzzers: quantitatively, in terms of the number of identifiably unique defects found within a fixed test-time budget, and qualitatively, in terms of the kinds of defects uncovered. This evaluation will allow the investigators to assess the impact of the techniques embodied in Xsmith, both individually and collectively, over a range of programming language implementations. Xsmith will be successful if it permits highly effective fuzz testers to be constructed with significantly less ad hoc code, and thus significantly less effort, than if they had been constructed from scratch.

Recent News

  • Mar 27 2021 — Xsmith presented at Racketfest ’21.
    Watch the recorded presentation.
  • Dec 14 2020 — Clotho 1.0.2 released.
  • Nov 19 2020 — Clotho 1.0.1 released.
  • Oct 19 2020 — Xsmith 2.0.2 released.
  • Oct 17 2020 — Clotho presented at Tenth RacketCon.
    Watch the recorded presentation.
  • Aug 31 2020 — Xsmith 2.0.1 released.
    This is the second public release of the Xsmith software.
  • Aug 28 2020 — Clotho paper presented at Scheme ’20.
    Watch the recorded presentation.
  • Jul 31 2020 — Clotho 1.0.0 released.
    This is the first public release of the Clotho software.

Available Software

  • Xsmith — a library and DSL for creating random program generators
    (read the documentation; get the source code)
    The easiest way to install Xsmith is to use the Racket package manager as described in the online Xsmith documentation.
  • Wasmlike (in development) — a random generator of WebAssembly (a.k.a. Wasm) programs
    (get the source code)
  • Harness (in development) — a test harness for running fuzzing campaigns on the Emulab testbed
    (get the source code)
  • Clotho — a library for parametric randomness
    (read the documentation; get the source code)
    Clotho provides controllable randomness functions for Racket programs, allowing sequences of random values to be recorded, replayed, and manipulated at a fine grain. Xsmith uses Clotho to implement “parametric random generation” of programs. The easiest way to install Clotho is to use the Racket package manager as described in the online Clotho documentation.
  • Racket RACR — a library for reference attribute grammar controlled rewriting
    (get the source code)
    This is a port of Christoff Bürger's RACR library from Scheme to Racket. Xsmith uses the Racket RACR library as described in the online Xsmith documentation.

Mailing Lists

Related Software

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant Number 1527638. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

current people

Eric Eide
Eric Eide
Faculty
Guy Watson
Guy Watson
Masters student

alumni

Pierce Darragh
Pierce Darragh
University of Maryland
Elijah Grubb
Elijah Grubb
University of Maryland