From Chang.Xu at anu.edu.au Wed Sep 1 19:31:33 2021 From: Chang.Xu at anu.edu.au (Chang Xu) Date: Thu, 2 Sep 2021 01:31:33 +0000 Subject: [xsmith-dev] Problems of Build a DSL Fuzzer by Using Xsmith Message-ID: Dear Xsmith Developer, I am a postgraduate student who really interested in this xsmith project. It is a really cool project which can randomly create different testing cases for different program languages. Currently, my project group developed a DSL and need a fuzzer to test it. Xsmith looks prefectly suit for our testing requirements. Now, we finished to read all supporting documents and trying to modified the example ?pythonseque.rtk? to suit our DSL. However, we met lots of problems during this process. For example: ?pythonseque.rtk? has an error said? replace-hold: all choices for filling in a XsmithAstHoleRefExpr hole were filtered out. Besides, we also confused about how to defined the relation between Definition, binder-info, reference-info, prop edit, lift-field and fresh (how to define, use, and set the variable/function). Also, sometimes, after set ?fresh-name-variable? the variable name is still ?lift_?. Last, we want to defined add tensor (multiple dimension array) as a variable type. But we are not sure how to do that. Could you help us when you have time? And maybe provide some study martial or video tutorial with detail of how to study Xsmith. Best regards, Chang -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Thu Sep 2 09:47:10 2021 From: william at hatch.uno (William G Hatch) Date: Thu, 2 Sep 2021 09:47:10 -0600 Subject: [xsmith-dev] Problems of Build a DSL Fuzzer by Using Xsmith In-Reply-To: References: Message-ID: >Now, we finished to read all supporting documents and trying to modified the example ?pythonseque.rtk? to suit our DSL. My first advice is not to follow `pythonesque.rkt`, it is out of date and basically abandoned. We should probably move it to a directory with a name that says as much. I recommend looking at the `simple/javascript.rkt` example and the racket-kernel-fuzzer example. If you want to look at one that doesn't use the canned-components library, I recommend looking at Schemely. However, the canned components library is recommended. >Besides, we also confused about how to defined the relation between Definition, binder-info, reference-info, prop edit, lift-field and fresh (how to define, use, and set the variable/function). If you use the canned components library (like the Javascript and Racket fuzzers do), you probably don't need to worry about binder-info and reference-info. The edit property is also probably something you don't need to worry about until after you have made a couple iterations of a working fuzzer. >Also, sometimes, after set ?fresh-name-variable? the variable name is still ?lift_?. At the moment you can't control the naming of variables that are lifted (see the lifting section in the documentation). I'm glad you're interested in Xsmith. My main suggestions off the bat are to get started from a better base (eg. javascript.rkt), starting small with something that works with minimal features and then building from there. Also use the git version of xsmith instead of the release version (we plan to do a new release soon with several good features and fixes). If you point me to your code I'm happy to take a look at it and give you some pointers, but try looking at better bases than pythonesque first. On Thu, Sep 02, 2021 at 01:31:33AM +0000, Chang Xu wrote: >Dear Xsmith Developer, > >I am a postgraduate student who really interested in this xsmith project. It is a really cool project which can randomly create different testing cases for different program languages. > >Currently, my project group developed a DSL and need a fuzzer to test it. Xsmith looks prefectly suit for our testing requirements. Now, we finished to read all supporting documents and trying to modified the example ?pythonseque.rtk? to suit our DSL. However, we met lots of problems during this process. For example: ?pythonseque.rtk? has an error said? replace-hold: all choices for filling in a XsmithAstHoleRefExpr hole were filtered out. > >Besides, we also confused about how to defined the relation between Definition, binder-info, reference-info, prop edit, lift-field and fresh (how to define, use, and set the variable/function). > >Also, sometimes, after set ?fresh-name-variable? the variable name is still ?lift_?. > >Last, we want to defined add tensor (multiple dimension array) as a variable type. But we are not sure how to do that. > >Could you help us when you have time? And maybe provide some study martial or video tutorial with detail of how to study Xsmith. > >Best regards, >Chang From sorawee.pwase at gmail.com Thu Sep 2 18:08:59 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Thu, 2 Sep 2021 17:08:59 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: Two more issues: - https://github.com/dafny-lang/dafny/issues/1414 - https://github.com/dafny-lang/dafny/issues/1414#issuecomment-912142122 There are also a lot more bugs discovered related to multiset with zero multiplicity. The bugs are scattered in several functions, but they all stem from misunderstanding the invariant of the data structure, so I don't know how to count the number of bugs properly. Perhaps we should count it as one for each affected language? On Sat, Aug 28, 2021 at 1:05 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Two more issues (though I filed them together in one bug report, to not > clutter GitHub issues) > > https://github.com/dafny-lang/dafny/issues/1402 > > On Fri, Aug 27, 2021 at 4:31 AM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> One more bug: >> >> https://github.com/dafny-lang/dafny/issues/1397 >> >> On Fri, Aug 27, 2021 at 2:36 AM Sorawee Porncharoenwase < >> sorawee.pwase at gmail.com> wrote: >> >>> Two more bugs: >>> - https://github.com/dafny-lang/dafny/issues/1396 >>> - https://github.com/dafny-lang/dafny/issues/1395 >>> >>> The latter, again, is not discovered by the fuzzer, but by trying to >>> formalizing it (well, I still couldn't implement it either -- see the >>> original thread for the problem and question). >>> >>> On Thu, Aug 26, 2021 at 4:49 AM Sorawee Porncharoenwase < >>> sorawee.pwase at gmail.com> wrote: >>> >>>> Here's another bug: https://github.com/dafny-lang/dafny/issues/1387 >>>> >>>> On Tue, Aug 24, 2021 at 2:40 PM Sorawee Porncharoenwase < >>>> sorawee.pwase at gmail.com> wrote: >>>> >>>>> One more bug: >>>>> >>>>> https://github.com/dafny-lang/dafny/issues/1384 >>>>> >>>>> On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < >>>>> sorawee.pwase at gmail.com> wrote: >>>>> >>>>>> More bugs! >>>>>> >>>>>> - >>>>>> https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 >>>>>> - https://github.com/dafny-lang/dafny/issues/1374 >>>>>> >>>>>> On Sun, Aug 22, 2021 at 12:56 PM John Regehr >>>>>> wrote: >>>>>> >>>>>>> I'm just sort of lurking here, but I've really been enjoying this >>>>>>> thread. Great work everyone!!! >>>>>>> >>>>>>> > The latter one is not discovered by fuzzing itself. I was writing >>>>>>> a type >>>>>>> > constraint to be put in the fuzzer, and thinking what will happen >>>>>>> if we >>>>>>> > violate the constraint, so I tried it out manually and discovered >>>>>>> the issue. >>>>>>> >>>>>>> In my experience this happens a lot :) >>>>>>> >>>>>>> John >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeide at cs.utah.edu Thu Sep 2 21:00:10 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Thu, 02 Sep 2021 21:00:10 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: (Sorawee Porncharoenwase's message of "Thu, 2 Sep 2021 17:08:59 -0700") References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: Sorawee Porncharoenwase writes: > There are also a lot more bugs discovered related to multiset with zero > multiplicity. The bugs are scattered in several functions, but they all stem > from misunderstanding the invariant of the data structure, so I don't know > how to count the number of bugs properly. Perhaps we should count it as one > for each affected language? A popular metric for counting unique bugs is "correcting commit." (Which of course can be gamed, but still, it has the advantages of being "standard" and automatably countable and up to the system maintainer, not the person counting the bugs.) -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA