From sorawee.pwase at gmail.com Thu Aug 19 18:13:08 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Thu, 19 Aug 2021 17:13:08 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Two more bugs: - https://github.com/dafny-lang/dafny/issues/1356 - https://github.com/dafny-lang/dafny/issues/1357 The latter bug is not discovered by differential testing, and in fact can't be discovered by differential testing, because all Dafny compiler backends are buggy in the same way. However, it is discovered by our verification testing. We essentially correlate printed output with source location, and add assertions to those locations. We expect the Dafny static verifier to output verified, but it returns not verified. On Tue, Jul 27, 2021 at 9:41 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Hmm, that was not a hard bug. Apparently, Dafny intentionally limits the > depth of an AST structure to be at most 20, causing the crash. Perhaps it > should not crash like that, but it seems to "work" the way Dafny developers > intend. > > On Tue, Jul 27, 2021 at 9:33 AM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> Here's another one: https://github.com/dafny-lang/dafny/issues/1331 >> >> On Sun, Jul 25, 2021 at 11:20 AM John Regehr wrote: >> >>> This is cool. I hope someone is tweeting about it :) >>> >>> John >>> >>> >>> On 7/25/21 12:08 AM, Sorawee Porncharoenwase wrote: >>> > Here's another one: https://github.com/dafny-lang/dafny/issues/1325 >>> > >>> > >>> > On Tue, Jul 20, 2021 at 11:02 AM Eric Eide >> > > wrote: >>> > >>> > Sorawee Porncharoenwase >> > > writes: >>> > >>> > > By the way, here are the results so far: >>> > > - https://github.com/dafny-lang/dafny/issues/1291 >>> > >>> > > - https://github.com/dafny-lang/dafny/issues/1297 >>> > >>> > >>> > Thank you! We are eager to keep track of these. >>> > >>> > -- >>> > >>> ------------------------------------------------------------------------------- >>> > Eric Eide > . >>> >>> > University of Utah School of Computing >>> > https://www.cs.utah.edu/~eeide/ >>> . >>> > +1 801-585-5512 . Salt Lake City, Utah, USA >>> > >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeide at cs.utah.edu Thu Aug 19 19:32:16 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Thu, 19 Aug 2021 19:32:16 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: (Sorawee Porncharoenwase's message of "Thu, 19 Aug 2021 17:13:08 -0700") References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Sorawee Porncharoenwase writes: > Two more bugs: Thank you for keeping us up to date! We are very excited that you are having such success fuzzing Dafny! -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From sorawee.pwase at gmail.com Thu Aug 19 19:35:20 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Thu, 19 Aug 2021 18:35:20 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: One more bug :) https://github.com/dafny-lang/dafny/issues/1358 On Thu, Aug 19, 2021 at 6:33 PM Eric Eide wrote: > Sorawee Porncharoenwase writes: > > > Two more bugs: > > Thank you for keeping us up to date! We are very excited that you are > having > such success fuzzing Dafny! > > -- > > ------------------------------------------------------------------------------- > Eric Eide . University of Utah School of > Computing > https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, > Utah, USA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Fri Aug 20 12:38:14 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 20 Aug 2021 11:38:14 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: A lot more bugs: - https://github.com/dafny-lang/dafny/issues/1359 and https://github.com/dafny-lang/dafny/issues/1359#issuecomment-902594485 - https://github.com/dafny-lang/dafny/issues/1360 - https://github.com/dafny-lang/dafny/issues/1361 and https://github.com/dafny-lang/dafny/issues/1361#issuecomment-902842190 - https://github.com/dafny-lang/dafny/issues/1362 On Thu, Aug 19, 2021 at 6:35 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > One more bug :) > > https://github.com/dafny-lang/dafny/issues/1358 > > On Thu, Aug 19, 2021 at 6:33 PM Eric Eide wrote: > >> Sorawee Porncharoenwase writes: >> >> > Two more bugs: >> >> Thank you for keeping us up to date! We are very excited that you are >> having >> such success fuzzing Dafny! >> >> -- >> >> ------------------------------------------------------------------------------- >> Eric Eide . University of Utah School of >> Computing >> https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, >> Utah, USA >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeide at cs.utah.edu Fri Aug 20 12:57:32 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Fri, 20 Aug 2021 12:57:32 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: (Sorawee Porncharoenwase's message of "Fri, 20 Aug 2021 11:38:14 -0700") References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Sorawee Porncharoenwase writes: > A lot more bugs: [...] Lots of bugs related to multisets. I don't know Dafny well --- maybe multisets are not heavily used? -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From sorawee.pwase at gmail.com Fri Aug 20 13:02:53 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 20 Aug 2021 12:02:53 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Yeah, I think that could definitely be the case. Some of these bugs depend on the trick of setting multiplicity to 0, which I think is uncommon. On Fri, Aug 20, 2021 at 11:58 AM Eric Eide wrote: > Sorawee Porncharoenwase writes: > > > A lot more bugs: [...] > > Lots of bugs related to multisets. I don't know Dafny well --- maybe > multisets > are not heavily used? > > -- > > ------------------------------------------------------------------------------- > Eric Eide . University of Utah School of > Computing > https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, > Utah, USA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Aug 20 14:42:01 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 20 Aug 2021 14:42:01 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Thanks for the update, and I'm really glad you're having such success! Are these 4 bugs or 6? Or is it uncertain whether the two comment links are separate bugs? As an aside, I recently wrote a built-in minimizer. I haven't really documented it aside from adding it to the `--help` text. I've found it helpful for minimizing some things I've been looking at, but I'm interested in any feedback you have. You give it a path to a script that uses a file called `reduction-candidate` in its current directory, and returns 0 for successful reductions (IE the bug is still there) and non-zero for failed reductions. Basically the idea is that you give it the seed (and other configuration) of the interesting test case, then it re-generates it, then adds a minimization step before printing. I hope it will save you time while you find more Dafny bugs. :) Also, I'm curious to hear about your testing setup. Are you using any of our harness repository tools, or did you set up your own infrastructure? How have you been minimizing test cases so far? How much have you been running your fuzzer (ballpark approximation)? Have you spent much time doing things like adjusting weights? How big is your fuzzer (eg. LOC)? On Fri, Aug 20, 2021 at 11:38:14AM -0700, Sorawee Porncharoenwase wrote: >A lot more bugs: > >- https://github.com/dafny-lang/dafny/issues/1359 and >https://github.com/dafny-lang/dafny/issues/1359#issuecomment-902594485 >- https://github.com/dafny-lang/dafny/issues/1360 >- https://github.com/dafny-lang/dafny/issues/1361 and >https://github.com/dafny-lang/dafny/issues/1361#issuecomment-902842190 >- https://github.com/dafny-lang/dafny/issues/1362 > >On Thu, Aug 19, 2021 at 6:35 PM Sorawee Porncharoenwase < >sorawee.pwase at gmail.com> wrote: > >> One more bug :) >> >> https://github.com/dafny-lang/dafny/issues/1358 >> >> On Thu, Aug 19, 2021 at 6:33 PM Eric Eide wrote: >> >>> Sorawee Porncharoenwase writes: >>> >>> > Two more bugs: >>> >>> Thank you for keeping us up to date! We are very excited that you are >>> having >>> such success fuzzing Dafny! >>> >>> -- >>> >>> ------------------------------------------------------------------------------- >>> Eric Eide . University of Utah School of >>> Computing >>> https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, >>> Utah, USA >>> >> From sorawee.pwase at gmail.com Fri Aug 20 15:07:29 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 20 Aug 2021 14:07:29 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: > > Are these 4 bugs or 6? Or is it uncertain whether the two comment > links are separate bugs? > I would count it as 6. For example, https://github.com/dafny-lang/dafny/issues/1359 and https://github.com/dafny-lang/dafny/issues/1359#issuecomment-902594485 are obviously very related to each other, but the operations are different (membership vs disjointness) and the bug occurs in different set of compilers (C# and JS vs only C#). > As an aside, I recently wrote a built-in minimizer. I haven't really > Yes, I successfully hooked it up last week, though eventually I still need to read the code to understand the problem and minimize it further. > Also, I'm curious to hear about your testing setup. Are you using any > of our harness repository tools, or did you set up your own > infrastructure? I didn't use the harness repo. One reason is that we are doing verification fuzzing, which requires it to run a generated program, then correlate the output with source location, then modify the AST, and run the modified program. This means the standard workflow doesn't quite work. We use AWS Batch as the environment to run the fuzzer, which is packaged in a Docker image. > How much have you been running your fuzzer (ballpark approximation)? 350000 programs per day. This is the number that I computed for Zvonimir a while back, so it might have changed. > Have you spent much time doing things like adjusting weights? Nope, not at all. > How big is your fuzzer (eg. LOC)? > My actual fuzzer is about 1KLOC (this is not pure Xsmith -- I defined some macros to make adding some constructs more concise). But there are other files too. The oracle is about 200LOC. The pretty printer is about 300LOC (I implement pretty printer separately, since I want to be able to perform tree surgery from the output of Xsmith). The output-source location correlation is about 300LOC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sat Aug 21 20:53:58 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sat, 21 Aug 2021 19:53:58 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Two more bugs: - https://github.com/dafny-lang/dafny/issues/1365 - https://github.com/dafny-lang/dafny/issues/1367 On Fri, Aug 20, 2021 at 2:07 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Are these 4 bugs or 6? Or is it uncertain whether the two comment >> links are separate bugs? >> > > I would count it as 6. For example, > https://github.com/dafny-lang/dafny/issues/1359 and > https://github.com/dafny-lang/dafny/issues/1359#issuecomment-902594485 > are obviously very related to each other, but the operations are different > (membership vs disjointness) and the bug occurs in different set of > compilers (C# and JS vs only C#). > > > >> As an aside, I recently wrote a built-in minimizer. I haven't really >> > > Yes, I successfully hooked it up last week, though eventually I still need > to read the code to understand the problem and minimize it further. > > >> Also, I'm curious to hear about your testing setup. Are you using any >> of our harness repository tools, or did you set up your own >> infrastructure? > > > I didn't use the harness repo. One reason is that we are doing > verification fuzzing, which requires it to run a generated program, then > correlate the output with source location, then modify the AST, and run the > modified program. This means the standard workflow doesn't quite work. We > use AWS Batch as the environment to run the fuzzer, which is packaged in a > Docker image. > > >> How much have you been running your fuzzer (ballpark approximation)? > > > 350000 programs per day. This is the number that I computed for Zvonimir a > while back, so it might have changed. > > >> Have you spent much time doing things like adjusting weights? > > > Nope, not at all. > > >> How big is your fuzzer (eg. LOC)? >> > > My actual fuzzer is about 1KLOC (this is not pure Xsmith -- I defined some > macros to make adding some constructs more concise). But there are other > files too. The oracle is about 200LOC. The pretty printer is about 300LOC > (I implement pretty printer separately, since I want to be able to perform > tree surgery from the output of Xsmith). The output-source location > correlation is about 300LOC. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sun Aug 22 13:17:19 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 22 Aug 2021 12:17:19 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Two more bugs: - https://github.com/dafny-lang/dafny/issues/1372 - https://github.com/dafny-lang/dafny/issues/1373 The latter one is not discovered by fuzzing itself. I was writing a type constraint to be put in the fuzzer, and thinking what will happen if we violate the constraint, so I tried it out manually and discovered the issue. On Sat, Aug 21, 2021 at 7:53 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Two more bugs: > > - https://github.com/dafny-lang/dafny/issues/1365 > - https://github.com/dafny-lang/dafny/issues/1367 > > On Fri, Aug 20, 2021 at 2:07 PM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> Are these 4 bugs or 6? Or is it uncertain whether the two comment >>> links are separate bugs? >>> >> >> I would count it as 6. For example, >> https://github.com/dafny-lang/dafny/issues/1359 and >> https://github.com/dafny-lang/dafny/issues/1359#issuecomment-902594485 >> are obviously very related to each other, but the operations are different >> (membership vs disjointness) and the bug occurs in different set of >> compilers (C# and JS vs only C#). >> >> >> >>> As an aside, I recently wrote a built-in minimizer. I haven't really >>> >> >> Yes, I successfully hooked it up last week, though eventually I still >> need to read the code to understand the problem and minimize it further. >> >> >>> Also, I'm curious to hear about your testing setup. Are you using any >>> of our harness repository tools, or did you set up your own >>> infrastructure? >> >> >> I didn't use the harness repo. One reason is that we are doing >> verification fuzzing, which requires it to run a generated program, then >> correlate the output with source location, then modify the AST, and run the >> modified program. This means the standard workflow doesn't quite work. We >> use AWS Batch as the environment to run the fuzzer, which is packaged in a >> Docker image. >> >> >>> How much have you been running your fuzzer (ballpark approximation)? >> >> >> 350000 programs per day. This is the number that I computed for Zvonimir >> a while back, so it might have changed. >> >> >>> Have you spent much time doing things like adjusting weights? >> >> >> Nope, not at all. >> >> >>> How big is your fuzzer (eg. LOC)? >>> >> >> My actual fuzzer is about 1KLOC (this is not pure Xsmith -- I defined >> some macros to make adding some constructs more concise). But there are >> other files too. The oracle is about 200LOC. The pretty printer is about >> 300LOC (I implement pretty printer separately, since I want to be able to >> perform tree surgery from the output of Xsmith). The output-source location >> correlation is about 300LOC. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeide at cs.utah.edu Sun Aug 22 13:31:45 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Sun, 22 Aug 2021 13:31:45 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: (Sorawee Porncharoenwase's message of "Sun, 22 Aug 2021 12:17:19 -0700") References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Sorawee Porncharoenwase writes: > Two more bugs: > > - https://github.com/dafny-lang/dafny/issues/1372 > - https://github.com/dafny-lang/dafny/issues/1373 Thanks! I've noticed that these bugs are not being resolved very quickly. As I look now, you have 16 open issues and zero closed. (One open bug is labeled with "resolution"?) Is this slowing down you? After a while, it becomes hard to find new bugs if old ones are not resolved. (Because the fuzzer hits the same bugs over an over.) Eric. -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From sorawee.pwase at gmail.com Sun Aug 22 13:43:07 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 22 Aug 2021 12:43:07 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Great question! I have an overly complicated setup to deal with this issue. - Yesterday, I did submit a couple of PRs to fix some of these bugs: https://github.com/dafny-lang/dafny/pull/1368, https://github.com/dafny-lang/dafny/pull/1369, and https://github.com/dafny-lang/dafny/pull/1370. And just now I finished implementing a patcher that takes as input a list of PRs and applies them before compiling Dafny - Some constructs are needed to be disabled temporarily to not trigger known bugs - Some other bugs (mostly printing issues) can be temporarily suppressed by normalizing the outputs before comparing them. But yes, the situation is not ideal and does slow me down. On Sun, Aug 22, 2021 at 12:32 PM Eric Eide wrote: > Sorawee Porncharoenwase writes: > > > Two more bugs: > > > > - https://github.com/dafny-lang/dafny/issues/1372 > > - https://github.com/dafny-lang/dafny/issues/1373 > > Thanks! > > I've noticed that these bugs are not being resolved very quickly. As I > look > now, you have 16 open issues and zero closed. (One open bug is labeled > with > "resolution"?) > > Is this slowing down you? After a while, it becomes hard to find new bugs > if > old ones are not resolved. (Because the fuzzer hits the same bugs over an > over.) > > Eric. > > -- > > ------------------------------------------------------------------------------- > Eric Eide . University of Utah School of > Computing > https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, > Utah, USA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sun Aug 22 13:46:11 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 22 Aug 2021 12:46:11 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Oh, and some rarely triggered bugs that have unique error messages are also whitelisted, so that when they occur, they are not reported as errors. On Sun, Aug 22, 2021 at 12:43 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Great question! I have an overly complicated setup to deal with this issue. > > - Yesterday, I did submit a couple of PRs to fix some of these bugs: > https://github.com/dafny-lang/dafny/pull/1368, > https://github.com/dafny-lang/dafny/pull/1369, and > https://github.com/dafny-lang/dafny/pull/1370. And just now I finished > implementing a patcher that takes as input a list of PRs and applies them > before compiling Dafny > - Some constructs are needed to be disabled temporarily to not trigger > known bugs > - Some other bugs (mostly printing issues) can be temporarily suppressed > by normalizing the outputs before comparing them. > > But yes, the situation is not ideal and does slow me down. > > On Sun, Aug 22, 2021 at 12:32 PM Eric Eide wrote: > >> Sorawee Porncharoenwase writes: >> >> > Two more bugs: >> > >> > - https://github.com/dafny-lang/dafny/issues/1372 >> > - https://github.com/dafny-lang/dafny/issues/1373 >> >> Thanks! >> >> I've noticed that these bugs are not being resolved very quickly. As I >> look >> now, you have 16 open issues and zero closed. (One open bug is labeled >> with >> "resolution"?) >> >> Is this slowing down you? After a while, it becomes hard to find new >> bugs if >> old ones are not resolved. (Because the fuzzer hits the same bugs over an >> over.) >> >> Eric. >> >> -- >> >> ------------------------------------------------------------------------------- >> Eric Eide . University of Utah School of >> Computing >> https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, >> Utah, USA >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From regehr at cs.utah.edu Sun Aug 22 13:56:11 2021 From: regehr at cs.utah.edu (John Regehr) Date: Sun, 22 Aug 2021 13:56:11 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> I'm just sort of lurking here, but I've really been enjoying this thread. Great work everyone!!! > The latter one is not discovered by fuzzing itself. I was writing a type > constraint to be put in the fuzzer, and thinking what will happen if we > violate the constraint, so I tried it out manually and discovered the issue. In my experience this happens a lot :) John From sorawee.pwase at gmail.com Sun Aug 22 18:34:07 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 22 Aug 2021 17:34:07 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: More bugs! - https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 - https://github.com/dafny-lang/dafny/issues/1374 On Sun, Aug 22, 2021 at 12:56 PM John Regehr wrote: > I'm just sort of lurking here, but I've really been enjoying this > thread. Great work everyone!!! > > > The latter one is not discovered by fuzzing itself. I was writing a type > > constraint to be put in the fuzzer, and thinking what will happen if we > > violate the constraint, so I tried it out manually and discovered the > issue. > > In my experience this happens a lot :) > > John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Mon Aug 23 01:12:38 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Mon, 23 Aug 2021 00:12:38 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: I want to express a type constraint that it?s either a product type of length 0, 2, or 3. Is there a way to do that? fresh-type-variable with arguments seems like a solution, except that the composite type isn?t allowed to be repeated. Usually, if I have a generic type of the same arity, I can simply push fresh-type-variable in. E.g., (fresh-type-variable (immutable (array-type ...)) (immutable (set-type ...))) => (immutable (fresh-type-variable (array-type ...) (set-type ...))) The problem is that product-type?s argument is a list of types of false, so fresh-type-variable doesn?t work there. I guess I can create generic types product0, product2, and product3, and put them in fresh-type-variable, but well, I really want to avoid doing this. On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > As always, thanks! :) > > On Fri, Jul 30, 2021 at 5:29 PM William G Hatch wrote: > >> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase wrote: >> >> When I've had this situation, as a workaround, I've just made a >> >> function that returns a fresh-type-variable with all of the supported >> >> base types and maybe some constructed types. It's... not great. >> >> >> > >> >My question is, say, I want to disallow a function, a list of functions, >> a >> >list of list of functions, a list of list of list of functions, and so >> on. >> >Of course, I can express things positively: it's an int or bool or >> >listof... what? A list of fresh-type-variable runs a risk of choosing >> >function type. So I could do a listof (int or bool or listof ...) for >> >another level, and keep increasing levels. Is this what you have in mind? >> >> Yeah, that's the problem with it. You can realistically only specify >> a small set of the infinite set of constructible types (granted in >> practice the set of types usable by xsmith is bounded as well since it >> bounds the tree depth of constructed types). But specifying that list >> is a chore and you probably get poor performance if you actually try >> to generate many combinations with constructed types. So in practice >> when I have this situation so far I just... don't generate as many >> things as I otherwise could. >> >> >> However, that feature just hasn't made it to the top of my priorities >> >> yet... Sorry :( >> >> >> > >> >No worries at all! If there's anything that I can help regarding this >> >feature, let me know. It looks like the representation of a type right >> now >> >consists of a lower bound and an upper bound, and probably this feature >> >will require changing the representation, correct? >> >> The upper and lower bounds are about subtyping. If a type changes >> through unification it can affect the types connected as upper/lower >> bounds. This feature shouldn't change that, other than being more >> information that will be conveyed to the related type variables. When >> a variable is constrained by unification to remove a type from its >> allowed list, all related types in its upper/lower bounds lists also >> remove that type. Eg. if variable A is (or string int (-> int int)) >> and variable B is (or (subtype-of string) (subtype-of int) (subtype-of >> (-> int int))) and variable A is constrained to definitely not be a >> function, variable B could no longer be a function and still be a >> subtype of A. If two variables are unified with `unify!` instead of >> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both >> directions) they are more or less merged into a single variable. (IE >> the object that you get from `fresh-type-variable` can be merged with >> others and they are always canonicalized before doing anything to >> them.) >> >> Right now a type variable has a list of possible types (or #f for >> unconstrained). The list can have any number of base types and every >> compound type, but it's limited in that it can only have one of each >> compound type (though each compound type may have type variables >> inside and thus represent potentially many types). >> >> I think the needed change would be to add a list of disallowed types. >> I think this list should probably be applied recursively -- IE it >> should be pushed to child types. So eg. if you disallow function >> types in a type variable it also disallows lists of functions, etc. >> I'm not certain if there are exceptions, though. Would it make sense >> to have a recursive and non-recursive version of this? >> >> At any rate, when checking if things `can-unify?`, you would use the >> disallowed list as well as the allowed list of both variables to >> determine the outcome. When variables are unified, they need to >> propagate their disallowed list to related type variables (IE the ones >> in the upper/lower bounds lists). >> >> Another thing that would need to be worked out is what exactly to >> store in the disallowed list. Constructors for compound types? Or >> maybe predicates? Probably base types and predicates is my guess. >> >> At any rate, since we keep emailing about the issue I've thought >> through more of it than before, so now I'm much more likely to >> actually sit down and do it soon. :P >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Mon Aug 23 11:06:14 2021 From: william at hatch.uno (William G Hatch) Date: Mon, 23 Aug 2021 11:06:14 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: I agree this is too bad. But I'm sorry to say that I'm not going to improve this right now. It would require fairly major work in the type unification machinery. The workaround is maybe not the best, but it's at least pretty easy to do. On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase wrote: >I want to express a type constraint that it?s either a product type of >length 0, 2, or 3. Is there a way to do that? fresh-type-variable with >arguments seems like a solution, except that the composite type isn?t >allowed to be repeated. Usually, if I have a generic type of the same >arity, I can simply push fresh-type-variable in. E.g., > >(fresh-type-variable > (immutable (array-type ...)) > (immutable (set-type ...))) > >=> > >(immutable (fresh-type-variable (array-type ...) (set-type ...))) > >The problem is that product-type?s argument is a list of types of false, so >fresh-type-variable doesn?t work there. > >I guess I can create generic types product0, product2, and product3, and >put them in fresh-type-variable, but well, I really want to avoid doing >this. > > > > > > > > > > > > >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < >sorawee.pwase at gmail.com> wrote: > >> As always, thanks! :) >> >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch wrote: >> >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase wrote: >>> >> When I've had this situation, as a workaround, I've just made a >>> >> function that returns a fresh-type-variable with all of the supported >>> >> base types and maybe some constructed types. It's... not great. >>> >> >>> > >>> >My question is, say, I want to disallow a function, a list of functions, >>> a >>> >list of list of functions, a list of list of list of functions, and so >>> on. >>> >Of course, I can express things positively: it's an int or bool or >>> >listof... what? A list of fresh-type-variable runs a risk of choosing >>> >function type. So I could do a listof (int or bool or listof ...) for >>> >another level, and keep increasing levels. Is this what you have in mind? >>> >>> Yeah, that's the problem with it. You can realistically only specify >>> a small set of the infinite set of constructible types (granted in >>> practice the set of types usable by xsmith is bounded as well since it >>> bounds the tree depth of constructed types). But specifying that list >>> is a chore and you probably get poor performance if you actually try >>> to generate many combinations with constructed types. So in practice >>> when I have this situation so far I just... don't generate as many >>> things as I otherwise could. >>> >>> >> However, that feature just hasn't made it to the top of my priorities >>> >> yet... Sorry :( >>> >> >>> > >>> >No worries at all! If there's anything that I can help regarding this >>> >feature, let me know. It looks like the representation of a type right >>> now >>> >consists of a lower bound and an upper bound, and probably this feature >>> >will require changing the representation, correct? >>> >>> The upper and lower bounds are about subtyping. If a type changes >>> through unification it can affect the types connected as upper/lower >>> bounds. This feature shouldn't change that, other than being more >>> information that will be conveyed to the related type variables. When >>> a variable is constrained by unification to remove a type from its >>> allowed list, all related types in its upper/lower bounds lists also >>> remove that type. Eg. if variable A is (or string int (-> int int)) >>> and variable B is (or (subtype-of string) (subtype-of int) (subtype-of >>> (-> int int))) and variable A is constrained to definitely not be a >>> function, variable B could no longer be a function and still be a >>> subtype of A. If two variables are unified with `unify!` instead of >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both >>> directions) they are more or less merged into a single variable. (IE >>> the object that you get from `fresh-type-variable` can be merged with >>> others and they are always canonicalized before doing anything to >>> them.) >>> >>> Right now a type variable has a list of possible types (or #f for >>> unconstrained). The list can have any number of base types and every >>> compound type, but it's limited in that it can only have one of each >>> compound type (though each compound type may have type variables >>> inside and thus represent potentially many types). >>> >>> I think the needed change would be to add a list of disallowed types. >>> I think this list should probably be applied recursively -- IE it >>> should be pushed to child types. So eg. if you disallow function >>> types in a type variable it also disallows lists of functions, etc. >>> I'm not certain if there are exceptions, though. Would it make sense >>> to have a recursive and non-recursive version of this? >>> >>> At any rate, when checking if things `can-unify?`, you would use the >>> disallowed list as well as the allowed list of both variables to >>> determine the outcome. When variables are unified, they need to >>> propagate their disallowed list to related type variables (IE the ones >>> in the upper/lower bounds lists). >>> >>> Another thing that would need to be worked out is what exactly to >>> store in the disallowed list. Constructors for compound types? Or >>> maybe predicates? Probably base types and predicates is my guess. >>> >>> At any rate, since we keep emailing about the issue I've thought >>> through more of it than before, so now I'm much more likely to >>> actually sit down and do it soon. :P >>> >> From sorawee.pwase at gmail.com Tue Aug 24 15:40:27 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 24 Aug 2021 14:40:27 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: One more bug: https://github.com/dafny-lang/dafny/issues/1384 On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > More bugs! > > - https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 > - https://github.com/dafny-lang/dafny/issues/1374 > > On Sun, Aug 22, 2021 at 12:56 PM John Regehr wrote: > >> I'm just sort of lurking here, but I've really been enjoying this >> thread. Great work everyone!!! >> >> > The latter one is not discovered by fuzzing itself. I was writing a >> type >> > constraint to be put in the fuzzer, and thinking what will happen if we >> > violate the constraint, so I tried it out manually and discovered the >> issue. >> >> In my experience this happens a lot :) >> >> John >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Thu Aug 26 05:49:49 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Thu, 26 Aug 2021 04:49:49 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: Here's another bug: https://github.com/dafny-lang/dafny/issues/1387 On Tue, Aug 24, 2021 at 2:40 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > One more bug: > > https://github.com/dafny-lang/dafny/issues/1384 > > On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> More bugs! >> >> - https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 >> - https://github.com/dafny-lang/dafny/issues/1374 >> >> On Sun, Aug 22, 2021 at 12:56 PM John Regehr wrote: >> >>> I'm just sort of lurking here, but I've really been enjoying this >>> thread. Great work everyone!!! >>> >>> > The latter one is not discovered by fuzzing itself. I was writing a >>> type >>> > constraint to be put in the fuzzer, and thinking what will happen if >>> we >>> > violate the constraint, so I tried it out manually and discovered the >>> issue. >>> >>> In my experience this happens a lot :) >>> >>> John >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Fri Aug 27 03:33:59 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 27 Aug 2021 02:33:59 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: I feel like this should be easy, but somehow the solution eludes me... An array in Dafny must be allocated and then stored in a variable as a part of the array declaration statement (essentially, there's no array literal). The variable can then be used freely in any expression position. However, the fact that the allocation is a statement means that in function expression (which can't have any statement), the only way to get an array value in is through parameters. In particular, there is no lifting target of array type inside function expressions. How should I deal with this situation? Is it possible to create a new kind of variable reference that "tries" the array type, and use stuff like xsmith_get-reference-for-child! to see if there is a lifting target, and if there's none, then try other types (say, integer) that are guaranteed to have a lifting target instead? That doesn't seem possible since the type of the node seems already predetermined by the time fresh is invoked. On Mon, Aug 23, 2021 at 10:06 AM William G Hatch wrote: > I agree this is too bad. But I'm sorry to say that I'm not going to > improve this right now. It would require fairly major work in the > type unification machinery. The workaround is maybe not the best, but > it's at least pretty easy to do. > > On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase wrote: > >I want to express a type constraint that it?s either a product type of > >length 0, 2, or 3. Is there a way to do that? fresh-type-variable with > >arguments seems like a solution, except that the composite type isn?t > >allowed to be repeated. Usually, if I have a generic type of the same > >arity, I can simply push fresh-type-variable in. E.g., > > > >(fresh-type-variable > > (immutable (array-type ...)) > > (immutable (set-type ...))) > > > >=> > > > >(immutable (fresh-type-variable (array-type ...) (set-type ...))) > > > >The problem is that product-type?s argument is a list of types of false, > so > >fresh-type-variable doesn?t work there. > > > >I guess I can create generic types product0, product2, and product3, and > >put them in fresh-type-variable, but well, I really want to avoid doing > >this. > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < > >sorawee.pwase at gmail.com> wrote: > > > >> As always, thanks! :) > >> > >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch > wrote: > >> > >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase > wrote: > >>> >> When I've had this situation, as a workaround, I've just made a > >>> >> function that returns a fresh-type-variable with all of the > supported > >>> >> base types and maybe some constructed types. It's... not great. > >>> >> > >>> > > >>> >My question is, say, I want to disallow a function, a list of > functions, > >>> a > >>> >list of list of functions, a list of list of list of functions, and so > >>> on. > >>> >Of course, I can express things positively: it's an int or bool or > >>> >listof... what? A list of fresh-type-variable runs a risk of choosing > >>> >function type. So I could do a listof (int or bool or listof ...) for > >>> >another level, and keep increasing levels. Is this what you have in > mind? > >>> > >>> Yeah, that's the problem with it. You can realistically only specify > >>> a small set of the infinite set of constructible types (granted in > >>> practice the set of types usable by xsmith is bounded as well since it > >>> bounds the tree depth of constructed types). But specifying that list > >>> is a chore and you probably get poor performance if you actually try > >>> to generate many combinations with constructed types. So in practice > >>> when I have this situation so far I just... don't generate as many > >>> things as I otherwise could. > >>> > >>> >> However, that feature just hasn't made it to the top of my > priorities > >>> >> yet... Sorry :( > >>> >> > >>> > > >>> >No worries at all! If there's anything that I can help regarding this > >>> >feature, let me know. It looks like the representation of a type right > >>> now > >>> >consists of a lower bound and an upper bound, and probably this > feature > >>> >will require changing the representation, correct? > >>> > >>> The upper and lower bounds are about subtyping. If a type changes > >>> through unification it can affect the types connected as upper/lower > >>> bounds. This feature shouldn't change that, other than being more > >>> information that will be conveyed to the related type variables. When > >>> a variable is constrained by unification to remove a type from its > >>> allowed list, all related types in its upper/lower bounds lists also > >>> remove that type. Eg. if variable A is (or string int (-> int int)) > >>> and variable B is (or (subtype-of string) (subtype-of int) (subtype-of > >>> (-> int int))) and variable A is constrained to definitely not be a > >>> function, variable B could no longer be a function and still be a > >>> subtype of A. If two variables are unified with `unify!` instead of > >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both > >>> directions) they are more or less merged into a single variable. (IE > >>> the object that you get from `fresh-type-variable` can be merged with > >>> others and they are always canonicalized before doing anything to > >>> them.) > >>> > >>> Right now a type variable has a list of possible types (or #f for > >>> unconstrained). The list can have any number of base types and every > >>> compound type, but it's limited in that it can only have one of each > >>> compound type (though each compound type may have type variables > >>> inside and thus represent potentially many types). > >>> > >>> I think the needed change would be to add a list of disallowed types. > >>> I think this list should probably be applied recursively -- IE it > >>> should be pushed to child types. So eg. if you disallow function > >>> types in a type variable it also disallows lists of functions, etc. > >>> I'm not certain if there are exceptions, though. Would it make sense > >>> to have a recursive and non-recursive version of this? > >>> > >>> At any rate, when checking if things `can-unify?`, you would use the > >>> disallowed list as well as the allowed list of both variables to > >>> determine the outcome. When variables are unified, they need to > >>> propagate their disallowed list to related type variables (IE the ones > >>> in the upper/lower bounds lists). > >>> > >>> Another thing that would need to be worked out is what exactly to > >>> store in the disallowed list. Constructors for compound types? Or > >>> maybe predicates? Probably base types and predicates is my guess. > >>> > >>> At any rate, since we keep emailing about the issue I've thought > >>> through more of it than before, so now I'm much more likely to > >>> actually sit down and do it soon. :P > >>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Fri Aug 27 03:36:09 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 27 Aug 2021 02:36:09 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: Two more bugs: - https://github.com/dafny-lang/dafny/issues/1396 - https://github.com/dafny-lang/dafny/issues/1395 The latter, again, is not discovered by the fuzzer, but by trying to formalizing it (well, I still couldn't implement it either -- see the original thread for the problem and question). On Thu, Aug 26, 2021 at 4:49 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Here's another bug: https://github.com/dafny-lang/dafny/issues/1387 > > On Tue, Aug 24, 2021 at 2:40 PM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> One more bug: >> >> https://github.com/dafny-lang/dafny/issues/1384 >> >> On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < >> sorawee.pwase at gmail.com> wrote: >> >>> More bugs! >>> >>> - https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 >>> - https://github.com/dafny-lang/dafny/issues/1374 >>> >>> On Sun, Aug 22, 2021 at 12:56 PM John Regehr wrote: >>> >>>> I'm just sort of lurking here, but I've really been enjoying this >>>> thread. Great work everyone!!! >>>> >>>> > The latter one is not discovered by fuzzing itself. I was writing a >>>> type >>>> > constraint to be put in the fuzzer, and thinking what will happen if >>>> we >>>> > violate the constraint, so I tried it out manually and discovered the >>>> issue. >>>> >>>> In my experience this happens a lot :) >>>> >>>> John >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Fri Aug 27 05:31:55 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 27 Aug 2021 04:31:55 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: One more bug: https://github.com/dafny-lang/dafny/issues/1397 On Fri, Aug 27, 2021 at 2:36 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Two more bugs: > - https://github.com/dafny-lang/dafny/issues/1396 > - https://github.com/dafny-lang/dafny/issues/1395 > > The latter, again, is not discovered by the fuzzer, but by trying to > formalizing it (well, I still couldn't implement it either -- see the > original thread for the problem and question). > > On Thu, Aug 26, 2021 at 4:49 AM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> Here's another bug: https://github.com/dafny-lang/dafny/issues/1387 >> >> On Tue, Aug 24, 2021 at 2:40 PM Sorawee Porncharoenwase < >> sorawee.pwase at gmail.com> wrote: >> >>> One more bug: >>> >>> https://github.com/dafny-lang/dafny/issues/1384 >>> >>> On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < >>> sorawee.pwase at gmail.com> wrote: >>> >>>> More bugs! >>>> >>>> - >>>> https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 >>>> - https://github.com/dafny-lang/dafny/issues/1374 >>>> >>>> On Sun, Aug 22, 2021 at 12:56 PM John Regehr >>>> wrote: >>>> >>>>> I'm just sort of lurking here, but I've really been enjoying this >>>>> thread. Great work everyone!!! >>>>> >>>>> > The latter one is not discovered by fuzzing itself. I was writing a >>>>> type >>>>> > constraint to be put in the fuzzer, and thinking what will happen if >>>>> we >>>>> > violate the constraint, so I tried it out manually and discovered >>>>> the issue. >>>>> >>>>> In my experience this happens a lot :) >>>>> >>>>> John >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Aug 27 15:10:42 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 27 Aug 2021 15:10:42 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: In Dafny do function expressions not generate closures? My first thought is that you can add a choice filter (with choice-filters-to-apply, defined probably with add-choice-method) that disallows expressions that need an array type unless there is a variable already visible or its in a context where it can lift one. > function expression (which can't have any statement) As an aside, I hate languages that do this... On Fri, Aug 27, 2021 at 02:33:59AM -0700, Sorawee Porncharoenwase wrote: >I feel like this should be easy, but somehow the solution eludes me... > >An array in Dafny must be allocated and then stored in a variable as a part >of the array declaration statement (essentially, there's no array literal). >The variable can then be used freely in any expression position. However, >the fact that the allocation is a statement means that in function >expression (which can't have any statement), the only way to get an array >value in is through parameters. In particular, there is no lifting target >of array type inside function expressions. > >How should I deal with this situation? Is it possible to create a new kind >of variable reference that "tries" the array type, and use stuff like >xsmith_get-reference-for-child! to see if there is a lifting target, and if >there's none, then try other types (say, integer) that are guaranteed to >have a lifting target instead? That doesn't seem possible since the type of >the node seems already predetermined by the time fresh is invoked. > >On Mon, Aug 23, 2021 at 10:06 AM William G Hatch wrote: > >> I agree this is too bad. But I'm sorry to say that I'm not going to >> improve this right now. It would require fairly major work in the >> type unification machinery. The workaround is maybe not the best, but >> it's at least pretty easy to do. >> >> On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase wrote: >> >I want to express a type constraint that it?s either a product type of >> >length 0, 2, or 3. Is there a way to do that? fresh-type-variable with >> >arguments seems like a solution, except that the composite type isn?t >> >allowed to be repeated. Usually, if I have a generic type of the same >> >arity, I can simply push fresh-type-variable in. E.g., >> > >> >(fresh-type-variable >> > (immutable (array-type ...)) >> > (immutable (set-type ...))) >> > >> >=> >> > >> >(immutable (fresh-type-variable (array-type ...) (set-type ...))) >> > >> >The problem is that product-type?s argument is a list of types of false, >> so >> >fresh-type-variable doesn?t work there. >> > >> >I guess I can create generic types product0, product2, and product3, and >> >put them in fresh-type-variable, but well, I really want to avoid doing >> >this. >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < >> >sorawee.pwase at gmail.com> wrote: >> > >> >> As always, thanks! :) >> >> >> >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch >> wrote: >> >> >> >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase >> wrote: >> >>> >> When I've had this situation, as a workaround, I've just made a >> >>> >> function that returns a fresh-type-variable with all of the >> supported >> >>> >> base types and maybe some constructed types. It's... not great. >> >>> >> >> >>> > >> >>> >My question is, say, I want to disallow a function, a list of >> functions, >> >>> a >> >>> >list of list of functions, a list of list of list of functions, and so >> >>> on. >> >>> >Of course, I can express things positively: it's an int or bool or >> >>> >listof... what? A list of fresh-type-variable runs a risk of choosing >> >>> >function type. So I could do a listof (int or bool or listof ...) for >> >>> >another level, and keep increasing levels. Is this what you have in >> mind? >> >>> >> >>> Yeah, that's the problem with it. You can realistically only specify >> >>> a small set of the infinite set of constructible types (granted in >> >>> practice the set of types usable by xsmith is bounded as well since it >> >>> bounds the tree depth of constructed types). But specifying that list >> >>> is a chore and you probably get poor performance if you actually try >> >>> to generate many combinations with constructed types. So in practice >> >>> when I have this situation so far I just... don't generate as many >> >>> things as I otherwise could. >> >>> >> >>> >> However, that feature just hasn't made it to the top of my >> priorities >> >>> >> yet... Sorry :( >> >>> >> >> >>> > >> >>> >No worries at all! If there's anything that I can help regarding this >> >>> >feature, let me know. It looks like the representation of a type right >> >>> now >> >>> >consists of a lower bound and an upper bound, and probably this >> feature >> >>> >will require changing the representation, correct? >> >>> >> >>> The upper and lower bounds are about subtyping. If a type changes >> >>> through unification it can affect the types connected as upper/lower >> >>> bounds. This feature shouldn't change that, other than being more >> >>> information that will be conveyed to the related type variables. When >> >>> a variable is constrained by unification to remove a type from its >> >>> allowed list, all related types in its upper/lower bounds lists also >> >>> remove that type. Eg. if variable A is (or string int (-> int int)) >> >>> and variable B is (or (subtype-of string) (subtype-of int) (subtype-of >> >>> (-> int int))) and variable A is constrained to definitely not be a >> >>> function, variable B could no longer be a function and still be a >> >>> subtype of A. If two variables are unified with `unify!` instead of >> >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both >> >>> directions) they are more or less merged into a single variable. (IE >> >>> the object that you get from `fresh-type-variable` can be merged with >> >>> others and they are always canonicalized before doing anything to >> >>> them.) >> >>> >> >>> Right now a type variable has a list of possible types (or #f for >> >>> unconstrained). The list can have any number of base types and every >> >>> compound type, but it's limited in that it can only have one of each >> >>> compound type (though each compound type may have type variables >> >>> inside and thus represent potentially many types). >> >>> >> >>> I think the needed change would be to add a list of disallowed types. >> >>> I think this list should probably be applied recursively -- IE it >> >>> should be pushed to child types. So eg. if you disallow function >> >>> types in a type variable it also disallows lists of functions, etc. >> >>> I'm not certain if there are exceptions, though. Would it make sense >> >>> to have a recursive and non-recursive version of this? >> >>> >> >>> At any rate, when checking if things `can-unify?`, you would use the >> >>> disallowed list as well as the allowed list of both variables to >> >>> determine the outcome. When variables are unified, they need to >> >>> propagate their disallowed list to related type variables (IE the ones >> >>> in the upper/lower bounds lists). >> >>> >> >>> Another thing that would need to be worked out is what exactly to >> >>> store in the disallowed list. Constructors for compound types? Or >> >>> maybe predicates? Probably base types and predicates is my guess. >> >>> >> >>> At any rate, since we keep emailing about the issue I've thought >> >>> through more of it than before, so now I'm much more likely to >> >>> actually sit down and do it soon. :P >> >>> >> >> >> From sorawee.pwase at gmail.com Fri Aug 27 15:25:38 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 27 Aug 2021 14:25:38 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: So, this is actually another question that I meant to ask. I have this code as a choice method for choice-filters-to-apply: [VariableReference (? () (define t ($xsmith_type (current-hole))) (define method? (can-unify? t (method-type (fresh-type-variable) (fresh-type-variable)))) (cond [method? (and (parent-node (current-hole)) (equal? (ast-node-type (parent-node (current-hole))) 'ProcedureApplicationForMethod))] ;; we are generating default return values. ;; at this point, there's no lifting target, so we can't have ;; variable reference [(and (parent-node (current-hole)) (equal? (ast-node-type (parent-node (current-hole))) 'TopLevelMethod)) #f] [else #t]))] I raised a concern earlier that can-unify? is not ideal because ($xsmith_type (current-hole)) is sometimes unconstrained, meaning it rejects more than it should, and you agree with that. What?s the right way to do this properly? On Fri, Aug 27, 2021 at 2:10 PM William G Hatch wrote: > In Dafny do function expressions not generate closures? > > My first thought is that you can add a choice filter (with > choice-filters-to-apply, defined probably with add-choice-method) that > disallows expressions that need an array type unless there is a > variable already visible or its in a context where it can lift one. > > > function expression (which can't have any statement) > > As an aside, I hate languages that do this... > > On Fri, Aug 27, 2021 at 02:33:59AM -0700, Sorawee Porncharoenwase wrote: > >I feel like this should be easy, but somehow the solution eludes me... > > > >An array in Dafny must be allocated and then stored in a variable as a > part > >of the array declaration statement (essentially, there's no array > literal). > >The variable can then be used freely in any expression position. However, > >the fact that the allocation is a statement means that in function > >expression (which can't have any statement), the only way to get an array > >value in is through parameters. In particular, there is no lifting target > >of array type inside function expressions. > > > >How should I deal with this situation? Is it possible to create a new kind > >of variable reference that "tries" the array type, and use stuff like > >xsmith_get-reference-for-child! to see if there is a lifting target, and > if > >there's none, then try other types (say, integer) that are guaranteed to > >have a lifting target instead? That doesn't seem possible since the type > of > >the node seems already predetermined by the time fresh is invoked. > > > >On Mon, Aug 23, 2021 at 10:06 AM William G Hatch > wrote: > > > >> I agree this is too bad. But I'm sorry to say that I'm not going to > >> improve this right now. It would require fairly major work in the > >> type unification machinery. The workaround is maybe not the best, but > >> it's at least pretty easy to do. > >> > >> On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase wrote: > >> >I want to express a type constraint that it?s either a product type of > >> >length 0, 2, or 3. Is there a way to do that? fresh-type-variable with > >> >arguments seems like a solution, except that the composite type isn?t > >> >allowed to be repeated. Usually, if I have a generic type of the same > >> >arity, I can simply push fresh-type-variable in. E.g., > >> > > >> >(fresh-type-variable > >> > (immutable (array-type ...)) > >> > (immutable (set-type ...))) > >> > > >> >=> > >> > > >> >(immutable (fresh-type-variable (array-type ...) (set-type ...))) > >> > > >> >The problem is that product-type?s argument is a list of types of > false, > >> so > >> >fresh-type-variable doesn?t work there. > >> > > >> >I guess I can create generic types product0, product2, and product3, > and > >> >put them in fresh-type-variable, but well, I really want to avoid doing > >> >this. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < > >> >sorawee.pwase at gmail.com> wrote: > >> > > >> >> As always, thanks! :) > >> >> > >> >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch > >> wrote: > >> >> > >> >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase > >> wrote: > >> >>> >> When I've had this situation, as a workaround, I've just made a > >> >>> >> function that returns a fresh-type-variable with all of the > >> supported > >> >>> >> base types and maybe some constructed types. It's... not great. > >> >>> >> > >> >>> > > >> >>> >My question is, say, I want to disallow a function, a list of > >> functions, > >> >>> a > >> >>> >list of list of functions, a list of list of list of functions, > and so > >> >>> on. > >> >>> >Of course, I can express things positively: it's an int or bool or > >> >>> >listof... what? A list of fresh-type-variable runs a risk of > choosing > >> >>> >function type. So I could do a listof (int or bool or listof ...) > for > >> >>> >another level, and keep increasing levels. Is this what you have in > >> mind? > >> >>> > >> >>> Yeah, that's the problem with it. You can realistically only > specify > >> >>> a small set of the infinite set of constructible types (granted in > >> >>> practice the set of types usable by xsmith is bounded as well since > it > >> >>> bounds the tree depth of constructed types). But specifying that > list > >> >>> is a chore and you probably get poor performance if you actually try > >> >>> to generate many combinations with constructed types. So in > practice > >> >>> when I have this situation so far I just... don't generate as many > >> >>> things as I otherwise could. > >> >>> > >> >>> >> However, that feature just hasn't made it to the top of my > >> priorities > >> >>> >> yet... Sorry :( > >> >>> >> > >> >>> > > >> >>> >No worries at all! If there's anything that I can help regarding > this > >> >>> >feature, let me know. It looks like the representation of a type > right > >> >>> now > >> >>> >consists of a lower bound and an upper bound, and probably this > >> feature > >> >>> >will require changing the representation, correct? > >> >>> > >> >>> The upper and lower bounds are about subtyping. If a type changes > >> >>> through unification it can affect the types connected as upper/lower > >> >>> bounds. This feature shouldn't change that, other than being more > >> >>> information that will be conveyed to the related type variables. > When > >> >>> a variable is constrained by unification to remove a type from its > >> >>> allowed list, all related types in its upper/lower bounds lists also > >> >>> remove that type. Eg. if variable A is (or string int (-> int int)) > >> >>> and variable B is (or (subtype-of string) (subtype-of int) > (subtype-of > >> >>> (-> int int))) and variable A is constrained to definitely not be a > >> >>> function, variable B could no longer be a function and still be a > >> >>> subtype of A. If two variables are unified with `unify!` instead of > >> >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in > both > >> >>> directions) they are more or less merged into a single variable. > (IE > >> >>> the object that you get from `fresh-type-variable` can be merged > with > >> >>> others and they are always canonicalized before doing anything to > >> >>> them.) > >> >>> > >> >>> Right now a type variable has a list of possible types (or #f for > >> >>> unconstrained). The list can have any number of base types and > every > >> >>> compound type, but it's limited in that it can only have one of each > >> >>> compound type (though each compound type may have type variables > >> >>> inside and thus represent potentially many types). > >> >>> > >> >>> I think the needed change would be to add a list of disallowed > types. > >> >>> I think this list should probably be applied recursively -- IE it > >> >>> should be pushed to child types. So eg. if you disallow function > >> >>> types in a type variable it also disallows lists of functions, etc. > >> >>> I'm not certain if there are exceptions, though. Would it make > sense > >> >>> to have a recursive and non-recursive version of this? > >> >>> > >> >>> At any rate, when checking if things `can-unify?`, you would use the > >> >>> disallowed list as well as the allowed list of both variables to > >> >>> determine the outcome. When variables are unified, they need to > >> >>> propagate their disallowed list to related type variables (IE the > ones > >> >>> in the upper/lower bounds lists). > >> >>> > >> >>> Another thing that would need to be worked out is what exactly to > >> >>> store in the disallowed list. Constructors for compound types? Or > >> >>> maybe predicates? Probably base types and predicates is my guess. > >> >>> > >> >>> At any rate, since we keep emailing about the issue I've thought > >> >>> through more of it than before, so now I'm much more likely to > >> >>> actually sit down and do it soon. :P > >> >>> > >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Fri Aug 27 17:03:30 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 27 Aug 2021 16:03:30 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Oh, and about this question: > In Dafny do function expressions not generate closures? > It does, but an important part of the language is top-level function expression definition, which would not have a lifting target. So yes, choice-filters-to-apply seems like a solution, but see my above question about how to use it properly. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Aug 27 18:24:16 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 27 Aug 2021 18:24:16 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: And it doesn't have global variables that you could lift to? I ask mostly because I don't have a great answer for the other question. (The answer should be ?oh, you can specify types that a variable can't be, and just use that when you make a decision that precludes certain types without having to fully commit to a type?, as we've discussed, but I still haven't implemented that.) On Fri, Aug 27, 2021 at 04:03:30PM -0700, Sorawee Porncharoenwase wrote: >Oh, and about this question: > >> In Dafny do function expressions not generate closures? >> > >It does, but an important part of the language is top-level function >expression definition, which would not have a lifting target. > >So yes, choice-filters-to-apply seems like a solution, but see my above >question about how to use it properly. > >Thanks! From william at hatch.uno Fri Aug 27 18:29:12 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 27 Aug 2021 18:29:12 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: As hinted in my other reply, the best answer would be that Xsmith type variables have a list of types that they can't be in addition to their list of possible types. Then you could test whether it is possibly, definitely, or definitely not a given thing, and if you make a choice that it isn't something you can specify just that without removing other flexibility later. But since that feature still doesn't exist, I don't have a great answer. Sometimes in situations like this I punt to concretizing the type. It removes flexibility for later choices, but lets me make a clear decision about the type now (without forcing all unconstrained variables into the same choice). On Fri, Aug 27, 2021 at 02:25:38PM -0700, Sorawee Porncharoenwase wrote: >So, this is actually another question that I meant to ask. I have this code >as a choice method for choice-filters-to-apply: > >[VariableReference > (? () > (define t ($xsmith_type (current-hole))) > (define method? (can-unify? t (method-type (fresh-type-variable) >(fresh-type-variable)))) > (cond > [method? > (and (parent-node (current-hole)) > (equal? (ast-node-type (parent-node (current-hole))) > 'ProcedureApplicationForMethod))] > ;; we are generating default return values. > ;; at this point, there's no lifting target, so we can't have > ;; variable reference > [(and (parent-node (current-hole)) > (equal? (ast-node-type (parent-node (current-hole))) > 'TopLevelMethod)) > #f] > [else #t]))] > >I raised a concern earlier that can-unify? is not ideal because > >($xsmith_type (current-hole)) > >is sometimes unconstrained, meaning it rejects more than it should, and you >agree with that. What?s the right way to do this properly? > > >On Fri, Aug 27, 2021 at 2:10 PM William G Hatch wrote: > >> In Dafny do function expressions not generate closures? >> >> My first thought is that you can add a choice filter (with >> choice-filters-to-apply, defined probably with add-choice-method) that >> disallows expressions that need an array type unless there is a >> variable already visible or its in a context where it can lift one. >> >> > function expression (which can't have any statement) >> >> As an aside, I hate languages that do this... >> >> On Fri, Aug 27, 2021 at 02:33:59AM -0700, Sorawee Porncharoenwase wrote: >> >I feel like this should be easy, but somehow the solution eludes me... >> > >> >An array in Dafny must be allocated and then stored in a variable as a >> part >> >of the array declaration statement (essentially, there's no array >> literal). >> >The variable can then be used freely in any expression position. However, >> >the fact that the allocation is a statement means that in function >> >expression (which can't have any statement), the only way to get an array >> >value in is through parameters. In particular, there is no lifting target >> >of array type inside function expressions. >> > >> >How should I deal with this situation? Is it possible to create a new kind >> >of variable reference that "tries" the array type, and use stuff like >> >xsmith_get-reference-for-child! to see if there is a lifting target, and >> if >> >there's none, then try other types (say, integer) that are guaranteed to >> >have a lifting target instead? That doesn't seem possible since the type >> of >> >the node seems already predetermined by the time fresh is invoked. >> > >> >On Mon, Aug 23, 2021 at 10:06 AM William G Hatch >> wrote: >> > >> >> I agree this is too bad. But I'm sorry to say that I'm not going to >> >> improve this right now. It would require fairly major work in the >> >> type unification machinery. The workaround is maybe not the best, but >> >> it's at least pretty easy to do. >> >> >> >> On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase wrote: >> >> >I want to express a type constraint that it?s either a product type of >> >> >length 0, 2, or 3. Is there a way to do that? fresh-type-variable with >> >> >arguments seems like a solution, except that the composite type isn?t >> >> >allowed to be repeated. Usually, if I have a generic type of the same >> >> >arity, I can simply push fresh-type-variable in. E.g., >> >> > >> >> >(fresh-type-variable >> >> > (immutable (array-type ...)) >> >> > (immutable (set-type ...))) >> >> > >> >> >=> >> >> > >> >> >(immutable (fresh-type-variable (array-type ...) (set-type ...))) >> >> > >> >> >The problem is that product-type?s argument is a list of types of >> false, >> >> so >> >> >fresh-type-variable doesn?t work there. >> >> > >> >> >I guess I can create generic types product0, product2, and product3, >> and >> >> >put them in fresh-type-variable, but well, I really want to avoid doing >> >> >this. >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < >> >> >sorawee.pwase at gmail.com> wrote: >> >> > >> >> >> As always, thanks! :) >> >> >> >> >> >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch >> >> wrote: >> >> >> >> >> >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase >> >> wrote: >> >> >>> >> When I've had this situation, as a workaround, I've just made a >> >> >>> >> function that returns a fresh-type-variable with all of the >> >> supported >> >> >>> >> base types and maybe some constructed types. It's... not great. >> >> >>> >> >> >> >>> > >> >> >>> >My question is, say, I want to disallow a function, a list of >> >> functions, >> >> >>> a >> >> >>> >list of list of functions, a list of list of list of functions, >> and so >> >> >>> on. >> >> >>> >Of course, I can express things positively: it's an int or bool or >> >> >>> >listof... what? A list of fresh-type-variable runs a risk of >> choosing >> >> >>> >function type. So I could do a listof (int or bool or listof ...) >> for >> >> >>> >another level, and keep increasing levels. Is this what you have in >> >> mind? >> >> >>> >> >> >>> Yeah, that's the problem with it. You can realistically only >> specify >> >> >>> a small set of the infinite set of constructible types (granted in >> >> >>> practice the set of types usable by xsmith is bounded as well since >> it >> >> >>> bounds the tree depth of constructed types). But specifying that >> list >> >> >>> is a chore and you probably get poor performance if you actually try >> >> >>> to generate many combinations with constructed types. So in >> practice >> >> >>> when I have this situation so far I just... don't generate as many >> >> >>> things as I otherwise could. >> >> >>> >> >> >>> >> However, that feature just hasn't made it to the top of my >> >> priorities >> >> >>> >> yet... Sorry :( >> >> >>> >> >> >> >>> > >> >> >>> >No worries at all! If there's anything that I can help regarding >> this >> >> >>> >feature, let me know. It looks like the representation of a type >> right >> >> >>> now >> >> >>> >consists of a lower bound and an upper bound, and probably this >> >> feature >> >> >>> >will require changing the representation, correct? >> >> >>> >> >> >>> The upper and lower bounds are about subtyping. If a type changes >> >> >>> through unification it can affect the types connected as upper/lower >> >> >>> bounds. This feature shouldn't change that, other than being more >> >> >>> information that will be conveyed to the related type variables. >> When >> >> >>> a variable is constrained by unification to remove a type from its >> >> >>> allowed list, all related types in its upper/lower bounds lists also >> >> >>> remove that type. Eg. if variable A is (or string int (-> int int)) >> >> >>> and variable B is (or (subtype-of string) (subtype-of int) >> (subtype-of >> >> >>> (-> int int))) and variable A is constrained to definitely not be a >> >> >>> function, variable B could no longer be a function and still be a >> >> >>> subtype of A. If two variables are unified with `unify!` instead of >> >> >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in >> both >> >> >>> directions) they are more or less merged into a single variable. >> (IE >> >> >>> the object that you get from `fresh-type-variable` can be merged >> with >> >> >>> others and they are always canonicalized before doing anything to >> >> >>> them.) >> >> >>> >> >> >>> Right now a type variable has a list of possible types (or #f for >> >> >>> unconstrained). The list can have any number of base types and >> every >> >> >>> compound type, but it's limited in that it can only have one of each >> >> >>> compound type (though each compound type may have type variables >> >> >>> inside and thus represent potentially many types). >> >> >>> >> >> >>> I think the needed change would be to add a list of disallowed >> types. >> >> >>> I think this list should probably be applied recursively -- IE it >> >> >>> should be pushed to child types. So eg. if you disallow function >> >> >>> types in a type variable it also disallows lists of functions, etc. >> >> >>> I'm not certain if there are exceptions, though. Would it make >> sense >> >> >>> to have a recursive and non-recursive version of this? >> >> >>> >> >> >>> At any rate, when checking if things `can-unify?`, you would use the >> >> >>> disallowed list as well as the allowed list of both variables to >> >> >>> determine the outcome. When variables are unified, they need to >> >> >>> propagate their disallowed list to related type variables (IE the >> ones >> >> >>> in the upper/lower bounds lists). >> >> >>> >> >> >>> Another thing that would need to be worked out is what exactly to >> >> >>> store in the disallowed list. Constructors for compound types? Or >> >> >>> maybe predicates? Probably base types and predicates is my guess. >> >> >>> >> >> >>> At any rate, since we keep emailing about the issue I've thought >> >> >>> through more of it than before, so now I'm much more likely to >> >> >>> actually sit down and do it soon. :P >> >> >>> >> >> >> >> >> >> From sorawee.pwase at gmail.com Sat Aug 28 14:05:52 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sat, 28 Aug 2021 13:05:52 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> <37039461-3c7d-74ca-defe-08c355c63594@cs.utah.edu> Message-ID: Two more issues (though I filed them together in one bug report, to not clutter GitHub issues) https://github.com/dafny-lang/dafny/issues/1402 On Fri, Aug 27, 2021 at 4:31 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > One more bug: > > https://github.com/dafny-lang/dafny/issues/1397 > > On Fri, Aug 27, 2021 at 2:36 AM Sorawee Porncharoenwase < > sorawee.pwase at gmail.com> wrote: > >> Two more bugs: >> - https://github.com/dafny-lang/dafny/issues/1396 >> - https://github.com/dafny-lang/dafny/issues/1395 >> >> The latter, again, is not discovered by the fuzzer, but by trying to >> formalizing it (well, I still couldn't implement it either -- see the >> original thread for the problem and question). >> >> On Thu, Aug 26, 2021 at 4:49 AM Sorawee Porncharoenwase < >> sorawee.pwase at gmail.com> wrote: >> >>> Here's another bug: https://github.com/dafny-lang/dafny/issues/1387 >>> >>> On Tue, Aug 24, 2021 at 2:40 PM Sorawee Porncharoenwase < >>> sorawee.pwase at gmail.com> wrote: >>> >>>> One more bug: >>>> >>>> https://github.com/dafny-lang/dafny/issues/1384 >>>> >>>> On Sun, Aug 22, 2021 at 5:34 PM Sorawee Porncharoenwase < >>>> sorawee.pwase at gmail.com> wrote: >>>> >>>>> More bugs! >>>>> >>>>> - >>>>> https://github.com/dafny-lang/dafny/issues/1372#issuecomment-903359423 >>>>> - https://github.com/dafny-lang/dafny/issues/1374 >>>>> >>>>> On Sun, Aug 22, 2021 at 12:56 PM John Regehr >>>>> wrote: >>>>> >>>>>> I'm just sort of lurking here, but I've really been enjoying this >>>>>> thread. Great work everyone!!! >>>>>> >>>>>> > The latter one is not discovered by fuzzing itself. I was writing a >>>>>> type >>>>>> > constraint to be put in the fuzzer, and thinking what will happen >>>>>> if we >>>>>> > violate the constraint, so I tried it out manually and discovered >>>>>> the issue. >>>>>> >>>>>> In my experience this happens a lot :) >>>>>> >>>>>> John >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sun Aug 29 20:44:36 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 29 Aug 2021 19:44:36 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Nope, no global variables. I?m not sure if the ability to exclude types will help in this case. Consider the following pseudocode function f(x : int): int { len(list(hole, hole)) } method main() { declare a: array(); declare b: array(); print list(a, b); } The problem occurs when hole has type array, because there is no lifting target. But also consider: method main() { declare a: array(); declare b: array(); function f(x : int): int { len(list(hole, hole)) }; print f(1); } In this case, hole can be a variable referencing a and b. And also: function f(x : array): int { len(list(hole, hole)) } method main() { } In this case, hole can be a variable referencing x. It sounds like you are suggesting that I exclude array. But what does that mean? I definitely don?t want to exclude a list from having an array as an element, because that works just fine in method, and even in functions that do have an access to an array variable. If we don?t care about completeness, I guess we can have two variants of Expression, where one is to be used in method, and another is to be used in function. Expression that is to be used in function would not allow array anywhere at all. E.g., an element of a list cannot be array. But that sounds really not ideal. I think I have a temporary solution, but it?s really hacky. I will disallow top-level functions in the grammar. Because the Main method has a list of declarations at the top, I can post-process them by randomly picking some lambdas and lifting them up to top-level functions, as long as they don?t transitively need an array declaration. On Fri, Aug 27, 2021 at 5:29 PM William G Hatch wrote: > As hinted in my other reply, the best answer would be that Xsmith type > variables have a list of types that they can't be in addition to their > list of possible types. Then you could test whether it is possibly, > definitely, or definitely not a given thing, and if you make a choice > that it isn't something you can specify just that without removing > other flexibility later. But since that feature still doesn't exist, > I don't have a great answer. Sometimes in situations like this I punt > to concretizing the type. It removes flexibility for later choices, > but lets me make a clear decision about the type now (without forcing > all unconstrained variables into the same choice). > > On Fri, Aug 27, 2021 at 02:25:38PM -0700, Sorawee Porncharoenwase wrote: > >So, this is actually another question that I meant to ask. I have this > code > >as a choice method for choice-filters-to-apply: > > > >[VariableReference > > (? () > > (define t ($xsmith_type (current-hole))) > > (define method? (can-unify? t (method-type (fresh-type-variable) > >(fresh-type-variable)))) > > (cond > > [method? > > (and (parent-node (current-hole)) > > (equal? (ast-node-type (parent-node (current-hole))) > > 'ProcedureApplicationForMethod))] > > ;; we are generating default return values. > > ;; at this point, there's no lifting target, so we can't have > > ;; variable reference > > [(and (parent-node (current-hole)) > > (equal? (ast-node-type (parent-node (current-hole))) > > 'TopLevelMethod)) > > #f] > > [else #t]))] > > > >I raised a concern earlier that can-unify? is not ideal because > > > >($xsmith_type (current-hole)) > > > >is sometimes unconstrained, meaning it rejects more than it should, and > you > >agree with that. What?s the right way to do this properly? > > > > > >On Fri, Aug 27, 2021 at 2:10 PM William G Hatch > wrote: > > > >> In Dafny do function expressions not generate closures? > >> > >> My first thought is that you can add a choice filter (with > >> choice-filters-to-apply, defined probably with add-choice-method) that > >> disallows expressions that need an array type unless there is a > >> variable already visible or its in a context where it can lift one. > >> > >> > function expression (which can't have any statement) > >> > >> As an aside, I hate languages that do this... > >> > >> On Fri, Aug 27, 2021 at 02:33:59AM -0700, Sorawee Porncharoenwase wrote: > >> >I feel like this should be easy, but somehow the solution eludes me... > >> > > >> >An array in Dafny must be allocated and then stored in a variable as a > >> part > >> >of the array declaration statement (essentially, there's no array > >> literal). > >> >The variable can then be used freely in any expression position. > However, > >> >the fact that the allocation is a statement means that in function > >> >expression (which can't have any statement), the only way to get an > array > >> >value in is through parameters. In particular, there is no lifting > target > >> >of array type inside function expressions. > >> > > >> >How should I deal with this situation? Is it possible to create a new > kind > >> >of variable reference that "tries" the array type, and use stuff like > >> >xsmith_get-reference-for-child! to see if there is a lifting target, > and > >> if > >> >there's none, then try other types (say, integer) that are guaranteed > to > >> >have a lifting target instead? That doesn't seem possible since the > type > >> of > >> >the node seems already predetermined by the time fresh is invoked. > >> > > >> >On Mon, Aug 23, 2021 at 10:06 AM William G Hatch > >> wrote: > >> > > >> >> I agree this is too bad. But I'm sorry to say that I'm not going to > >> >> improve this right now. It would require fairly major work in the > >> >> type unification machinery. The workaround is maybe not the best, > but > >> >> it's at least pretty easy to do. > >> >> > >> >> On Mon, Aug 23, 2021 at 12:12:38AM -0700, Sorawee Porncharoenwase > wrote: > >> >> >I want to express a type constraint that it?s either a product type > of > >> >> >length 0, 2, or 3. Is there a way to do that? fresh-type-variable > with > >> >> >arguments seems like a solution, except that the composite type > isn?t > >> >> >allowed to be repeated. Usually, if I have a generic type of the > same > >> >> >arity, I can simply push fresh-type-variable in. E.g., > >> >> > > >> >> >(fresh-type-variable > >> >> > (immutable (array-type ...)) > >> >> > (immutable (set-type ...))) > >> >> > > >> >> >=> > >> >> > > >> >> >(immutable (fresh-type-variable (array-type ...) (set-type ...))) > >> >> > > >> >> >The problem is that product-type?s argument is a list of types of > >> false, > >> >> so > >> >> >fresh-type-variable doesn?t work there. > >> >> > > >> >> >I guess I can create generic types product0, product2, and product3, > >> and > >> >> >put them in fresh-type-variable, but well, I really want to avoid > doing > >> >> >this. > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> >On Fri, Jul 30, 2021 at 5:35 PM Sorawee Porncharoenwase < > >> >> >sorawee.pwase at gmail.com> wrote: > >> >> > > >> >> >> As always, thanks! :) > >> >> >> > >> >> >> On Fri, Jul 30, 2021 at 5:29 PM William G Hatch > > >> >> wrote: > >> >> >> > >> >> >>> On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase > >> >> wrote: > >> >> >>> >> When I've had this situation, as a workaround, I've just made > a > >> >> >>> >> function that returns a fresh-type-variable with all of the > >> >> supported > >> >> >>> >> base types and maybe some constructed types. It's... not > great. > >> >> >>> >> > >> >> >>> > > >> >> >>> >My question is, say, I want to disallow a function, a list of > >> >> functions, > >> >> >>> a > >> >> >>> >list of list of functions, a list of list of list of functions, > >> and so > >> >> >>> on. > >> >> >>> >Of course, I can express things positively: it's an int or bool > or > >> >> >>> >listof... what? A list of fresh-type-variable runs a risk of > >> choosing > >> >> >>> >function type. So I could do a listof (int or bool or listof > ...) > >> for > >> >> >>> >another level, and keep increasing levels. Is this what you > have in > >> >> mind? > >> >> >>> > >> >> >>> Yeah, that's the problem with it. You can realistically only > >> specify > >> >> >>> a small set of the infinite set of constructible types (granted > in > >> >> >>> practice the set of types usable by xsmith is bounded as well > since > >> it > >> >> >>> bounds the tree depth of constructed types). But specifying that > >> list > >> >> >>> is a chore and you probably get poor performance if you actually > try > >> >> >>> to generate many combinations with constructed types. So in > >> practice > >> >> >>> when I have this situation so far I just... don't generate as > many > >> >> >>> things as I otherwise could. > >> >> >>> > >> >> >>> >> However, that feature just hasn't made it to the top of my > >> >> priorities > >> >> >>> >> yet... Sorry :( > >> >> >>> >> > >> >> >>> > > >> >> >>> >No worries at all! If there's anything that I can help regarding > >> this > >> >> >>> >feature, let me know. It looks like the representation of a type > >> right > >> >> >>> now > >> >> >>> >consists of a lower bound and an upper bound, and probably this > >> >> feature > >> >> >>> >will require changing the representation, correct? > >> >> >>> > >> >> >>> The upper and lower bounds are about subtyping. If a type > changes > >> >> >>> through unification it can affect the types connected as > upper/lower > >> >> >>> bounds. This feature shouldn't change that, other than being > more > >> >> >>> information that will be conveyed to the related type variables. > >> When > >> >> >>> a variable is constrained by unification to remove a type from > its > >> >> >>> allowed list, all related types in its upper/lower bounds lists > also > >> >> >>> remove that type. Eg. if variable A is (or string int (-> int > int)) > >> >> >>> and variable B is (or (subtype-of string) (subtype-of int) > >> (subtype-of > >> >> >>> (-> int int))) and variable A is constrained to definitely not > be a > >> >> >>> function, variable B could no longer be a function and still be a > >> >> >>> subtype of A. If two variables are unified with `unify!` > instead of > >> >> >>> `subtype-unify!` (or equivalently by being `subtype-unify!`ed in > >> both > >> >> >>> directions) they are more or less merged into a single variable. > >> (IE > >> >> >>> the object that you get from `fresh-type-variable` can be merged > >> with > >> >> >>> others and they are always canonicalized before doing anything to > >> >> >>> them.) > >> >> >>> > >> >> >>> Right now a type variable has a list of possible types (or #f for > >> >> >>> unconstrained). The list can have any number of base types and > >> every > >> >> >>> compound type, but it's limited in that it can only have one of > each > >> >> >>> compound type (though each compound type may have type variables > >> >> >>> inside and thus represent potentially many types). > >> >> >>> > >> >> >>> I think the needed change would be to add a list of disallowed > >> types. > >> >> >>> I think this list should probably be applied recursively -- IE it > >> >> >>> should be pushed to child types. So eg. if you disallow function > >> >> >>> types in a type variable it also disallows lists of functions, > etc. > >> >> >>> I'm not certain if there are exceptions, though. Would it make > >> sense > >> >> >>> to have a recursive and non-recursive version of this? > >> >> >>> > >> >> >>> At any rate, when checking if things `can-unify?`, you would use > the > >> >> >>> disallowed list as well as the allowed list of both variables to > >> >> >>> determine the outcome. When variables are unified, they need to > >> >> >>> propagate their disallowed list to related type variables (IE the > >> ones > >> >> >>> in the upper/lower bounds lists). > >> >> >>> > >> >> >>> Another thing that would need to be worked out is what exactly to > >> >> >>> store in the disallowed list. Constructors for compound types? > Or > >> >> >>> maybe predicates? Probably base types and predicates is my > guess. > >> >> >>> > >> >> >>> At any rate, since we keep emailing about the issue I've thought > >> >> >>> through more of it than before, so now I'm much more likely to > >> >> >>> actually sit down and do it soon. :P > >> >> >>> > >> >> >> > >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: