From sorawee.pwase at gmail.com Sun Jul 11 15:42:49 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 11 Jul 2021 14:42:49 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Hi everyone! I have a couple more questions: 1) Is there a support for multiple bindings, just like define-values in Scheme? 2) I?m adding first-order functions whose call is limited in some context. The special call is approximated by how MyDefinition only allows ProcedureApplication as the RHS in the following code, which is added to the somelisp example: (add-to-grammar somelisp [MyProgram #f ([decls : Definition *] [body : SubProgram]) #:prop type-info [(fresh-type-variable) (? (n t) (hash 'decls (? (c) (fresh-type-variable)) 'body (fresh-type-variable)))]] [SubProgram #f ([decls : MyDefinition *] [body : Expression]) #:prop type-info [(fresh-type-variable) (? (n t) (hash 'decls (? (c) (fresh-type-variable)) 'body (fresh-type-variable)))]] [MyDefinition #f ([type] [name = (fresh-var-name "b_")] [e : ProcedureApplication]) #:prop binder-info () #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) (add-property somelisp lift-type->ast-binder-type [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) with MyProgram as the top-level node. (I structured the grammar like that so that the RHS of MyDefinition can be lifted to Definition. I previously tried lumping Definition and MyDefinition together in MyProgram and got an error _xsmith_resolve-reference: Unbound reference: lift_2 ? not sure what it means). This fails in two ways: - Sometimes I get an error: RACR exception: "AG evaluator exception;" "Cannot access" 'xsmith_no-holes-in-subtree? "attribute - the given node is a bud.". Indeed, when I use --s-exp-on-error, I can see that it?s a bud node. But this is weird because if I specify Expression, a ProcedureApplication can be generated just fine. - Sometimes I get an error: xsmith: internal error -- no destinations for lift from: XsmithAstHoleExpression,. In the last email, I asked if lift-type->ast-binder-type could be used nondeterministically in this manner, and my experience with it so far seems to indicate no. Perhaps a better example to illustrate this point is to first adjust MyDefinition to allow an arbitrary Expression as the RHS. Deterministically returning either MyDefinition or Definition will successfully generate the program, but that?s not what I want ? I want both to be generated. However, randomizing will result in the above error. Does that mean lift-type->ast-binder-type is not the right tool for me (though I am required to specify it, as there are multiple binder-info?)? Thanks, Sorawee On Wed, Jun 23, 2021 at 3:27 PM William G Hatch wrote: > On Wed, Jun 23, 2021 at 01:36:08PM -0700, Sorawee Porncharoenwase wrote: > >Thanks for your help! > > > >>Every node in the AST has type information. > >> > >Sorry, I should have framed the question better. > > > >Every node has type information, but IIUC it is annotated externally, > >right? What I?m curious about is why Definition requires an explicit type > >field in the AST node? Can?t the type information be stored externally as > >an attribute? And my question was that if there?s a reason this can?t be > >done, are there any other similar situations too? (That is, it will > require > >an explicit type field in the AST node). > > TL;DR: Storing types on other nodes is generally not necessary. It's > done for definitions because definitions/references is where code > relationships go from being a tree to being a more complicated graph, > while other nodes don't have that problem. > > One reason we store a type in a definition node is because most > definitions are made by lifting, and thus have a specific type needed > before creating the right-hand-side of the definition. If we didn't > store the type in that case, we would need to search for variable > use-sites when computing a type later. Also, for variables that can > be mutated (or that hold containers), it can be important to have the > precise type for creating assignment nodes. Eg. in the face of > subtyping, assignment needs to have an invariant type relationship. > And it's just easier to go about that stuff if the exact type is > always explicitly stored rather than needing to compute it. If we > didn't store the type, it could remain only partially constrained for > a long time, and potentially be used for mutation in multiple places, > and basically it would just make the type analysis a lot more > complicated. Another reason is for the RACR cache. Computing a > definition's type would require looking at its references, which would > make the dependency graph on the type attribute a lot more connected > and tangled, meaning the caches would be flushed a lot more frequently > and require more recomputation. > > Generally, when you are making your own node types, you need to store > type information only when you make a type decision that's not fully > determined by the node's parent and child types. Eg. there may be > cases where legally you can have either of type A or B, but you need > or want to make a decision that requires some data beyond what the > children have or before generating children. I think this came up in > a fuzzer that had type annotations in the printer (or maybe it was > that there were two different functions/operations but we encoded them > as one node where the printing just depended on the type... It's in > the WASM fuzzer, which I haven't had as much part in so I don't > remember the details), but where some types at print time were still > not constrained to a single printable type. We had to choose one to > print, and then make that choice consistent. But I think at that > point it wasn't actually necessary to store the choice in the node, > because printing is the last thing that's ever done to the AST and > there just wasn't another opportunity to make another choice. But if > you do need to make a choice about types that won't be consistently > computed in the future by the normal type checking apparatus, then you > need to store the type choice and unify with it. I think it shouldn't > be necessary. None of my fuzzers store type fields in anything but > name binding nodes. > > >>The key to using Definitions in a statement language is basically a > >> >`Block`-like statement that can have a series of statements that > >> >include definitions needs to have a list of definitions and a list of > >> >statements separately. This is a bit of a limitation, since many > >> >languages can interleave definitions and statements, and Xsmith > >> >basically can't. At any rate, the `Block` node from canned-components > >> >does this. > >> > >Ah, I see. Perhaps one can hack by adding another subtype of Block node > >that, when rendering, will splice its body out. That would give an > >appearance of interleaving statements and declarations, though I think > >there could be a problem with variable scope -- if there's a variable > >shadowing in a block, splicing the body out could create a redefinition, > >which might be invalid in some languages. > > Yes, exactly. That said, in practice xsmith just chooses a fresh name > for every variable, so unless/until we change that to actually produce > duplicate variable names, it will never be an issue. > > >>I hope you find Xsmith useful! What are you intending to fuzz? I'm > >> >also happy to look at any code you have on Github or such for some > >> >debugging help. > >> > >We are fuzzing Dafny compilers. > > Cool! Let me know if you put your fuzzer on github or something. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Tue Jul 13 15:11:13 2021 From: william at hatch.uno (William G Hatch) Date: Tue, 13 Jul 2021 15:11:13 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Hello again. >1) Is there a support for multiple bindings, just like define-values in >Scheme? No, but you can maybe simulate this to some degree. In canned-components there is a DefinitionNoRhs node that I made for adding definition nodes for things like loops and list comprehensions. The idea is that you can use it in a node that introduces a binding where the RHS is some implicit operation on some other piece of data the node has (eg. a list expression). You could make a subclass of Block that has a field for multi-value-RHSs, insert DefinitionNoRhs nodes in the definitions field, and keep track of the relationship somehow. It might be a bit of work, but I think it should be doable. Ideally this is something that could be added to canned-components, but I don't see myself writing it at the moment. >2) I?m adding first-order functions whose call is limited in some context. >The special call is approximated by how MyDefinition only allows >ProcedureApplication as the RHS in the following code Hmm, it seems like first-order functions would be encoded as a function application node that only allows a reference as its RHS rather than allowing arbitrary expressions. But that said, I'm not sure I fully understand what you're trying to do. >I previously tried lumping Definition and MyDefinition >together in MyProgram and got an error _xsmith_resolve-reference: Unbound >reference: lift_2 ? not sure what it means). That's probably a bug. ?Lifting? is what I've called the situation where a reference is generated that causes a new definition to be added such that the variable is in scope at the use site. IE the binding is lifted from the use site to some outer node that contains definitions. `lift_X` is the name Xsmith gives to definitions that are created automatically in this way (which is ultimately most definitions). The exception is caused because somehow the lift ended up in a place where it wasn't actually in scope. That shouldn't happen. Anyway, if you point me to a version of a fuzzer that does this, I'll try to debug it. >This fails in two ways: > > - Sometimes I get an error: RACR exception: "AG evaluator exception;" > "Cannot access" 'xsmith_no-holes-in-subtree? "attribute - the given node is > a bud.". Indeed, when I use --s-exp-on-error, I can see that it?s a bud > node. But this is weird because if I specify Expression, a > ProcedureApplication can be generated just fine. Probably something is going wrong in the `fresh` property. Have you changed that? > - Sometimes I get an error: xsmith: internal error -- no destinations > for lift from: XsmithAstHoleExpression,. I should change that error message. Anyway, it seems to think there is no outer node that has a `Definition *` child that can be the target for lifting a definition. Generally this happens when you forget to have an outer Program node or something that can hold definitions. Could you point me to a complete example that I can run that has these problems? I want to be able to run the code to see the errors and debug them, but the provided snippet isn't quite enough. Thanks, William Hatch On Sun, Jul 11, 2021 at 02:42:49PM -0700, Sorawee Porncharoenwase wrote: >Hi everyone! > >I have a couple more questions: > >1) Is there a support for multiple bindings, just like define-values in >Scheme? >2) I?m adding first-order functions whose call is limited in some context. >The special call is approximated by how MyDefinition only allows >ProcedureApplication as the RHS in the following code, which is added to >the somelisp example: > >(add-to-grammar > somelisp > [MyProgram #f > ([decls : Definition *] > [body : SubProgram]) > #:prop type-info > [(fresh-type-variable) > (? (n t) > (hash 'decls (? (c) (fresh-type-variable)) > 'body (fresh-type-variable)))]] > [SubProgram #f > ([decls : MyDefinition *] > [body : Expression]) > #:prop type-info > [(fresh-type-variable) > (? (n t) > (hash 'decls (? (c) (fresh-type-variable)) > 'body (fresh-type-variable)))]] > [MyDefinition #f ([type] > [name = (fresh-var-name "b_")] > [e : ProcedureApplication]) > #:prop binder-info () > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) > >(add-property > somelisp > lift-type->ast-binder-type > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >with MyProgram as the top-level node. > >(I structured the grammar like that so that the RHS of MyDefinition can be >lifted to Definition. I previously tried lumping Definition and MyDefinition >together in MyProgram and got an error _xsmith_resolve-reference: Unbound >reference: lift_2 ? not sure what it means). > >This fails in two ways: > > - Sometimes I get an error: RACR exception: "AG evaluator exception;" > "Cannot access" 'xsmith_no-holes-in-subtree? "attribute - the given node is > a bud.". Indeed, when I use --s-exp-on-error, I can see that it?s a bud > node. But this is weird because if I specify Expression, a > ProcedureApplication can be generated just fine. > - Sometimes I get an error: xsmith: internal error -- no destinations > for lift from: XsmithAstHoleExpression,. > >In the last email, I asked if lift-type->ast-binder-type could be used >nondeterministically in this manner, and my experience with it so far seems >to indicate no. Perhaps a better example to illustrate this point is to >first adjust MyDefinition to allow an arbitrary Expression as the RHS. >Deterministically returning either MyDefinition or Definition will >successfully generate the program, but that?s not what I want ? I want both >to be generated. However, randomizing will result in the above error. Does >that mean lift-type->ast-binder-type is not the right tool for me (though I >am required to specify it, as there are multiple binder-info?)? > >Thanks, >Sorawee > >On Wed, Jun 23, 2021 at 3:27 PM William G Hatch wrote: > >> On Wed, Jun 23, 2021 at 01:36:08PM -0700, Sorawee Porncharoenwase wrote: >> >Thanks for your help! >> > >> >>Every node in the AST has type information. >> >> >> >Sorry, I should have framed the question better. >> > >> >Every node has type information, but IIUC it is annotated externally, >> >right? What I?m curious about is why Definition requires an explicit type >> >field in the AST node? Can?t the type information be stored externally as >> >an attribute? And my question was that if there?s a reason this can?t be >> >done, are there any other similar situations too? (That is, it will >> require >> >an explicit type field in the AST node). >> >> TL;DR: Storing types on other nodes is generally not necessary. It's >> done for definitions because definitions/references is where code >> relationships go from being a tree to being a more complicated graph, >> while other nodes don't have that problem. >> >> One reason we store a type in a definition node is because most >> definitions are made by lifting, and thus have a specific type needed >> before creating the right-hand-side of the definition. If we didn't >> store the type in that case, we would need to search for variable >> use-sites when computing a type later. Also, for variables that can >> be mutated (or that hold containers), it can be important to have the >> precise type for creating assignment nodes. Eg. in the face of >> subtyping, assignment needs to have an invariant type relationship. >> And it's just easier to go about that stuff if the exact type is >> always explicitly stored rather than needing to compute it. If we >> didn't store the type, it could remain only partially constrained for >> a long time, and potentially be used for mutation in multiple places, >> and basically it would just make the type analysis a lot more >> complicated. Another reason is for the RACR cache. Computing a >> definition's type would require looking at its references, which would >> make the dependency graph on the type attribute a lot more connected >> and tangled, meaning the caches would be flushed a lot more frequently >> and require more recomputation. >> >> Generally, when you are making your own node types, you need to store >> type information only when you make a type decision that's not fully >> determined by the node's parent and child types. Eg. there may be >> cases where legally you can have either of type A or B, but you need >> or want to make a decision that requires some data beyond what the >> children have or before generating children. I think this came up in >> a fuzzer that had type annotations in the printer (or maybe it was >> that there were two different functions/operations but we encoded them >> as one node where the printing just depended on the type... It's in >> the WASM fuzzer, which I haven't had as much part in so I don't >> remember the details), but where some types at print time were still >> not constrained to a single printable type. We had to choose one to >> print, and then make that choice consistent. But I think at that >> point it wasn't actually necessary to store the choice in the node, >> because printing is the last thing that's ever done to the AST and >> there just wasn't another opportunity to make another choice. But if >> you do need to make a choice about types that won't be consistently >> computed in the future by the normal type checking apparatus, then you >> need to store the type choice and unify with it. I think it shouldn't >> be necessary. None of my fuzzers store type fields in anything but >> name binding nodes. >> >> >>The key to using Definitions in a statement language is basically a >> >> >`Block`-like statement that can have a series of statements that >> >> >include definitions needs to have a list of definitions and a list of >> >> >statements separately. This is a bit of a limitation, since many >> >> >languages can interleave definitions and statements, and Xsmith >> >> >basically can't. At any rate, the `Block` node from canned-components >> >> >does this. >> >> >> >Ah, I see. Perhaps one can hack by adding another subtype of Block node >> >that, when rendering, will splice its body out. That would give an >> >appearance of interleaving statements and declarations, though I think >> >there could be a problem with variable scope -- if there's a variable >> >shadowing in a block, splicing the body out could create a redefinition, >> >which might be invalid in some languages. >> >> Yes, exactly. That said, in practice xsmith just chooses a fresh name >> for every variable, so unless/until we change that to actually produce >> duplicate variable names, it will never be an issue. >> >> >>I hope you find Xsmith useful! What are you intending to fuzz? I'm >> >> >also happy to look at any code you have on Github or such for some >> >> >debugging help. >> >> >> >We are fuzzing Dafny compilers. >> >> Cool! Let me know if you put your fuzzer on github or something. >> From sorawee.pwase at gmail.com Tue Jul 13 15:55:10 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 13 Jul 2021 14:55:10 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: > > Hmm, it seems like first-order functions would be encoded as a > function application node that only allows a reference as its RHS > rather than allowing arbitrary expressions. But that said, I'm not > sure I fully understand what you're trying to do. > I?m not sure what you mean by that. But: 1. I have no problem with the definition site. That?s easy to deal with. 2. I also have no problem with restricting function application. That?s easy to deal with as well. 3. The problem is primarily with a bare variable reference bound to a closure value, which must only occur in function application. For example, the following code is invalid. (define (f) (void)) (define g f) ;; this is invalid (define h (g)) (define a (list f)) ;; this is also invalid (define b (f)) ;; this is ok I have been using choice-filters-to-apply to restrict variable references whose type can-unify? with function-type that doesn't directly appear under function application. But this is obviously not ideal because the filtered nodes are mostly those whose type is totally unrestricted. So filtering them would prevent them from being concretized to other non-function types as well. (Also note that I tried unify! these unrestricted type variables with some concrete non-function type variables, but down the road, I get an internal error about unification failure.). Perhaps I'm not supposed to unify! inside add-choice-method? That's probably a bug. ?Lifting? is what I've called the situation > where a reference is generated that causes a new definition to be > added such that the variable is in scope at the use site. IE the > binding is lifted from the use site to some outer node that contains > definitions. `lift_X` is the name Xsmith gives to definitions that > are created automatically in this way (which is ultimately most > definitions). The exception is caused because somehow the lift ended > up in a place where it wasn't actually in scope. That shouldn't > happen. Anyway, if you point me to a version of a fuzzer that does > this, I'll try to debug it. > #lang clotho (require xsmith racr xsmith/racr-convenience xsmith/canned-components racket/string racket/list racket/pretty) ;; We first define a basic component and add a bunch of expressions. (define-basic-spec-component somelisp) (add-basic-expressions somelisp #:ProgramWithSequence #t #:VoidExpression #t #:AssignmentExpression #t #:VariableReference #t #:ProcedureApplication #t #:IfExpression #t #:ExpressionSequence #t #:LetSequential #t #:LambdaWithExpression #t #:Numbers #t #:Booleans #t #:Strings #t #:ImmutableList #t) (add-loop-over-container somelisp #:name Map #:collection-type-constructor (? (inner) (immutable (list-type inner))) #:loop-type-constructor (? (inner) (immutable (list-type inner)))) (add-to-grammar somelisp [MyProgram #f ([decls : Definition *] [declsTwo : MyDefinition *] [body : Expression]) #:prop type-info [(fresh-type-variable) (? (n t) (hash 'decls (? (c) (fresh-type-variable)) 'declsTwo (? (c) (fresh-type-variable)) 'body (fresh-type-variable)))]] [MyDefinition #f ([type] [name = (fresh-var-name "b_")] [e : ProcedureApplication]) #:prop binder-info () #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) (add-property somelisp lift-type->ast-binder-type [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) ;; Sometimes we have a type variable that is not concrete but needs to be. ;; Here we provide a list of options we can pick for an unconstrained ;; type variable. (define (type-thunks-for-concretization) (list (?()int-type) (?()bool-type) (?()string-type) (?()(immutable (list-type (fresh-type-variable)))))) (define (somelisp-format-render s-exps) (define out (open-output-string)) (for ([symex s-exps]) (pretty-print symex out 1)) (get-output-string out)) (define-xsmith-interface-functions [somelisp] #:program-node MyProgram #:type-thunks type-thunks-for-concretization #:comment-wrap (? (lines) (string-join (map (? (l) (format ";; ~a" l)) lines) "\n")) #:format-render somelisp-format-render) (module+ main (somelisp-command-line)) > Probably something is going wrong in the `fresh` property. Have you > changed that? > I don?t think so. See the code at the end of the email. > I should change that error message. Anyway, it seems to think there > is no outer node that has a `Definition *` child that can be the > target for lifting a definition. Generally this happens when you > forget to have an outer Program node or something that can hold > definitions. > I don?t think that?s the case, but I could be wrong. #lang clotho (require xsmith racr xsmith/racr-convenience xsmith/canned-components racket/string racket/list racket/pretty) ;; We first define a basic component and add a bunch of expressions. (define-basic-spec-component somelisp) (add-basic-expressions somelisp #:ProgramWithSequence #t #:VoidExpression #t #:AssignmentExpression #t #:VariableReference #t #:ProcedureApplication #t #:IfExpression #t #:ExpressionSequence #t #:LetSequential #t #:LambdaWithExpression #t #:Numbers #t #:Booleans #t #:Strings #t #:ImmutableList #t) (add-loop-over-container somelisp #:name Map #:collection-type-constructor (? (inner) (immutable (list-type inner))) #:loop-type-constructor (? (inner) (immutable (list-type inner)))) (add-to-grammar somelisp [MyProgram #f ([decls : Definition *] [body : SubProgram]) #:prop type-info [(fresh-type-variable) (? (n t) (hash 'decls (? (c) (fresh-type-variable)) 'body (fresh-type-variable)))]] [SubProgram #f ([decls : MyDefinition *] [body : Expression]) #:prop type-info [(fresh-type-variable) (? (n t) (hash 'decls (? (c) (fresh-type-variable)) 'body (fresh-type-variable)))]] [MyDefinition #f ([type] [name = (fresh-var-name "b_")] [e : ProcedureApplication]) #:prop binder-info () #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) (add-property somelisp lift-type->ast-binder-type [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) ;; Sometimes we have a type variable that is not concrete but needs to be. ;; Here we provide a list of options we can pick for an unconstrained ;; type variable. (define (type-thunks-for-concretization) (list (?()int-type) (?()bool-type) (?()string-type) (?()(immutable (list-type (fresh-type-variable)))))) (define (somelisp-format-render s-exps) (define out (open-output-string)) (for ([symex s-exps]) (pretty-print symex out 1)) (get-output-string out)) (define-xsmith-interface-functions [somelisp] #:program-node MyProgram #:type-thunks type-thunks-for-concretization #:comment-wrap (? (lines) (string-join (map (? (l) (format ";; ~a" l)) lines) "\n")) #:format-render somelisp-format-render) (module+ main (somelisp-command-line)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmatos at linki.tools Thu Jul 15 01:11:27 2021 From: pmatos at linki.tools (Paulo Matos) Date: Thu, 15 Jul 2021 09:11:27 +0200 Subject: [xsmith-dev] WebAssembly fuzzer Message-ID: <87y2a8gig0.fsf@linki.tools> Hi, Only just managed to catch up to Eric Eide presentation in Racketfest where he mentions not just a JS fuzzer but also a WebAssembly fuzzer. This is something I am working on as part of my day job and could possibly spend some time improving it. I am assuming the JS fuzzer is: https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/simple/javascript.rkt However, I cannot find any reference to a Wasm fuzzer. A reference to it would be great, thanks. -- Paulo Matos From eeide at cs.utah.edu Thu Jul 15 08:09:33 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Thu, 15 Jul 2021 08:09:33 -0600 Subject: [xsmith-dev] WebAssembly fuzzer In-Reply-To: <87y2a8gig0.fsf@linki.tools> (Paulo Matos's message of "Thu, 15 Jul 2021 09:11:27 +0200") References: <87y2a8gig0.fsf@linki.tools> Message-ID: Paulo Matos writes: > Only just managed to catch up to Eric Eide presentation in Racketfest > where he mentions not just a JS fuzzer but also a WebAssembly fuzzer. The Wasm fuzzer is here: https://gitlab.flux.utah.edu/xsmith/webassembly-sandbox This is being developed by Guy Watson (who is subscribed to this mailing list) as part of his MS thesis work at the University of Utah. I am sure Guy would appreciate any feedback you have! If you find bugs with this fuzzer, please let us know! We are tracking bugs discovered via Xsmith-based fuzzers. E.g., give us a pointer to the PR. Eric. -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From eeide at cs.utah.edu Thu Jul 15 08:10:42 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Thu, 15 Jul 2021 08:10:42 -0600 Subject: [xsmith-dev] JavaScript Fuzzer (Re: WebAssembly fuzzer) In-Reply-To: <87y2a8gig0.fsf@linki.tools> (Paulo Matos's message of "Thu, 15 Jul 2021 09:11:27 +0200") References: <87y2a8gig0.fsf@linki.tools> Message-ID: Paulo Matos writes: > I am assuming the JS fuzzer is: > https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/simple/javascript.rkt Yes, that's the JS fuzzer. -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From william at hatch.uno Thu Jul 15 10:21:11 2021 From: william at hatch.uno (William G Hatch) Date: Thu, 15 Jul 2021 10:21:11 -0600 Subject: [xsmith-dev] WebAssembly fuzzer In-Reply-To: <87y2a8gig0.fsf@linki.tools> References: <87y2a8gig0.fsf@linki.tools> Message-ID: I feel I should point out that the JS fuzzer hasn't really been tested. We haven't set up a Javascript fuzzing environment yet, and we haven't run a lot of the generated programs. So it may spit out syntax errors or something still that we haven't noticed. That said, it should be close. If there are serious bugs in code generation (aside from any bugs that may be in the Xsmith library) they are likely in the printer. I would like to get some setup scripts and harness configuration to fuzz Javascript soon, but if left to me I'm not sure when I'll do that vs other things I'm working on. On Thu, Jul 15, 2021 at 09:11:27AM +0200, Paulo Matos wrote: >Hi, > >Only just managed to catch up to Eric Eide presentation in Racketfest >where he mentions not just a JS fuzzer but also a WebAssembly fuzzer. > >This is something I am working on as part of my day job and could >possibly spend some time improving it. > >I am assuming the JS fuzzer is: >https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/simple/javascript.rkt > >However, I cannot find any reference to a Wasm fuzzer. A reference to it >would be great, thanks. > >-- >Paulo Matos From william at hatch.uno Thu Jul 15 20:59:47 2021 From: william at hatch.uno (William G Hatch) Date: Thu, 15 Jul 2021 20:59:47 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Sorry for the delay. I wanted to do the debugging before responding, but then I didn't get to it for a while. Anyway, I now see what's going wrong, and I've fixed some bugs. First, that issue with the bud-node was a bug in the `edit` property which the canned-components `ProcedureApplication` uses. It's fixed now. >> Hmm, it seems like first-order functions would be encoded as a >> function application node that only allows a reference as its RHS >> rather than allowing arbitrary expressions. But that said, I'm not >> sure I fully understand what you're trying to do. >> >I?m not sure what you mean by that. But: Erm, I misspoke (mistyped) a bit. I meant that it would be function application that only allows a reference in the procedure position. I don't know why I wrote RHS. Anyway, the strategy of having separate definition nodes for functions and everything else is used in Cish. I don't recommend following anything Cish does. However, it accomplishes this by having the program node have non-function definitions first, and disallowing use of functions on the RHS of those functions. The problem where it can't find a reference is a bug (that I haven't fixed yet) caused by the lifting mechanics not really being completely in sync with the reference mechanics with respect to nodes that have multiple definition fields. When there are multiple definition fields, definitions in the lower field can reference the definitions in the higher field, but not vice versa. The lifting machinery doesn't know about that ordering, it just stuffs the lifted definition into the front of the definitions in whatever field it is targetting. Thus it places a definition in a place that the reference machinery thinks is out of scope. So... if you want to have different binders that get different fields in the program node you can only make it work if you disallow stuff to be sure you never need to reference something that will be lifted into the lower field. That said, I've just fixed some issues to allow definition nodes to be subtypable, so now you can have a definition node that is a subtype of Definition, but has some different rules. Importantly, though, it can be lifted to the same field. Eg. if I change the MyDefinition declaration in your example to look like this: ``` [MyDefinition Definition () #:prop fresh (hash 'Expression (make-hole 'ProcedureApplication))] ``` It can now be lifted among normal definitions, but always have ProcedureApplication as its RHS. However, I'm still not convinced that you actually want ProcedureApplication as the RHS. Maybe you want to restrict lambda nodes to only be generated on the RHS of a FunctionDefinition node (that's a subtype of Definition), and make FunctionDefinition always have a Lambda node as its RHS. There is an example in the documentation that is like this, in the explanation of the `choice-filters-to-apply` property. But I'll paste it here: ``` (add-choice-method my-component no-lambda-except-global-def [#f (? () #t)] [Lambda (? () (and (parent-node current-hole) (equal? (ast-node-type (parent-node current-hole)) 'Definition) (parent-node (parent-node current-hole)) (equal? (ast-node-type (parent-node (parent-node current-hole))) 'Program)))]) (add-property my-component choice-filters-to-apply [#f (no-lambda-except-global-def)]) ``` You can use something like that to allow Lambda nodes to only be generated on the RHS of FunctionDefinition nodes, and define FunctionDefinition like MyDefinition above to always generate Lambda as its RHS, and I think it should work for a first-order language. >(Also note that I tried unify! these unrestricted type variables >with some concrete non-function type variables, but down the road, I get an >internal error about unification failure.). Perhaps I'm not supposed to >unify! inside add-choice-method? Well, probably not. Generally the only place you want to `unify!` is inside the type property in the function that determines child types based on the parent type. Inside that function you can `unify!`, but the unification needs to be consistent (IE not unified based on the result of `random`) OR the choice needs to be saved in a way that it can be looked up when that function is run again. The problem is that the choice method gets its type information based on a hole node that is later replaced, so the type is recomputed without necessarily consulting the type that the hole said it was. You should be able to constrain the type using unification if you save that type to re-unify it consistently later. But without seeing the exact error and code that generated it, I'm not 100% certain. > In the last email, I asked if lift-type->ast-binder-type could be used > nondeterministically in this manner, and my experience with it so far seems > to indicate no. I think this should work, barring the issues mentioned above about the lifting vs referencing bug when you have the different definition types in different fields. Adjusting your example to have MyDefinition as a subtype, and then only having a Definition field in MyProgram, random choice seems to work fine. But sometimes it generates gargantuan programs (that I kill before it finishes) because it keeps choosing MyDefinition, requiring a function application, which likely lifts a reference to a function, which might choose MyDefinition, etc, in a bad lift chain that generates bigger and bigger types. Anyway, does that all help? On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase wrote: >> >> Hmm, it seems like first-order functions would be encoded as a >> function application node that only allows a reference as its RHS >> rather than allowing arbitrary expressions. But that said, I'm not >> sure I fully understand what you're trying to do. >> >I?m not sure what you mean by that. But: > > 1. I have no problem with the definition site. That?s easy to deal with. > 2. I also have no problem with restricting function application. That?s > easy to deal with as well. > 3. The problem is primarily with a bare variable reference bound to a > closure value, which must only occur in function application. For example, > the following code is invalid. > >(define (f) (void)) > >(define g f) ;; this is invalid >(define h (g)) >(define a (list f)) ;; this is also invalid >(define b (f)) ;; this is ok > >I have been using choice-filters-to-apply to restrict variable references >whose type can-unify? with function-type that doesn't directly appear under >function application. But this is obviously not ideal because the filtered >nodes are mostly those whose type is totally unrestricted. So filtering >them would prevent them from being concretized to other non-function types >as well. (Also note that I tried unify! these unrestricted type variables >with some concrete non-function type variables, but down the road, I get an >internal error about unification failure.). Perhaps I'm not supposed to >unify! inside add-choice-method? > >That's probably a bug. ?Lifting? is what I've called the situation >> where a reference is generated that causes a new definition to be >> added such that the variable is in scope at the use site. IE the >> binding is lifted from the use site to some outer node that contains >> definitions. `lift_X` is the name Xsmith gives to definitions that >> are created automatically in this way (which is ultimately most >> definitions). The exception is caused because somehow the lift ended >> up in a place where it wasn't actually in scope. That shouldn't >> happen. Anyway, if you point me to a version of a fuzzer that does >> this, I'll try to debug it. >> >#lang clotho > >(require > xsmith > racr > xsmith/racr-convenience > xsmith/canned-components > racket/string > racket/list > racket/pretty) > >;; We first define a basic component and add a bunch of expressions. > >(define-basic-spec-component somelisp) > >(add-basic-expressions somelisp > #:ProgramWithSequence #t > #:VoidExpression #t > #:AssignmentExpression #t > #:VariableReference #t > #:ProcedureApplication #t > #:IfExpression #t > #:ExpressionSequence #t > #:LetSequential #t > #:LambdaWithExpression #t > #:Numbers #t > #:Booleans #t > #:Strings #t > #:ImmutableList #t) > >(add-loop-over-container > somelisp > #:name Map > #:collection-type-constructor (? (inner) (immutable (list-type inner))) > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >(add-to-grammar > somelisp > [MyProgram #f > ([decls : Definition *] > [declsTwo : MyDefinition *] > [body : Expression]) > #:prop type-info > [(fresh-type-variable) > (? (n t) > (hash 'decls (? (c) (fresh-type-variable)) > 'declsTwo (? (c) (fresh-type-variable)) > 'body (fresh-type-variable)))]] > [MyDefinition #f ([type] > [name = (fresh-var-name "b_")] > [e : ProcedureApplication]) > #:prop binder-info () > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) > >(add-property > somelisp > lift-type->ast-binder-type > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >;; Sometimes we have a type variable that is not concrete but needs to be. >;; Here we provide a list of options we can pick for an unconstrained >;; type variable. >(define (type-thunks-for-concretization) > (list > (?()int-type) > (?()bool-type) > (?()string-type) > (?()(immutable (list-type (fresh-type-variable)))))) > >(define (somelisp-format-render s-exps) > (define out (open-output-string)) > (for ([symex s-exps]) > (pretty-print symex out 1)) > (get-output-string out)) > >(define-xsmith-interface-functions > [somelisp] > #:program-node MyProgram > #:type-thunks type-thunks-for-concretization > #:comment-wrap (? (lines) > (string-join > (map (? (l) (format ";; ~a" l)) lines) > "\n")) > #:format-render somelisp-format-render) > >(module+ main (somelisp-command-line)) > > >> Probably something is going wrong in the `fresh` property. Have you >> changed that? >> >I don?t think so. See the code at the end of the email. > >> I should change that error message. Anyway, it seems to think there >> is no outer node that has a `Definition *` child that can be the >> target for lifting a definition. Generally this happens when you >> forget to have an outer Program node or something that can hold >> definitions. >> >I don?t think that?s the case, but I could be wrong. > >#lang clotho > >(require > xsmith > racr > xsmith/racr-convenience > xsmith/canned-components > racket/string > racket/list > racket/pretty) > >;; We first define a basic component and add a bunch of expressions. > >(define-basic-spec-component somelisp) > >(add-basic-expressions somelisp > #:ProgramWithSequence #t > #:VoidExpression #t > #:AssignmentExpression #t > #:VariableReference #t > #:ProcedureApplication #t > #:IfExpression #t > #:ExpressionSequence #t > #:LetSequential #t > #:LambdaWithExpression #t > #:Numbers #t > #:Booleans #t > #:Strings #t > #:ImmutableList #t) > >(add-loop-over-container > somelisp > #:name Map > #:collection-type-constructor (? (inner) (immutable (list-type inner))) > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >(add-to-grammar > somelisp > [MyProgram #f > ([decls : Definition *] > [body : SubProgram]) > #:prop type-info > [(fresh-type-variable) > (? (n t) > (hash 'decls (? (c) (fresh-type-variable)) > 'body (fresh-type-variable)))]] > [SubProgram #f > ([decls : MyDefinition *] > [body : Expression]) > #:prop type-info > [(fresh-type-variable) > (? (n t) > (hash 'decls (? (c) (fresh-type-variable)) > 'body (fresh-type-variable)))]] > [MyDefinition #f ([type] > [name = (fresh-var-name "b_")] > [e : ProcedureApplication]) > #:prop binder-info () > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e t))]]) > >(add-property > somelisp > lift-type->ast-binder-type > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >;; Sometimes we have a type variable that is not concrete but needs to be. >;; Here we provide a list of options we can pick for an unconstrained >;; type variable. >(define (type-thunks-for-concretization) > (list > (?()int-type) > (?()bool-type) > (?()string-type) > (?()(immutable (list-type (fresh-type-variable)))))) > >(define (somelisp-format-render s-exps) > (define out (open-output-string)) > (for ([symex s-exps]) > (pretty-print symex out 1)) > (get-output-string out)) > >(define-xsmith-interface-functions > [somelisp] > #:program-node MyProgram > #:type-thunks type-thunks-for-concretization > #:comment-wrap (? (lines) > (string-join > (map (? (l) (format ";; ~a" l)) lines) > "\n")) > #:format-render somelisp-format-render) > >(module+ main (somelisp-command-line)) From sorawee.pwase at gmail.com Thu Jul 15 21:09:13 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Thu, 15 Jul 2021 20:09:13 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Yes, this is very helpful. I will let you know if I encounter any further issues after updating XSmith. By the way, do you want any help with Scribble? I notice a lot of typos and link errors and can submit PRs to fix them. On Thu, Jul 15, 2021 at 8:00 PM William G Hatch wrote: > Sorry for the delay. I wanted to do the debugging before responding, > but then I didn't get to it for a while. > > Anyway, I now see what's going wrong, and I've fixed some bugs. > > First, that issue with the bud-node was a bug in the `edit` property > which the canned-components `ProcedureApplication` uses. It's fixed > now. > > >> Hmm, it seems like first-order functions would be encoded as a > >> function application node that only allows a reference as its RHS > >> rather than allowing arbitrary expressions. But that said, I'm not > >> sure I fully understand what you're trying to do. > >> > >I?m not sure what you mean by that. But: > > Erm, I misspoke (mistyped) a bit. I meant that it would be function > application that only allows a reference in the procedure position. I > don't know why I wrote RHS. > > Anyway, the strategy of having separate definition nodes for functions > and everything else is used in Cish. I don't recommend following > anything Cish does. However, it accomplishes this by having the > program node have non-function definitions first, and disallowing use > of functions on the RHS of those functions. > > The problem where it can't find a reference is a bug (that I haven't > fixed yet) caused by the lifting mechanics not really being completely > in sync with the reference mechanics with respect to nodes that have > multiple definition fields. When there are multiple definition > fields, definitions in the lower field can reference the definitions > in the higher field, but not vice versa. The lifting machinery > doesn't know about that ordering, it just stuffs the lifted definition > into the front of the definitions in whatever field it is targetting. > Thus it places a definition in a place that the reference machinery > thinks is out of scope. > > So... if you want to have different binders that get different fields > in the program node you can only make it work if you disallow stuff to > be sure you never need to reference something that will be lifted into > the lower field. > > That said, I've just fixed some issues to allow definition nodes to be > subtypable, so now you can have a definition node that is a subtype of > Definition, but has some different rules. Importantly, though, it can > be lifted to the same field. > > Eg. if I change the MyDefinition declaration in your example to look > like this: > > ``` > [MyDefinition Definition () > #:prop fresh (hash 'Expression (make-hole > 'ProcedureApplication))] > ``` > > It can now be lifted among normal definitions, but always have > ProcedureApplication as its RHS. > > However, I'm still not convinced that you actually want > ProcedureApplication as the RHS. Maybe you want to restrict lambda > nodes to only be generated on the RHS of a FunctionDefinition node > (that's a subtype of Definition), and make FunctionDefinition always > have a Lambda node as its RHS. > > There is an example in the documentation that is like this, in the > explanation of the `choice-filters-to-apply` property. But I'll paste > it here: > > ``` > (add-choice-method > my-component > no-lambda-except-global-def > [#f (? () #t)] > [Lambda (? () > (and (parent-node current-hole) > (equal? (ast-node-type (parent-node current-hole)) > 'Definition) > (parent-node (parent-node current-hole)) > (equal? (ast-node-type (parent-node (parent-node > current-hole))) > 'Program)))]) > (add-property my-component > choice-filters-to-apply > [#f (no-lambda-except-global-def)]) > ``` > > You can use something like that to allow Lambda nodes to only be > generated on the RHS of FunctionDefinition nodes, and define > FunctionDefinition like MyDefinition above to always generate Lambda > as its RHS, and I think it should work for a first-order language. > > > >(Also note that I tried unify! these unrestricted type variables > >with some concrete non-function type variables, but down the road, I get > an > >internal error about unification failure.). Perhaps I'm not supposed to > >unify! inside add-choice-method? > > Well, probably not. Generally the only place you want to `unify!` is > inside the type property in the function that determines child types > based on the parent type. Inside that function you can `unify!`, but > the unification needs to be consistent (IE not unified based on the > result of `random`) OR the choice needs to be saved in a way that it > can be looked up when that function is run again. > > The problem is that the choice method gets its type information based > on a hole node that is later replaced, so the type is recomputed > without necessarily consulting the type that the hole said it was. > You should be able to constrain the type using unification if you save > that type to re-unify it consistently later. But without seeing the > exact error and code that generated it, I'm not 100% certain. > > > > In the last email, I asked if lift-type->ast-binder-type could be used > > nondeterministically in this manner, and my experience with it so far > seems > > to indicate no. > > I think this should work, barring the issues mentioned above about the > lifting vs referencing bug when you have the different definition > types in different fields. Adjusting your example to have > MyDefinition as a subtype, and then only having a Definition field in > MyProgram, random choice seems to work fine. But sometimes it > generates gargantuan programs (that I kill before it finishes) because > it keeps choosing MyDefinition, requiring a function application, > which likely lifts a reference to a function, which might choose > MyDefinition, etc, in a bad lift chain that generates bigger and > bigger types. > > > Anyway, does that all help? > > > On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase wrote: > >> > >> Hmm, it seems like first-order functions would be encoded as a > >> function application node that only allows a reference as its RHS > >> rather than allowing arbitrary expressions. But that said, I'm not > >> sure I fully understand what you're trying to do. > >> > >I?m not sure what you mean by that. But: > > > > 1. I have no problem with the definition site. That?s easy to deal > with. > > 2. I also have no problem with restricting function application. That?s > > easy to deal with as well. > > 3. The problem is primarily with a bare variable reference bound to a > > closure value, which must only occur in function application. For > example, > > the following code is invalid. > > > >(define (f) (void)) > > > >(define g f) ;; this is invalid > >(define h (g)) > >(define a (list f)) ;; this is also invalid > >(define b (f)) ;; this is ok > > > >I have been using choice-filters-to-apply to restrict variable references > >whose type can-unify? with function-type that doesn't directly appear > under > >function application. But this is obviously not ideal because the filtered > >nodes are mostly those whose type is totally unrestricted. So filtering > >them would prevent them from being concretized to other non-function types > >as well. (Also note that I tried unify! these unrestricted type variables > >with some concrete non-function type variables, but down the road, I get > an > >internal error about unification failure.). Perhaps I'm not supposed to > >unify! inside add-choice-method? > > > >That's probably a bug. ?Lifting? is what I've called the situation > >> where a reference is generated that causes a new definition to be > >> added such that the variable is in scope at the use site. IE the > >> binding is lifted from the use site to some outer node that contains > >> definitions. `lift_X` is the name Xsmith gives to definitions that > >> are created automatically in this way (which is ultimately most > >> definitions). The exception is caused because somehow the lift ended > >> up in a place where it wasn't actually in scope. That shouldn't > >> happen. Anyway, if you point me to a version of a fuzzer that does > >> this, I'll try to debug it. > >> > >#lang clotho > > > >(require > > xsmith > > racr > > xsmith/racr-convenience > > xsmith/canned-components > > racket/string > > racket/list > > racket/pretty) > > > >;; We first define a basic component and add a bunch of expressions. > > > >(define-basic-spec-component somelisp) > > > >(add-basic-expressions somelisp > > #:ProgramWithSequence #t > > #:VoidExpression #t > > #:AssignmentExpression #t > > #:VariableReference #t > > #:ProcedureApplication #t > > #:IfExpression #t > > #:ExpressionSequence #t > > #:LetSequential #t > > #:LambdaWithExpression #t > > #:Numbers #t > > #:Booleans #t > > #:Strings #t > > #:ImmutableList #t) > > > >(add-loop-over-container > > somelisp > > #:name Map > > #:collection-type-constructor (? (inner) (immutable (list-type inner))) > > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > > > >(add-to-grammar > > somelisp > > [MyProgram #f > > ([decls : Definition *] > > [declsTwo : MyDefinition *] > > [body : Expression]) > > #:prop type-info > > [(fresh-type-variable) > > (? (n t) > > (hash 'decls (? (c) (fresh-type-variable)) > > 'declsTwo (? (c) (fresh-type-variable)) > > 'body (fresh-type-variable)))]] > > [MyDefinition #f ([type] > > [name = (fresh-var-name "b_")] > > [e : ProcedureApplication]) > > #:prop binder-info () > > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e > t))]]) > > > >(add-property > > somelisp > > lift-type->ast-binder-type > > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > > > >;; Sometimes we have a type variable that is not concrete but needs to be. > >;; Here we provide a list of options we can pick for an unconstrained > >;; type variable. > >(define (type-thunks-for-concretization) > > (list > > (?()int-type) > > (?()bool-type) > > (?()string-type) > > (?()(immutable (list-type (fresh-type-variable)))))) > > > >(define (somelisp-format-render s-exps) > > (define out (open-output-string)) > > (for ([symex s-exps]) > > (pretty-print symex out 1)) > > (get-output-string out)) > > > >(define-xsmith-interface-functions > > [somelisp] > > #:program-node MyProgram > > #:type-thunks type-thunks-for-concretization > > #:comment-wrap (? (lines) > > (string-join > > (map (? (l) (format ";; ~a" l)) lines) > > "\n")) > > #:format-render somelisp-format-render) > > > >(module+ main (somelisp-command-line)) > > > > > >> Probably something is going wrong in the `fresh` property. Have you > >> changed that? > >> > >I don?t think so. See the code at the end of the email. > > > >> I should change that error message. Anyway, it seems to think there > >> is no outer node that has a `Definition *` child that can be the > >> target for lifting a definition. Generally this happens when you > >> forget to have an outer Program node or something that can hold > >> definitions. > >> > >I don?t think that?s the case, but I could be wrong. > > > >#lang clotho > > > >(require > > xsmith > > racr > > xsmith/racr-convenience > > xsmith/canned-components > > racket/string > > racket/list > > racket/pretty) > > > >;; We first define a basic component and add a bunch of expressions. > > > >(define-basic-spec-component somelisp) > > > >(add-basic-expressions somelisp > > #:ProgramWithSequence #t > > #:VoidExpression #t > > #:AssignmentExpression #t > > #:VariableReference #t > > #:ProcedureApplication #t > > #:IfExpression #t > > #:ExpressionSequence #t > > #:LetSequential #t > > #:LambdaWithExpression #t > > #:Numbers #t > > #:Booleans #t > > #:Strings #t > > #:ImmutableList #t) > > > >(add-loop-over-container > > somelisp > > #:name Map > > #:collection-type-constructor (? (inner) (immutable (list-type inner))) > > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > > > >(add-to-grammar > > somelisp > > [MyProgram #f > > ([decls : Definition *] > > [body : SubProgram]) > > #:prop type-info > > [(fresh-type-variable) > > (? (n t) > > (hash 'decls (? (c) (fresh-type-variable)) > > 'body (fresh-type-variable)))]] > > [SubProgram #f > > ([decls : MyDefinition *] > > [body : Expression]) > > #:prop type-info > > [(fresh-type-variable) > > (? (n t) > > (hash 'decls (? (c) (fresh-type-variable)) > > 'body (fresh-type-variable)))]] > > [MyDefinition #f ([type] > > [name = (fresh-var-name "b_")] > > [e : ProcedureApplication]) > > #:prop binder-info () > > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e > t))]]) > > > >(add-property > > somelisp > > lift-type->ast-binder-type > > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > > > >;; Sometimes we have a type variable that is not concrete but needs to be. > >;; Here we provide a list of options we can pick for an unconstrained > >;; type variable. > >(define (type-thunks-for-concretization) > > (list > > (?()int-type) > > (?()bool-type) > > (?()string-type) > > (?()(immutable (list-type (fresh-type-variable)))))) > > > >(define (somelisp-format-render s-exps) > > (define out (open-output-string)) > > (for ([symex s-exps]) > > (pretty-print symex out 1)) > > (get-output-string out)) > > > >(define-xsmith-interface-functions > > [somelisp] > > #:program-node MyProgram > > #:type-thunks type-thunks-for-concretization > > #:comment-wrap (? (lines) > > (string-join > > (map (? (l) (format ";; ~a" l)) lines) > > "\n")) > > #:format-render somelisp-format-render) > > > >(module+ main (somelisp-command-line)) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Thu Jul 15 22:21:33 2021 From: william at hatch.uno (William G Hatch) Date: Thu, 15 Jul 2021 22:21:33 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: That would be great, thanks! On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase wrote: >Yes, this is very helpful. I will let you know if I encounter any further >issues after updating XSmith. > >By the way, do you want any help with Scribble? I notice a lot of typos and >link errors and can submit PRs to fix them. > >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch wrote: > >> Sorry for the delay. I wanted to do the debugging before responding, >> but then I didn't get to it for a while. >> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >> >> First, that issue with the bud-node was a bug in the `edit` property >> which the canned-components `ProcedureApplication` uses. It's fixed >> now. >> >> >> Hmm, it seems like first-order functions would be encoded as a >> >> function application node that only allows a reference as its RHS >> >> rather than allowing arbitrary expressions. But that said, I'm not >> >> sure I fully understand what you're trying to do. >> >> >> >I?m not sure what you mean by that. But: >> >> Erm, I misspoke (mistyped) a bit. I meant that it would be function >> application that only allows a reference in the procedure position. I >> don't know why I wrote RHS. >> >> Anyway, the strategy of having separate definition nodes for functions >> and everything else is used in Cish. I don't recommend following >> anything Cish does. However, it accomplishes this by having the >> program node have non-function definitions first, and disallowing use >> of functions on the RHS of those functions. >> >> The problem where it can't find a reference is a bug (that I haven't >> fixed yet) caused by the lifting mechanics not really being completely >> in sync with the reference mechanics with respect to nodes that have >> multiple definition fields. When there are multiple definition >> fields, definitions in the lower field can reference the definitions >> in the higher field, but not vice versa. The lifting machinery >> doesn't know about that ordering, it just stuffs the lifted definition >> into the front of the definitions in whatever field it is targetting. >> Thus it places a definition in a place that the reference machinery >> thinks is out of scope. >> >> So... if you want to have different binders that get different fields >> in the program node you can only make it work if you disallow stuff to >> be sure you never need to reference something that will be lifted into >> the lower field. >> >> That said, I've just fixed some issues to allow definition nodes to be >> subtypable, so now you can have a definition node that is a subtype of >> Definition, but has some different rules. Importantly, though, it can >> be lifted to the same field. >> >> Eg. if I change the MyDefinition declaration in your example to look >> like this: >> >> ``` >> [MyDefinition Definition () >> #:prop fresh (hash 'Expression (make-hole >> 'ProcedureApplication))] >> ``` >> >> It can now be lifted among normal definitions, but always have >> ProcedureApplication as its RHS. >> >> However, I'm still not convinced that you actually want >> ProcedureApplication as the RHS. Maybe you want to restrict lambda >> nodes to only be generated on the RHS of a FunctionDefinition node >> (that's a subtype of Definition), and make FunctionDefinition always >> have a Lambda node as its RHS. >> >> There is an example in the documentation that is like this, in the >> explanation of the `choice-filters-to-apply` property. But I'll paste >> it here: >> >> ``` >> (add-choice-method >> my-component >> no-lambda-except-global-def >> [#f (? () #t)] >> [Lambda (? () >> (and (parent-node current-hole) >> (equal? (ast-node-type (parent-node current-hole)) >> 'Definition) >> (parent-node (parent-node current-hole)) >> (equal? (ast-node-type (parent-node (parent-node >> current-hole))) >> 'Program)))]) >> (add-property my-component >> choice-filters-to-apply >> [#f (no-lambda-except-global-def)]) >> ``` >> >> You can use something like that to allow Lambda nodes to only be >> generated on the RHS of FunctionDefinition nodes, and define >> FunctionDefinition like MyDefinition above to always generate Lambda >> as its RHS, and I think it should work for a first-order language. >> >> >> >(Also note that I tried unify! these unrestricted type variables >> >with some concrete non-function type variables, but down the road, I get >> an >> >internal error about unification failure.). Perhaps I'm not supposed to >> >unify! inside add-choice-method? >> >> Well, probably not. Generally the only place you want to `unify!` is >> inside the type property in the function that determines child types >> based on the parent type. Inside that function you can `unify!`, but >> the unification needs to be consistent (IE not unified based on the >> result of `random`) OR the choice needs to be saved in a way that it >> can be looked up when that function is run again. >> >> The problem is that the choice method gets its type information based >> on a hole node that is later replaced, so the type is recomputed >> without necessarily consulting the type that the hole said it was. >> You should be able to constrain the type using unification if you save >> that type to re-unify it consistently later. But without seeing the >> exact error and code that generated it, I'm not 100% certain. >> >> >> > In the last email, I asked if lift-type->ast-binder-type could be used >> > nondeterministically in this manner, and my experience with it so far >> seems >> > to indicate no. >> >> I think this should work, barring the issues mentioned above about the >> lifting vs referencing bug when you have the different definition >> types in different fields. Adjusting your example to have >> MyDefinition as a subtype, and then only having a Definition field in >> MyProgram, random choice seems to work fine. But sometimes it >> generates gargantuan programs (that I kill before it finishes) because >> it keeps choosing MyDefinition, requiring a function application, >> which likely lifts a reference to a function, which might choose >> MyDefinition, etc, in a bad lift chain that generates bigger and >> bigger types. >> >> >> Anyway, does that all help? >> >> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase wrote: >> >> >> >> Hmm, it seems like first-order functions would be encoded as a >> >> function application node that only allows a reference as its RHS >> >> rather than allowing arbitrary expressions. But that said, I'm not >> >> sure I fully understand what you're trying to do. >> >> >> >I?m not sure what you mean by that. But: >> > >> > 1. I have no problem with the definition site. That?s easy to deal >> with. >> > 2. I also have no problem with restricting function application. That?s >> > easy to deal with as well. >> > 3. The problem is primarily with a bare variable reference bound to a >> > closure value, which must only occur in function application. For >> example, >> > the following code is invalid. >> > >> >(define (f) (void)) >> > >> >(define g f) ;; this is invalid >> >(define h (g)) >> >(define a (list f)) ;; this is also invalid >> >(define b (f)) ;; this is ok >> > >> >I have been using choice-filters-to-apply to restrict variable references >> >whose type can-unify? with function-type that doesn't directly appear >> under >> >function application. But this is obviously not ideal because the filtered >> >nodes are mostly those whose type is totally unrestricted. So filtering >> >them would prevent them from being concretized to other non-function types >> >as well. (Also note that I tried unify! these unrestricted type variables >> >with some concrete non-function type variables, but down the road, I get >> an >> >internal error about unification failure.). Perhaps I'm not supposed to >> >unify! inside add-choice-method? >> > >> >That's probably a bug. ?Lifting? is what I've called the situation >> >> where a reference is generated that causes a new definition to be >> >> added such that the variable is in scope at the use site. IE the >> >> binding is lifted from the use site to some outer node that contains >> >> definitions. `lift_X` is the name Xsmith gives to definitions that >> >> are created automatically in this way (which is ultimately most >> >> definitions). The exception is caused because somehow the lift ended >> >> up in a place where it wasn't actually in scope. That shouldn't >> >> happen. Anyway, if you point me to a version of a fuzzer that does >> >> this, I'll try to debug it. >> >> >> >#lang clotho >> > >> >(require >> > xsmith >> > racr >> > xsmith/racr-convenience >> > xsmith/canned-components >> > racket/string >> > racket/list >> > racket/pretty) >> > >> >;; We first define a basic component and add a bunch of expressions. >> > >> >(define-basic-spec-component somelisp) >> > >> >(add-basic-expressions somelisp >> > #:ProgramWithSequence #t >> > #:VoidExpression #t >> > #:AssignmentExpression #t >> > #:VariableReference #t >> > #:ProcedureApplication #t >> > #:IfExpression #t >> > #:ExpressionSequence #t >> > #:LetSequential #t >> > #:LambdaWithExpression #t >> > #:Numbers #t >> > #:Booleans #t >> > #:Strings #t >> > #:ImmutableList #t) >> > >> >(add-loop-over-container >> > somelisp >> > #:name Map >> > #:collection-type-constructor (? (inner) (immutable (list-type inner))) >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >> > >> >(add-to-grammar >> > somelisp >> > [MyProgram #f >> > ([decls : Definition *] >> > [declsTwo : MyDefinition *] >> > [body : Expression]) >> > #:prop type-info >> > [(fresh-type-variable) >> > (? (n t) >> > (hash 'decls (? (c) (fresh-type-variable)) >> > 'declsTwo (? (c) (fresh-type-variable)) >> > 'body (fresh-type-variable)))]] >> > [MyDefinition #f ([type] >> > [name = (fresh-var-name "b_")] >> > [e : ProcedureApplication]) >> > #:prop binder-info () >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e >> t))]]) >> > >> >(add-property >> > somelisp >> > lift-type->ast-binder-type >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >> > >> >;; Sometimes we have a type variable that is not concrete but needs to be. >> >;; Here we provide a list of options we can pick for an unconstrained >> >;; type variable. >> >(define (type-thunks-for-concretization) >> > (list >> > (?()int-type) >> > (?()bool-type) >> > (?()string-type) >> > (?()(immutable (list-type (fresh-type-variable)))))) >> > >> >(define (somelisp-format-render s-exps) >> > (define out (open-output-string)) >> > (for ([symex s-exps]) >> > (pretty-print symex out 1)) >> > (get-output-string out)) >> > >> >(define-xsmith-interface-functions >> > [somelisp] >> > #:program-node MyProgram >> > #:type-thunks type-thunks-for-concretization >> > #:comment-wrap (? (lines) >> > (string-join >> > (map (? (l) (format ";; ~a" l)) lines) >> > "\n")) >> > #:format-render somelisp-format-render) >> > >> >(module+ main (somelisp-command-line)) >> > >> > >> >> Probably something is going wrong in the `fresh` property. Have you >> >> changed that? >> >> >> >I don?t think so. See the code at the end of the email. >> > >> >> I should change that error message. Anyway, it seems to think there >> >> is no outer node that has a `Definition *` child that can be the >> >> target for lifting a definition. Generally this happens when you >> >> forget to have an outer Program node or something that can hold >> >> definitions. >> >> >> >I don?t think that?s the case, but I could be wrong. >> > >> >#lang clotho >> > >> >(require >> > xsmith >> > racr >> > xsmith/racr-convenience >> > xsmith/canned-components >> > racket/string >> > racket/list >> > racket/pretty) >> > >> >;; We first define a basic component and add a bunch of expressions. >> > >> >(define-basic-spec-component somelisp) >> > >> >(add-basic-expressions somelisp >> > #:ProgramWithSequence #t >> > #:VoidExpression #t >> > #:AssignmentExpression #t >> > #:VariableReference #t >> > #:ProcedureApplication #t >> > #:IfExpression #t >> > #:ExpressionSequence #t >> > #:LetSequential #t >> > #:LambdaWithExpression #t >> > #:Numbers #t >> > #:Booleans #t >> > #:Strings #t >> > #:ImmutableList #t) >> > >> >(add-loop-over-container >> > somelisp >> > #:name Map >> > #:collection-type-constructor (? (inner) (immutable (list-type inner))) >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >> > >> >(add-to-grammar >> > somelisp >> > [MyProgram #f >> > ([decls : Definition *] >> > [body : SubProgram]) >> > #:prop type-info >> > [(fresh-type-variable) >> > (? (n t) >> > (hash 'decls (? (c) (fresh-type-variable)) >> > 'body (fresh-type-variable)))]] >> > [SubProgram #f >> > ([decls : MyDefinition *] >> > [body : Expression]) >> > #:prop type-info >> > [(fresh-type-variable) >> > (? (n t) >> > (hash 'decls (? (c) (fresh-type-variable)) >> > 'body (fresh-type-variable)))]] >> > [MyDefinition #f ([type] >> > [name = (fresh-var-name "b_")] >> > [e : ProcedureApplication]) >> > #:prop binder-info () >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash 'e >> t))]]) >> > >> >(add-property >> > somelisp >> > lift-type->ast-binder-type >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >> > >> >;; Sometimes we have a type variable that is not concrete but needs to be. >> >;; Here we provide a list of options we can pick for an unconstrained >> >;; type variable. >> >(define (type-thunks-for-concretization) >> > (list >> > (?()int-type) >> > (?()bool-type) >> > (?()string-type) >> > (?()(immutable (list-type (fresh-type-variable)))))) >> > >> >(define (somelisp-format-render s-exps) >> > (define out (open-output-string)) >> > (for ([symex s-exps]) >> > (pretty-print symex out 1)) >> > (get-output-string out)) >> > >> >(define-xsmith-interface-functions >> > [somelisp] >> > #:program-node MyProgram >> > #:type-thunks type-thunks-for-concretization >> > #:comment-wrap (? (lines) >> > (string-join >> > (map (? (l) (format ";; ~a" l)) lines) >> > "\n")) >> > #:format-render somelisp-format-render) >> > >> >(module+ main (somelisp-command-line)) >> From sorawee.pwase at gmail.com Fri Jul 16 19:02:44 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 16 Jul 2021 18:02:44 -0700 Subject: [xsmith-dev] Patches Message-ID: Hi everyone, Since I don't have a Utah GitLab account, I can't submit a PR, so I attach patch files here instead. fix-print-choice-log.patch fixes a problem with the --print-choice-log option, which attempts to create a string of negative length. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-print-choice-log.patch Type: application/x-patch Size: 912 bytes Desc: not available URL: From sorawee.pwase at gmail.com Fri Jul 16 21:00:39 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 16 Jul 2021 20:00:39 -0700 Subject: [xsmith-dev] Patches In-Reply-To: References: Message-ID: Sorry, I meant to include another patch but accidentally sent the email prematurely. doc.patch is my first pass on the documentation. On Fri, Jul 16, 2021 at 6:02 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Hi everyone, > > Since I don't have a Utah GitLab account, I can't submit a PR, so I attach > patch files here instead. > > fix-print-choice-log.patch fixes a problem with the --print-choice-log > option, which attempts to create a string of negative length. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: doc.patch Type: application/octet-stream Size: 60538 bytes Desc: not available URL: From sorawee.pwase at gmail.com Fri Jul 16 21:32:13 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 16 Jul 2021 20:32:13 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Do you have a roadmap for implementing "Provide mechanism for precluding types in `fresh-type-variable` invocations"? ( https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really want this feature. On Thu, Jul 15, 2021 at 9:22 PM William G Hatch wrote: > That would be great, thanks! > > On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase wrote: > >Yes, this is very helpful. I will let you know if I encounter any further > >issues after updating XSmith. > > > >By the way, do you want any help with Scribble? I notice a lot of typos > and > >link errors and can submit PRs to fix them. > > > >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > wrote: > > > >> Sorry for the delay. I wanted to do the debugging before responding, > >> but then I didn't get to it for a while. > >> > >> Anyway, I now see what's going wrong, and I've fixed some bugs. > >> > >> First, that issue with the bud-node was a bug in the `edit` property > >> which the canned-components `ProcedureApplication` uses. It's fixed > >> now. > >> > >> >> Hmm, it seems like first-order functions would be encoded as a > >> >> function application node that only allows a reference as its RHS > >> >> rather than allowing arbitrary expressions. But that said, I'm not > >> >> sure I fully understand what you're trying to do. > >> >> > >> >I?m not sure what you mean by that. But: > >> > >> Erm, I misspoke (mistyped) a bit. I meant that it would be function > >> application that only allows a reference in the procedure position. I > >> don't know why I wrote RHS. > >> > >> Anyway, the strategy of having separate definition nodes for functions > >> and everything else is used in Cish. I don't recommend following > >> anything Cish does. However, it accomplishes this by having the > >> program node have non-function definitions first, and disallowing use > >> of functions on the RHS of those functions. > >> > >> The problem where it can't find a reference is a bug (that I haven't > >> fixed yet) caused by the lifting mechanics not really being completely > >> in sync with the reference mechanics with respect to nodes that have > >> multiple definition fields. When there are multiple definition > >> fields, definitions in the lower field can reference the definitions > >> in the higher field, but not vice versa. The lifting machinery > >> doesn't know about that ordering, it just stuffs the lifted definition > >> into the front of the definitions in whatever field it is targetting. > >> Thus it places a definition in a place that the reference machinery > >> thinks is out of scope. > >> > >> So... if you want to have different binders that get different fields > >> in the program node you can only make it work if you disallow stuff to > >> be sure you never need to reference something that will be lifted into > >> the lower field. > >> > >> That said, I've just fixed some issues to allow definition nodes to be > >> subtypable, so now you can have a definition node that is a subtype of > >> Definition, but has some different rules. Importantly, though, it can > >> be lifted to the same field. > >> > >> Eg. if I change the MyDefinition declaration in your example to look > >> like this: > >> > >> ``` > >> [MyDefinition Definition () > >> #:prop fresh (hash 'Expression (make-hole > >> 'ProcedureApplication))] > >> ``` > >> > >> It can now be lifted among normal definitions, but always have > >> ProcedureApplication as its RHS. > >> > >> However, I'm still not convinced that you actually want > >> ProcedureApplication as the RHS. Maybe you want to restrict lambda > >> nodes to only be generated on the RHS of a FunctionDefinition node > >> (that's a subtype of Definition), and make FunctionDefinition always > >> have a Lambda node as its RHS. > >> > >> There is an example in the documentation that is like this, in the > >> explanation of the `choice-filters-to-apply` property. But I'll paste > >> it here: > >> > >> ``` > >> (add-choice-method > >> my-component > >> no-lambda-except-global-def > >> [#f (? () #t)] > >> [Lambda (? () > >> (and (parent-node current-hole) > >> (equal? (ast-node-type (parent-node current-hole)) > >> 'Definition) > >> (parent-node (parent-node current-hole)) > >> (equal? (ast-node-type (parent-node (parent-node > >> current-hole))) > >> 'Program)))]) > >> (add-property my-component > >> choice-filters-to-apply > >> [#f (no-lambda-except-global-def)]) > >> ``` > >> > >> You can use something like that to allow Lambda nodes to only be > >> generated on the RHS of FunctionDefinition nodes, and define > >> FunctionDefinition like MyDefinition above to always generate Lambda > >> as its RHS, and I think it should work for a first-order language. > >> > >> > >> >(Also note that I tried unify! these unrestricted type variables > >> >with some concrete non-function type variables, but down the road, I > get > >> an > >> >internal error about unification failure.). Perhaps I'm not supposed to > >> >unify! inside add-choice-method? > >> > >> Well, probably not. Generally the only place you want to `unify!` is > >> inside the type property in the function that determines child types > >> based on the parent type. Inside that function you can `unify!`, but > >> the unification needs to be consistent (IE not unified based on the > >> result of `random`) OR the choice needs to be saved in a way that it > >> can be looked up when that function is run again. > >> > >> The problem is that the choice method gets its type information based > >> on a hole node that is later replaced, so the type is recomputed > >> without necessarily consulting the type that the hole said it was. > >> You should be able to constrain the type using unification if you save > >> that type to re-unify it consistently later. But without seeing the > >> exact error and code that generated it, I'm not 100% certain. > >> > >> > >> > In the last email, I asked if lift-type->ast-binder-type could be used > >> > nondeterministically in this manner, and my experience with it so far > >> seems > >> > to indicate no. > >> > >> I think this should work, barring the issues mentioned above about the > >> lifting vs referencing bug when you have the different definition > >> types in different fields. Adjusting your example to have > >> MyDefinition as a subtype, and then only having a Definition field in > >> MyProgram, random choice seems to work fine. But sometimes it > >> generates gargantuan programs (that I kill before it finishes) because > >> it keeps choosing MyDefinition, requiring a function application, > >> which likely lifts a reference to a function, which might choose > >> MyDefinition, etc, in a bad lift chain that generates bigger and > >> bigger types. > >> > >> > >> Anyway, does that all help? > >> > >> > >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase wrote: > >> >> > >> >> Hmm, it seems like first-order functions would be encoded as a > >> >> function application node that only allows a reference as its RHS > >> >> rather than allowing arbitrary expressions. But that said, I'm not > >> >> sure I fully understand what you're trying to do. > >> >> > >> >I?m not sure what you mean by that. But: > >> > > >> > 1. I have no problem with the definition site. That?s easy to deal > >> with. > >> > 2. I also have no problem with restricting function application. > That?s > >> > easy to deal with as well. > >> > 3. The problem is primarily with a bare variable reference bound to > a > >> > closure value, which must only occur in function application. For > >> example, > >> > the following code is invalid. > >> > > >> >(define (f) (void)) > >> > > >> >(define g f) ;; this is invalid > >> >(define h (g)) > >> >(define a (list f)) ;; this is also invalid > >> >(define b (f)) ;; this is ok > >> > > >> >I have been using choice-filters-to-apply to restrict variable > references > >> >whose type can-unify? with function-type that doesn't directly appear > >> under > >> >function application. But this is obviously not ideal because the > filtered > >> >nodes are mostly those whose type is totally unrestricted. So filtering > >> >them would prevent them from being concretized to other non-function > types > >> >as well. (Also note that I tried unify! these unrestricted type > variables > >> >with some concrete non-function type variables, but down the road, I > get > >> an > >> >internal error about unification failure.). Perhaps I'm not supposed to > >> >unify! inside add-choice-method? > >> > > >> >That's probably a bug. ?Lifting? is what I've called the situation > >> >> where a reference is generated that causes a new definition to be > >> >> added such that the variable is in scope at the use site. IE the > >> >> binding is lifted from the use site to some outer node that contains > >> >> definitions. `lift_X` is the name Xsmith gives to definitions that > >> >> are created automatically in this way (which is ultimately most > >> >> definitions). The exception is caused because somehow the lift ended > >> >> up in a place where it wasn't actually in scope. That shouldn't > >> >> happen. Anyway, if you point me to a version of a fuzzer that does > >> >> this, I'll try to debug it. > >> >> > >> >#lang clotho > >> > > >> >(require > >> > xsmith > >> > racr > >> > xsmith/racr-convenience > >> > xsmith/canned-components > >> > racket/string > >> > racket/list > >> > racket/pretty) > >> > > >> >;; We first define a basic component and add a bunch of expressions. > >> > > >> >(define-basic-spec-component somelisp) > >> > > >> >(add-basic-expressions somelisp > >> > #:ProgramWithSequence #t > >> > #:VoidExpression #t > >> > #:AssignmentExpression #t > >> > #:VariableReference #t > >> > #:ProcedureApplication #t > >> > #:IfExpression #t > >> > #:ExpressionSequence #t > >> > #:LetSequential #t > >> > #:LambdaWithExpression #t > >> > #:Numbers #t > >> > #:Booleans #t > >> > #:Strings #t > >> > #:ImmutableList #t) > >> > > >> >(add-loop-over-container > >> > somelisp > >> > #:name Map > >> > #:collection-type-constructor (? (inner) (immutable (list-type > inner))) > >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >> > > >> >(add-to-grammar > >> > somelisp > >> > [MyProgram #f > >> > ([decls : Definition *] > >> > [declsTwo : MyDefinition *] > >> > [body : Expression]) > >> > #:prop type-info > >> > [(fresh-type-variable) > >> > (? (n t) > >> > (hash 'decls (? (c) (fresh-type-variable)) > >> > 'declsTwo (? (c) (fresh-type-variable)) > >> > 'body (fresh-type-variable)))]] > >> > [MyDefinition #f ([type] > >> > [name = (fresh-var-name "b_")] > >> > [e : ProcedureApplication]) > >> > #:prop binder-info () > >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash > 'e > >> t))]]) > >> > > >> >(add-property > >> > somelisp > >> > lift-type->ast-binder-type > >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >> > > >> >;; Sometimes we have a type variable that is not concrete but needs to > be. > >> >;; Here we provide a list of options we can pick for an unconstrained > >> >;; type variable. > >> >(define (type-thunks-for-concretization) > >> > (list > >> > (?()int-type) > >> > (?()bool-type) > >> > (?()string-type) > >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> > > >> >(define (somelisp-format-render s-exps) > >> > (define out (open-output-string)) > >> > (for ([symex s-exps]) > >> > (pretty-print symex out 1)) > >> > (get-output-string out)) > >> > > >> >(define-xsmith-interface-functions > >> > [somelisp] > >> > #:program-node MyProgram > >> > #:type-thunks type-thunks-for-concretization > >> > #:comment-wrap (? (lines) > >> > (string-join > >> > (map (? (l) (format ";; ~a" l)) lines) > >> > "\n")) > >> > #:format-render somelisp-format-render) > >> > > >> >(module+ main (somelisp-command-line)) > >> > > >> > > >> >> Probably something is going wrong in the `fresh` property. Have you > >> >> changed that? > >> >> > >> >I don?t think so. See the code at the end of the email. > >> > > >> >> I should change that error message. Anyway, it seems to think there > >> >> is no outer node that has a `Definition *` child that can be the > >> >> target for lifting a definition. Generally this happens when you > >> >> forget to have an outer Program node or something that can hold > >> >> definitions. > >> >> > >> >I don?t think that?s the case, but I could be wrong. > >> > > >> >#lang clotho > >> > > >> >(require > >> > xsmith > >> > racr > >> > xsmith/racr-convenience > >> > xsmith/canned-components > >> > racket/string > >> > racket/list > >> > racket/pretty) > >> > > >> >;; We first define a basic component and add a bunch of expressions. > >> > > >> >(define-basic-spec-component somelisp) > >> > > >> >(add-basic-expressions somelisp > >> > #:ProgramWithSequence #t > >> > #:VoidExpression #t > >> > #:AssignmentExpression #t > >> > #:VariableReference #t > >> > #:ProcedureApplication #t > >> > #:IfExpression #t > >> > #:ExpressionSequence #t > >> > #:LetSequential #t > >> > #:LambdaWithExpression #t > >> > #:Numbers #t > >> > #:Booleans #t > >> > #:Strings #t > >> > #:ImmutableList #t) > >> > > >> >(add-loop-over-container > >> > somelisp > >> > #:name Map > >> > #:collection-type-constructor (? (inner) (immutable (list-type > inner))) > >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >> > > >> >(add-to-grammar > >> > somelisp > >> > [MyProgram #f > >> > ([decls : Definition *] > >> > [body : SubProgram]) > >> > #:prop type-info > >> > [(fresh-type-variable) > >> > (? (n t) > >> > (hash 'decls (? (c) (fresh-type-variable)) > >> > 'body (fresh-type-variable)))]] > >> > [SubProgram #f > >> > ([decls : MyDefinition *] > >> > [body : Expression]) > >> > #:prop type-info > >> > [(fresh-type-variable) > >> > (? (n t) > >> > (hash 'decls (? (c) (fresh-type-variable)) > >> > 'body (fresh-type-variable)))]] > >> > [MyDefinition #f ([type] > >> > [name = (fresh-var-name "b_")] > >> > [e : ProcedureApplication]) > >> > #:prop binder-info () > >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash > 'e > >> t))]]) > >> > > >> >(add-property > >> > somelisp > >> > lift-type->ast-binder-type > >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >> > > >> >;; Sometimes we have a type variable that is not concrete but needs to > be. > >> >;; Here we provide a list of options we can pick for an unconstrained > >> >;; type variable. > >> >(define (type-thunks-for-concretization) > >> > (list > >> > (?()int-type) > >> > (?()bool-type) > >> > (?()string-type) > >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> > > >> >(define (somelisp-format-render s-exps) > >> > (define out (open-output-string)) > >> > (for ([symex s-exps]) > >> > (pretty-print symex out 1)) > >> > (get-output-string out)) > >> > > >> >(define-xsmith-interface-functions > >> > [somelisp] > >> > #:program-node MyProgram > >> > #:type-thunks type-thunks-for-concretization > >> > #:comment-wrap (? (lines) > >> > (string-join > >> > (map (? (l) (format ";; ~a" l)) lines) > >> > "\n")) > >> > #:format-render somelisp-format-render) > >> > > >> >(module+ main (somelisp-command-line)) > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeide at cs.utah.edu Fri Jul 16 23:44:52 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Fri, 16 Jul 2021 23:44:52 -0600 Subject: [xsmith-dev] Patches In-Reply-To: (Sorawee Porncharoenwase's message of "Fri, 16 Jul 2021 18:02:44 -0700") References: Message-ID: Sorawee Porncharoenwase writes: > Since I don't have a Utah GitLab account, I can't submit a PR, so I attach > patch files here instead. Thank you! (FWIW, you can request an account on the Flux GitLab server. If you request one let me know, so that I can make sure the request is approved quickly.) -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From sorawee.pwase at gmail.com Mon Jul 19 17:15:04 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Mon, 19 Jul 2021 16:15:04 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Also, I'm curious about the setup of various fuzzers that you are running. How many programs are you able to generate and test per minute? Do you have any particular parallelization setup that works well for you? On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Do you have a roadmap for implementing "Provide mechanism for precluding > types in `fresh-type-variable` invocations"? ( > https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really > want this feature. > > On Thu, Jul 15, 2021 at 9:22 PM William G Hatch wrote: > >> That would be great, thanks! >> >> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase wrote: >> >Yes, this is very helpful. I will let you know if I encounter any further >> >issues after updating XSmith. >> > >> >By the way, do you want any help with Scribble? I notice a lot of typos >> and >> >link errors and can submit PRs to fix them. >> > >> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >> wrote: >> > >> >> Sorry for the delay. I wanted to do the debugging before responding, >> >> but then I didn't get to it for a while. >> >> >> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >> >> >> >> First, that issue with the bud-node was a bug in the `edit` property >> >> which the canned-components `ProcedureApplication` uses. It's fixed >> >> now. >> >> >> >> >> Hmm, it seems like first-order functions would be encoded as a >> >> >> function application node that only allows a reference as its RHS >> >> >> rather than allowing arbitrary expressions. But that said, I'm not >> >> >> sure I fully understand what you're trying to do. >> >> >> >> >> >I?m not sure what you mean by that. But: >> >> >> >> Erm, I misspoke (mistyped) a bit. I meant that it would be function >> >> application that only allows a reference in the procedure position. I >> >> don't know why I wrote RHS. >> >> >> >> Anyway, the strategy of having separate definition nodes for functions >> >> and everything else is used in Cish. I don't recommend following >> >> anything Cish does. However, it accomplishes this by having the >> >> program node have non-function definitions first, and disallowing use >> >> of functions on the RHS of those functions. >> >> >> >> The problem where it can't find a reference is a bug (that I haven't >> >> fixed yet) caused by the lifting mechanics not really being completely >> >> in sync with the reference mechanics with respect to nodes that have >> >> multiple definition fields. When there are multiple definition >> >> fields, definitions in the lower field can reference the definitions >> >> in the higher field, but not vice versa. The lifting machinery >> >> doesn't know about that ordering, it just stuffs the lifted definition >> >> into the front of the definitions in whatever field it is targetting. >> >> Thus it places a definition in a place that the reference machinery >> >> thinks is out of scope. >> >> >> >> So... if you want to have different binders that get different fields >> >> in the program node you can only make it work if you disallow stuff to >> >> be sure you never need to reference something that will be lifted into >> >> the lower field. >> >> >> >> That said, I've just fixed some issues to allow definition nodes to be >> >> subtypable, so now you can have a definition node that is a subtype of >> >> Definition, but has some different rules. Importantly, though, it can >> >> be lifted to the same field. >> >> >> >> Eg. if I change the MyDefinition declaration in your example to look >> >> like this: >> >> >> >> ``` >> >> [MyDefinition Definition () >> >> #:prop fresh (hash 'Expression (make-hole >> >> 'ProcedureApplication))] >> >> ``` >> >> >> >> It can now be lifted among normal definitions, but always have >> >> ProcedureApplication as its RHS. >> >> >> >> However, I'm still not convinced that you actually want >> >> ProcedureApplication as the RHS. Maybe you want to restrict lambda >> >> nodes to only be generated on the RHS of a FunctionDefinition node >> >> (that's a subtype of Definition), and make FunctionDefinition always >> >> have a Lambda node as its RHS. >> >> >> >> There is an example in the documentation that is like this, in the >> >> explanation of the `choice-filters-to-apply` property. But I'll paste >> >> it here: >> >> >> >> ``` >> >> (add-choice-method >> >> my-component >> >> no-lambda-except-global-def >> >> [#f (? () #t)] >> >> [Lambda (? () >> >> (and (parent-node current-hole) >> >> (equal? (ast-node-type (parent-node current-hole)) >> >> 'Definition) >> >> (parent-node (parent-node current-hole)) >> >> (equal? (ast-node-type (parent-node (parent-node >> >> current-hole))) >> >> 'Program)))]) >> >> (add-property my-component >> >> choice-filters-to-apply >> >> [#f (no-lambda-except-global-def)]) >> >> ``` >> >> >> >> You can use something like that to allow Lambda nodes to only be >> >> generated on the RHS of FunctionDefinition nodes, and define >> >> FunctionDefinition like MyDefinition above to always generate Lambda >> >> as its RHS, and I think it should work for a first-order language. >> >> >> >> >> >> >(Also note that I tried unify! these unrestricted type variables >> >> >with some concrete non-function type variables, but down the road, I >> get >> >> an >> >> >internal error about unification failure.). Perhaps I'm not supposed >> to >> >> >unify! inside add-choice-method? >> >> >> >> Well, probably not. Generally the only place you want to `unify!` is >> >> inside the type property in the function that determines child types >> >> based on the parent type. Inside that function you can `unify!`, but >> >> the unification needs to be consistent (IE not unified based on the >> >> result of `random`) OR the choice needs to be saved in a way that it >> >> can be looked up when that function is run again. >> >> >> >> The problem is that the choice method gets its type information based >> >> on a hole node that is later replaced, so the type is recomputed >> >> without necessarily consulting the type that the hole said it was. >> >> You should be able to constrain the type using unification if you save >> >> that type to re-unify it consistently later. But without seeing the >> >> exact error and code that generated it, I'm not 100% certain. >> >> >> >> >> >> > In the last email, I asked if lift-type->ast-binder-type could be >> used >> >> > nondeterministically in this manner, and my experience with it so far >> >> seems >> >> > to indicate no. >> >> >> >> I think this should work, barring the issues mentioned above about the >> >> lifting vs referencing bug when you have the different definition >> >> types in different fields. Adjusting your example to have >> >> MyDefinition as a subtype, and then only having a Definition field in >> >> MyProgram, random choice seems to work fine. But sometimes it >> >> generates gargantuan programs (that I kill before it finishes) because >> >> it keeps choosing MyDefinition, requiring a function application, >> >> which likely lifts a reference to a function, which might choose >> >> MyDefinition, etc, in a bad lift chain that generates bigger and >> >> bigger types. >> >> >> >> >> >> Anyway, does that all help? >> >> >> >> >> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase >> wrote: >> >> >> >> >> >> Hmm, it seems like first-order functions would be encoded as a >> >> >> function application node that only allows a reference as its RHS >> >> >> rather than allowing arbitrary expressions. But that said, I'm not >> >> >> sure I fully understand what you're trying to do. >> >> >> >> >> >I?m not sure what you mean by that. But: >> >> > >> >> > 1. I have no problem with the definition site. That?s easy to deal >> >> with. >> >> > 2. I also have no problem with restricting function application. >> That?s >> >> > easy to deal with as well. >> >> > 3. The problem is primarily with a bare variable reference bound >> to a >> >> > closure value, which must only occur in function application. For >> >> example, >> >> > the following code is invalid. >> >> > >> >> >(define (f) (void)) >> >> > >> >> >(define g f) ;; this is invalid >> >> >(define h (g)) >> >> >(define a (list f)) ;; this is also invalid >> >> >(define b (f)) ;; this is ok >> >> > >> >> >I have been using choice-filters-to-apply to restrict variable >> references >> >> >whose type can-unify? with function-type that doesn't directly appear >> >> under >> >> >function application. But this is obviously not ideal because the >> filtered >> >> >nodes are mostly those whose type is totally unrestricted. So >> filtering >> >> >them would prevent them from being concretized to other non-function >> types >> >> >as well. (Also note that I tried unify! these unrestricted type >> variables >> >> >with some concrete non-function type variables, but down the road, I >> get >> >> an >> >> >internal error about unification failure.). Perhaps I'm not supposed >> to >> >> >unify! inside add-choice-method? >> >> > >> >> >That's probably a bug. ?Lifting? is what I've called the situation >> >> >> where a reference is generated that causes a new definition to be >> >> >> added such that the variable is in scope at the use site. IE the >> >> >> binding is lifted from the use site to some outer node that contains >> >> >> definitions. `lift_X` is the name Xsmith gives to definitions that >> >> >> are created automatically in this way (which is ultimately most >> >> >> definitions). The exception is caused because somehow the lift >> ended >> >> >> up in a place where it wasn't actually in scope. That shouldn't >> >> >> happen. Anyway, if you point me to a version of a fuzzer that does >> >> >> this, I'll try to debug it. >> >> >> >> >> >#lang clotho >> >> > >> >> >(require >> >> > xsmith >> >> > racr >> >> > xsmith/racr-convenience >> >> > xsmith/canned-components >> >> > racket/string >> >> > racket/list >> >> > racket/pretty) >> >> > >> >> >;; We first define a basic component and add a bunch of expressions. >> >> > >> >> >(define-basic-spec-component somelisp) >> >> > >> >> >(add-basic-expressions somelisp >> >> > #:ProgramWithSequence #t >> >> > #:VoidExpression #t >> >> > #:AssignmentExpression #t >> >> > #:VariableReference #t >> >> > #:ProcedureApplication #t >> >> > #:IfExpression #t >> >> > #:ExpressionSequence #t >> >> > #:LetSequential #t >> >> > #:LambdaWithExpression #t >> >> > #:Numbers #t >> >> > #:Booleans #t >> >> > #:Strings #t >> >> > #:ImmutableList #t) >> >> > >> >> >(add-loop-over-container >> >> > somelisp >> >> > #:name Map >> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> inner))) >> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >> >> > >> >> >(add-to-grammar >> >> > somelisp >> >> > [MyProgram #f >> >> > ([decls : Definition *] >> >> > [declsTwo : MyDefinition *] >> >> > [body : Expression]) >> >> > #:prop type-info >> >> > [(fresh-type-variable) >> >> > (? (n t) >> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> > 'declsTwo (? (c) (fresh-type-variable)) >> >> > 'body (fresh-type-variable)))]] >> >> > [MyDefinition #f ([type] >> >> > [name = (fresh-var-name "b_")] >> >> > [e : ProcedureApplication]) >> >> > #:prop binder-info () >> >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash >> 'e >> >> t))]]) >> >> > >> >> >(add-property >> >> > somelisp >> >> > lift-type->ast-binder-type >> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >> >> > >> >> >;; Sometimes we have a type variable that is not concrete but needs >> to be. >> >> >;; Here we provide a list of options we can pick for an unconstrained >> >> >;; type variable. >> >> >(define (type-thunks-for-concretization) >> >> > (list >> >> > (?()int-type) >> >> > (?()bool-type) >> >> > (?()string-type) >> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> > >> >> >(define (somelisp-format-render s-exps) >> >> > (define out (open-output-string)) >> >> > (for ([symex s-exps]) >> >> > (pretty-print symex out 1)) >> >> > (get-output-string out)) >> >> > >> >> >(define-xsmith-interface-functions >> >> > [somelisp] >> >> > #:program-node MyProgram >> >> > #:type-thunks type-thunks-for-concretization >> >> > #:comment-wrap (? (lines) >> >> > (string-join >> >> > (map (? (l) (format ";; ~a" l)) lines) >> >> > "\n")) >> >> > #:format-render somelisp-format-render) >> >> > >> >> >(module+ main (somelisp-command-line)) >> >> > >> >> > >> >> >> Probably something is going wrong in the `fresh` property. Have you >> >> >> changed that? >> >> >> >> >> >I don?t think so. See the code at the end of the email. >> >> > >> >> >> I should change that error message. Anyway, it seems to think there >> >> >> is no outer node that has a `Definition *` child that can be the >> >> >> target for lifting a definition. Generally this happens when you >> >> >> forget to have an outer Program node or something that can hold >> >> >> definitions. >> >> >> >> >> >I don?t think that?s the case, but I could be wrong. >> >> > >> >> >#lang clotho >> >> > >> >> >(require >> >> > xsmith >> >> > racr >> >> > xsmith/racr-convenience >> >> > xsmith/canned-components >> >> > racket/string >> >> > racket/list >> >> > racket/pretty) >> >> > >> >> >;; We first define a basic component and add a bunch of expressions. >> >> > >> >> >(define-basic-spec-component somelisp) >> >> > >> >> >(add-basic-expressions somelisp >> >> > #:ProgramWithSequence #t >> >> > #:VoidExpression #t >> >> > #:AssignmentExpression #t >> >> > #:VariableReference #t >> >> > #:ProcedureApplication #t >> >> > #:IfExpression #t >> >> > #:ExpressionSequence #t >> >> > #:LetSequential #t >> >> > #:LambdaWithExpression #t >> >> > #:Numbers #t >> >> > #:Booleans #t >> >> > #:Strings #t >> >> > #:ImmutableList #t) >> >> > >> >> >(add-loop-over-container >> >> > somelisp >> >> > #:name Map >> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> inner))) >> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >> >> > >> >> >(add-to-grammar >> >> > somelisp >> >> > [MyProgram #f >> >> > ([decls : Definition *] >> >> > [body : SubProgram]) >> >> > #:prop type-info >> >> > [(fresh-type-variable) >> >> > (? (n t) >> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> > 'body (fresh-type-variable)))]] >> >> > [SubProgram #f >> >> > ([decls : MyDefinition *] >> >> > [body : Expression]) >> >> > #:prop type-info >> >> > [(fresh-type-variable) >> >> > (? (n t) >> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> > 'body (fresh-type-variable)))]] >> >> > [MyDefinition #f ([type] >> >> > [name = (fresh-var-name "b_")] >> >> > [e : ProcedureApplication]) >> >> > #:prop binder-info () >> >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash >> 'e >> >> t))]]) >> >> > >> >> >(add-property >> >> > somelisp >> >> > lift-type->ast-binder-type >> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >> >> > >> >> >;; Sometimes we have a type variable that is not concrete but needs >> to be. >> >> >;; Here we provide a list of options we can pick for an unconstrained >> >> >;; type variable. >> >> >(define (type-thunks-for-concretization) >> >> > (list >> >> > (?()int-type) >> >> > (?()bool-type) >> >> > (?()string-type) >> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> > >> >> >(define (somelisp-format-render s-exps) >> >> > (define out (open-output-string)) >> >> > (for ([symex s-exps]) >> >> > (pretty-print symex out 1)) >> >> > (get-output-string out)) >> >> > >> >> >(define-xsmith-interface-functions >> >> > [somelisp] >> >> > #:program-node MyProgram >> >> > #:type-thunks type-thunks-for-concretization >> >> > #:comment-wrap (? (lines) >> >> > (string-join >> >> > (map (? (l) (format ";; ~a" l)) lines) >> >> > "\n")) >> >> > #:format-render somelisp-format-render) >> >> > >> >> >(module+ main (somelisp-command-line)) >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Tue Jul 20 10:34:38 2021 From: william at hatch.uno (William G Hatch) Date: Tue, 20 Jul 2021 10:34:38 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: >> Do you have a roadmap for implementing ... Not at the moment. Right now I'm focused on getting some of our mostly-finished fuzzers *really* finished and ready for fuzzing, running some fuzzing campaigns to find some bugs, and some features that are immediately helpful for running fuzzing (eg. some stuff to support feedback-directed fuzzing and choice weighting). >Also, I'm curious about the setup of various fuzzers that you are running. >How many programs are you able to generate and test per minute? Do you have >any particular parallelization setup that works well for you? There is a harness repository that we use that was mostly written by Guy Watson: https://gitlab.flux.utah.edu/xsmith/harness There is a `harness.py` script that takes a configuration file to run an xsmith fuzzer, running its output through various implementations of whatever language, comparing the output, and binning the result (eg. everything matching, diff, crash, everything crashing (IE bad program), xsmith itself crashing, ...). That harness script is basically for running a single job, and there are 2 ways of doing it in parallel. The first one is the `go-button`, which can run fuzzing on multiple machines at once. It is fairly dependent on Emulab/Cloudlab (https://www.emulab.net/) to have things set up just right, and requires at least 2 machines. It also generates a bunch of live graphs about fuzzing. It uses ansible to run setup scripts to prep each machine before starting the fuzz campaign. I mostly use `local-multi-harness` instead, which just runs on a single machine, but runs multiple harness processes. It doesn't automatically run the setup scripts like the `go-button` does, so eg. if you want to use the `racket-kernel-fuzzer` configuration you need to run the `racket-install.rkt` script first manually. As far as speed, xsmith is not particularly fast. There is a server mode that at least saves on startup time, but I think the harness might not run it particularly well (Guy correct me if I'm wrong -- at any rate I haven't been using it with server mode, but that might be because I never bothered changing my config after bugs were fixed). Aside from Racket's slow startup speed (particularly bad with Xsmith because it loads a lot of stuff), the runtime varies a lot based on the max-depth configuration, the size of the grammar, etc. I haven't focused a lot on performance, necessarily. Basically whenever I decide Xsmith is too slow for my taste I spend some time trying to optimize it, and generally make a few gains until I think ?that's enough optimization for now?. If you can find any ways to make improvements, that would be great! I might have taken all of the low-hanging fruit in my previous optimization passes, however. In previous optimization passes I got the most out of optimizing in the type system, but I made some major improvements to type checking speed last time so it may not dominate anymore. As an aside, a question for you: Are there multiple Dafny implementations? Or different optimization levels? Or are you running different versions of the compiler against each other? On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase wrote: >Also, I'm curious about the setup of various fuzzers that you are running. >How many programs are you able to generate and test per minute? Do you have >any particular parallelization setup that works well for you? > >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >sorawee.pwase at gmail.com> wrote: > >> Do you have a roadmap for implementing "Provide mechanism for precluding >> types in `fresh-type-variable` invocations"? ( >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really >> want this feature. >> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch wrote: >> >>> That would be great, thanks! >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase wrote: >>> >Yes, this is very helpful. I will let you know if I encounter any further >>> >issues after updating XSmith. >>> > >>> >By the way, do you want any help with Scribble? I notice a lot of typos >>> and >>> >link errors and can submit PRs to fix them. >>> > >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >>> wrote: >>> > >>> >> Sorry for the delay. I wanted to do the debugging before responding, >>> >> but then I didn't get to it for a while. >>> >> >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >>> >> >>> >> First, that issue with the bud-node was a bug in the `edit` property >>> >> which the canned-components `ProcedureApplication` uses. It's fixed >>> >> now. >>> >> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >>> >> >> function application node that only allows a reference as its RHS >>> >> >> rather than allowing arbitrary expressions. But that said, I'm not >>> >> >> sure I fully understand what you're trying to do. >>> >> >> >>> >> >I?m not sure what you mean by that. But: >>> >> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be function >>> >> application that only allows a reference in the procedure position. I >>> >> don't know why I wrote RHS. >>> >> >>> >> Anyway, the strategy of having separate definition nodes for functions >>> >> and everything else is used in Cish. I don't recommend following >>> >> anything Cish does. However, it accomplishes this by having the >>> >> program node have non-function definitions first, and disallowing use >>> >> of functions on the RHS of those functions. >>> >> >>> >> The problem where it can't find a reference is a bug (that I haven't >>> >> fixed yet) caused by the lifting mechanics not really being completely >>> >> in sync with the reference mechanics with respect to nodes that have >>> >> multiple definition fields. When there are multiple definition >>> >> fields, definitions in the lower field can reference the definitions >>> >> in the higher field, but not vice versa. The lifting machinery >>> >> doesn't know about that ordering, it just stuffs the lifted definition >>> >> into the front of the definitions in whatever field it is targetting. >>> >> Thus it places a definition in a place that the reference machinery >>> >> thinks is out of scope. >>> >> >>> >> So... if you want to have different binders that get different fields >>> >> in the program node you can only make it work if you disallow stuff to >>> >> be sure you never need to reference something that will be lifted into >>> >> the lower field. >>> >> >>> >> That said, I've just fixed some issues to allow definition nodes to be >>> >> subtypable, so now you can have a definition node that is a subtype of >>> >> Definition, but has some different rules. Importantly, though, it can >>> >> be lifted to the same field. >>> >> >>> >> Eg. if I change the MyDefinition declaration in your example to look >>> >> like this: >>> >> >>> >> ``` >>> >> [MyDefinition Definition () >>> >> #:prop fresh (hash 'Expression (make-hole >>> >> 'ProcedureApplication))] >>> >> ``` >>> >> >>> >> It can now be lifted among normal definitions, but always have >>> >> ProcedureApplication as its RHS. >>> >> >>> >> However, I'm still not convinced that you actually want >>> >> ProcedureApplication as the RHS. Maybe you want to restrict lambda >>> >> nodes to only be generated on the RHS of a FunctionDefinition node >>> >> (that's a subtype of Definition), and make FunctionDefinition always >>> >> have a Lambda node as its RHS. >>> >> >>> >> There is an example in the documentation that is like this, in the >>> >> explanation of the `choice-filters-to-apply` property. But I'll paste >>> >> it here: >>> >> >>> >> ``` >>> >> (add-choice-method >>> >> my-component >>> >> no-lambda-except-global-def >>> >> [#f (? () #t)] >>> >> [Lambda (? () >>> >> (and (parent-node current-hole) >>> >> (equal? (ast-node-type (parent-node current-hole)) >>> >> 'Definition) >>> >> (parent-node (parent-node current-hole)) >>> >> (equal? (ast-node-type (parent-node (parent-node >>> >> current-hole))) >>> >> 'Program)))]) >>> >> (add-property my-component >>> >> choice-filters-to-apply >>> >> [#f (no-lambda-except-global-def)]) >>> >> ``` >>> >> >>> >> You can use something like that to allow Lambda nodes to only be >>> >> generated on the RHS of FunctionDefinition nodes, and define >>> >> FunctionDefinition like MyDefinition above to always generate Lambda >>> >> as its RHS, and I think it should work for a first-order language. >>> >> >>> >> >>> >> >(Also note that I tried unify! these unrestricted type variables >>> >> >with some concrete non-function type variables, but down the road, I >>> get >>> >> an >>> >> >internal error about unification failure.). Perhaps I'm not supposed >>> to >>> >> >unify! inside add-choice-method? >>> >> >>> >> Well, probably not. Generally the only place you want to `unify!` is >>> >> inside the type property in the function that determines child types >>> >> based on the parent type. Inside that function you can `unify!`, but >>> >> the unification needs to be consistent (IE not unified based on the >>> >> result of `random`) OR the choice needs to be saved in a way that it >>> >> can be looked up when that function is run again. >>> >> >>> >> The problem is that the choice method gets its type information based >>> >> on a hole node that is later replaced, so the type is recomputed >>> >> without necessarily consulting the type that the hole said it was. >>> >> You should be able to constrain the type using unification if you save >>> >> that type to re-unify it consistently later. But without seeing the >>> >> exact error and code that generated it, I'm not 100% certain. >>> >> >>> >> >>> >> > In the last email, I asked if lift-type->ast-binder-type could be >>> used >>> >> > nondeterministically in this manner, and my experience with it so far >>> >> seems >>> >> > to indicate no. >>> >> >>> >> I think this should work, barring the issues mentioned above about the >>> >> lifting vs referencing bug when you have the different definition >>> >> types in different fields. Adjusting your example to have >>> >> MyDefinition as a subtype, and then only having a Definition field in >>> >> MyProgram, random choice seems to work fine. But sometimes it >>> >> generates gargantuan programs (that I kill before it finishes) because >>> >> it keeps choosing MyDefinition, requiring a function application, >>> >> which likely lifts a reference to a function, which might choose >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and >>> >> bigger types. >>> >> >>> >> >>> >> Anyway, does that all help? >>> >> >>> >> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase >>> wrote: >>> >> >> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >>> >> >> function application node that only allows a reference as its RHS >>> >> >> rather than allowing arbitrary expressions. But that said, I'm not >>> >> >> sure I fully understand what you're trying to do. >>> >> >> >>> >> >I?m not sure what you mean by that. But: >>> >> > >>> >> > 1. I have no problem with the definition site. That?s easy to deal >>> >> with. >>> >> > 2. I also have no problem with restricting function application. >>> That?s >>> >> > easy to deal with as well. >>> >> > 3. The problem is primarily with a bare variable reference bound >>> to a >>> >> > closure value, which must only occur in function application. For >>> >> example, >>> >> > the following code is invalid. >>> >> > >>> >> >(define (f) (void)) >>> >> > >>> >> >(define g f) ;; this is invalid >>> >> >(define h (g)) >>> >> >(define a (list f)) ;; this is also invalid >>> >> >(define b (f)) ;; this is ok >>> >> > >>> >> >I have been using choice-filters-to-apply to restrict variable >>> references >>> >> >whose type can-unify? with function-type that doesn't directly appear >>> >> under >>> >> >function application. But this is obviously not ideal because the >>> filtered >>> >> >nodes are mostly those whose type is totally unrestricted. So >>> filtering >>> >> >them would prevent them from being concretized to other non-function >>> types >>> >> >as well. (Also note that I tried unify! these unrestricted type >>> variables >>> >> >with some concrete non-function type variables, but down the road, I >>> get >>> >> an >>> >> >internal error about unification failure.). Perhaps I'm not supposed >>> to >>> >> >unify! inside add-choice-method? >>> >> > >>> >> >That's probably a bug. ?Lifting? is what I've called the situation >>> >> >> where a reference is generated that causes a new definition to be >>> >> >> added such that the variable is in scope at the use site. IE the >>> >> >> binding is lifted from the use site to some outer node that contains >>> >> >> definitions. `lift_X` is the name Xsmith gives to definitions that >>> >> >> are created automatically in this way (which is ultimately most >>> >> >> definitions). The exception is caused because somehow the lift >>> ended >>> >> >> up in a place where it wasn't actually in scope. That shouldn't >>> >> >> happen. Anyway, if you point me to a version of a fuzzer that does >>> >> >> this, I'll try to debug it. >>> >> >> >>> >> >#lang clotho >>> >> > >>> >> >(require >>> >> > xsmith >>> >> > racr >>> >> > xsmith/racr-convenience >>> >> > xsmith/canned-components >>> >> > racket/string >>> >> > racket/list >>> >> > racket/pretty) >>> >> > >>> >> >;; We first define a basic component and add a bunch of expressions. >>> >> > >>> >> >(define-basic-spec-component somelisp) >>> >> > >>> >> >(add-basic-expressions somelisp >>> >> > #:ProgramWithSequence #t >>> >> > #:VoidExpression #t >>> >> > #:AssignmentExpression #t >>> >> > #:VariableReference #t >>> >> > #:ProcedureApplication #t >>> >> > #:IfExpression #t >>> >> > #:ExpressionSequence #t >>> >> > #:LetSequential #t >>> >> > #:LambdaWithExpression #t >>> >> > #:Numbers #t >>> >> > #:Booleans #t >>> >> > #:Strings #t >>> >> > #:ImmutableList #t) >>> >> > >>> >> >(add-loop-over-container >>> >> > somelisp >>> >> > #:name Map >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >>> inner))) >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >>> >> > >>> >> >(add-to-grammar >>> >> > somelisp >>> >> > [MyProgram #f >>> >> > ([decls : Definition *] >>> >> > [declsTwo : MyDefinition *] >>> >> > [body : Expression]) >>> >> > #:prop type-info >>> >> > [(fresh-type-variable) >>> >> > (? (n t) >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >> > 'declsTwo (? (c) (fresh-type-variable)) >>> >> > 'body (fresh-type-variable)))]] >>> >> > [MyDefinition #f ([type] >>> >> > [name = (fresh-var-name "b_")] >>> >> > [e : ProcedureApplication]) >>> >> > #:prop binder-info () >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash >>> 'e >>> >> t))]]) >>> >> > >>> >> >(add-property >>> >> > somelisp >>> >> > lift-type->ast-binder-type >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >>> >> > >>> >> >;; Sometimes we have a type variable that is not concrete but needs >>> to be. >>> >> >;; Here we provide a list of options we can pick for an unconstrained >>> >> >;; type variable. >>> >> >(define (type-thunks-for-concretization) >>> >> > (list >>> >> > (?()int-type) >>> >> > (?()bool-type) >>> >> > (?()string-type) >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >>> >> > >>> >> >(define (somelisp-format-render s-exps) >>> >> > (define out (open-output-string)) >>> >> > (for ([symex s-exps]) >>> >> > (pretty-print symex out 1)) >>> >> > (get-output-string out)) >>> >> > >>> >> >(define-xsmith-interface-functions >>> >> > [somelisp] >>> >> > #:program-node MyProgram >>> >> > #:type-thunks type-thunks-for-concretization >>> >> > #:comment-wrap (? (lines) >>> >> > (string-join >>> >> > (map (? (l) (format ";; ~a" l)) lines) >>> >> > "\n")) >>> >> > #:format-render somelisp-format-render) >>> >> > >>> >> >(module+ main (somelisp-command-line)) >>> >> > >>> >> > >>> >> >> Probably something is going wrong in the `fresh` property. Have you >>> >> >> changed that? >>> >> >> >>> >> >I don?t think so. See the code at the end of the email. >>> >> > >>> >> >> I should change that error message. Anyway, it seems to think there >>> >> >> is no outer node that has a `Definition *` child that can be the >>> >> >> target for lifting a definition. Generally this happens when you >>> >> >> forget to have an outer Program node or something that can hold >>> >> >> definitions. >>> >> >> >>> >> >I don?t think that?s the case, but I could be wrong. >>> >> > >>> >> >#lang clotho >>> >> > >>> >> >(require >>> >> > xsmith >>> >> > racr >>> >> > xsmith/racr-convenience >>> >> > xsmith/canned-components >>> >> > racket/string >>> >> > racket/list >>> >> > racket/pretty) >>> >> > >>> >> >;; We first define a basic component and add a bunch of expressions. >>> >> > >>> >> >(define-basic-spec-component somelisp) >>> >> > >>> >> >(add-basic-expressions somelisp >>> >> > #:ProgramWithSequence #t >>> >> > #:VoidExpression #t >>> >> > #:AssignmentExpression #t >>> >> > #:VariableReference #t >>> >> > #:ProcedureApplication #t >>> >> > #:IfExpression #t >>> >> > #:ExpressionSequence #t >>> >> > #:LetSequential #t >>> >> > #:LambdaWithExpression #t >>> >> > #:Numbers #t >>> >> > #:Booleans #t >>> >> > #:Strings #t >>> >> > #:ImmutableList #t) >>> >> > >>> >> >(add-loop-over-container >>> >> > somelisp >>> >> > #:name Map >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >>> inner))) >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) >>> >> > >>> >> >(add-to-grammar >>> >> > somelisp >>> >> > [MyProgram #f >>> >> > ([decls : Definition *] >>> >> > [body : SubProgram]) >>> >> > #:prop type-info >>> >> > [(fresh-type-variable) >>> >> > (? (n t) >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >> > 'body (fresh-type-variable)))]] >>> >> > [SubProgram #f >>> >> > ([decls : MyDefinition *] >>> >> > [body : Expression]) >>> >> > #:prop type-info >>> >> > [(fresh-type-variable) >>> >> > (? (n t) >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >> > 'body (fresh-type-variable)))]] >>> >> > [MyDefinition #f ([type] >>> >> > [name = (fresh-var-name "b_")] >>> >> > [e : ProcedureApplication]) >>> >> > #:prop binder-info () >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) (hash >>> 'e >>> >> t))]]) >>> >> > >>> >> >(add-property >>> >> > somelisp >>> >> > lift-type->ast-binder-type >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) >>> >> > >>> >> >;; Sometimes we have a type variable that is not concrete but needs >>> to be. >>> >> >;; Here we provide a list of options we can pick for an unconstrained >>> >> >;; type variable. >>> >> >(define (type-thunks-for-concretization) >>> >> > (list >>> >> > (?()int-type) >>> >> > (?()bool-type) >>> >> > (?()string-type) >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >>> >> > >>> >> >(define (somelisp-format-render s-exps) >>> >> > (define out (open-output-string)) >>> >> > (for ([symex s-exps]) >>> >> > (pretty-print symex out 1)) >>> >> > (get-output-string out)) >>> >> > >>> >> >(define-xsmith-interface-functions >>> >> > [somelisp] >>> >> > #:program-node MyProgram >>> >> > #:type-thunks type-thunks-for-concretization >>> >> > #:comment-wrap (? (lines) >>> >> > (string-join >>> >> > (map (? (l) (format ";; ~a" l)) lines) >>> >> > "\n")) >>> >> > #:format-render somelisp-format-render) >>> >> > >>> >> >(module+ main (somelisp-command-line)) >>> >> >>> >> From sorawee.pwase at gmail.com Tue Jul 20 10:57:43 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 20 Jul 2021 09:57:43 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: We are primarily focused on different compiler backends for Dafny (Dafny can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). On Tue, Jul 20, 2021 at 9:34 AM William G Hatch wrote: > >> Do you have a roadmap for implementing ... > > Not at the moment. Right now I'm focused on getting some of our > mostly-finished fuzzers *really* finished and ready for fuzzing, > running some fuzzing campaigns to find some bugs, and some features > that are immediately helpful for running fuzzing (eg. some stuff to > support feedback-directed fuzzing and choice weighting). > > > >Also, I'm curious about the setup of various fuzzers that you are running. > >How many programs are you able to generate and test per minute? Do you > have > >any particular parallelization setup that works well for you? > > There is a harness repository that we use that was mostly written by > Guy Watson: > > https://gitlab.flux.utah.edu/xsmith/harness > > There is a `harness.py` script that takes a configuration file to run > an xsmith fuzzer, running its output through various implementations > of whatever language, comparing the output, and binning the result > (eg. everything matching, diff, crash, everything crashing (IE bad > program), xsmith itself crashing, ...). That harness script is > basically for running a single job, and there are 2 ways of doing it > in parallel. > > The first one is the `go-button`, which can run fuzzing on multiple > machines at once. It is fairly dependent on Emulab/Cloudlab > (https://www.emulab.net/) to have things set up just right, and > requires at least 2 machines. It also generates a bunch of live > graphs about fuzzing. It uses ansible to run setup scripts to prep > each machine before starting the fuzz campaign. > > I mostly use `local-multi-harness` instead, which just runs on a > single machine, but runs multiple harness processes. It doesn't > automatically run the setup scripts like the `go-button` does, so > eg. if you want to use the `racket-kernel-fuzzer` configuration you > need to run the `racket-install.rkt` script first manually. > > As far as speed, xsmith is not particularly fast. There is a server > mode that at least saves on startup time, but I think the harness > might not run it particularly well (Guy correct me if I'm wrong -- at > any rate I haven't been using it with server mode, but that might be > because I never bothered changing my config after bugs were fixed). > Aside from Racket's slow startup speed (particularly bad with Xsmith > because it loads a lot of stuff), the runtime varies a lot based on > the max-depth configuration, the size of the grammar, etc. > > I haven't focused a lot on performance, necessarily. Basically > whenever I decide Xsmith is too slow for my taste I spend some time > trying to optimize it, and generally make a few gains until I think > ?that's enough optimization for now?. If you can find any ways to > make improvements, that would be great! I might have taken all of the > low-hanging fruit in my previous optimization passes, however. In > previous optimization passes I got the most out of optimizing in the > type system, but I made some major improvements to type checking speed > last time so it may not dominate anymore. > > > As an aside, a question for you: Are there multiple Dafny > implementations? Or different optimization levels? Or are you > running different versions of the compiler against each other? > > > > On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase wrote: > >Also, I'm curious about the setup of various fuzzers that you are running. > >How many programs are you able to generate and test per minute? Do you > have > >any particular parallelization setup that works well for you? > > > >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < > >sorawee.pwase at gmail.com> wrote: > > > >> Do you have a roadmap for implementing "Provide mechanism for precluding > >> types in `fresh-type-variable` invocations"? ( > >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really > >> want this feature. > >> > >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch > wrote: > >> > >>> That would be great, thanks! > >>> > >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase > wrote: > >>> >Yes, this is very helpful. I will let you know if I encounter any > further > >>> >issues after updating XSmith. > >>> > > >>> >By the way, do you want any help with Scribble? I notice a lot of > typos > >>> and > >>> >link errors and can submit PRs to fix them. > >>> > > >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > >>> wrote: > >>> > > >>> >> Sorry for the delay. I wanted to do the debugging before > responding, > >>> >> but then I didn't get to it for a while. > >>> >> > >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. > >>> >> > >>> >> First, that issue with the bud-node was a bug in the `edit` property > >>> >> which the canned-components `ProcedureApplication` uses. It's fixed > >>> >> now. > >>> >> > >>> >> >> Hmm, it seems like first-order functions would be encoded as a > >>> >> >> function application node that only allows a reference as its RHS > >>> >> >> rather than allowing arbitrary expressions. But that said, I'm > not > >>> >> >> sure I fully understand what you're trying to do. > >>> >> >> > >>> >> >I?m not sure what you mean by that. But: > >>> >> > >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be function > >>> >> application that only allows a reference in the procedure > position. I > >>> >> don't know why I wrote RHS. > >>> >> > >>> >> Anyway, the strategy of having separate definition nodes for > functions > >>> >> and everything else is used in Cish. I don't recommend following > >>> >> anything Cish does. However, it accomplishes this by having the > >>> >> program node have non-function definitions first, and disallowing > use > >>> >> of functions on the RHS of those functions. > >>> >> > >>> >> The problem where it can't find a reference is a bug (that I haven't > >>> >> fixed yet) caused by the lifting mechanics not really being > completely > >>> >> in sync with the reference mechanics with respect to nodes that have > >>> >> multiple definition fields. When there are multiple definition > >>> >> fields, definitions in the lower field can reference the definitions > >>> >> in the higher field, but not vice versa. The lifting machinery > >>> >> doesn't know about that ordering, it just stuffs the lifted > definition > >>> >> into the front of the definitions in whatever field it is > targetting. > >>> >> Thus it places a definition in a place that the reference machinery > >>> >> thinks is out of scope. > >>> >> > >>> >> So... if you want to have different binders that get different > fields > >>> >> in the program node you can only make it work if you disallow stuff > to > >>> >> be sure you never need to reference something that will be lifted > into > >>> >> the lower field. > >>> >> > >>> >> That said, I've just fixed some issues to allow definition nodes to > be > >>> >> subtypable, so now you can have a definition node that is a subtype > of > >>> >> Definition, but has some different rules. Importantly, though, it > can > >>> >> be lifted to the same field. > >>> >> > >>> >> Eg. if I change the MyDefinition declaration in your example to look > >>> >> like this: > >>> >> > >>> >> ``` > >>> >> [MyDefinition Definition () > >>> >> #:prop fresh (hash 'Expression (make-hole > >>> >> 'ProcedureApplication))] > >>> >> ``` > >>> >> > >>> >> It can now be lifted among normal definitions, but always have > >>> >> ProcedureApplication as its RHS. > >>> >> > >>> >> However, I'm still not convinced that you actually want > >>> >> ProcedureApplication as the RHS. Maybe you want to restrict lambda > >>> >> nodes to only be generated on the RHS of a FunctionDefinition node > >>> >> (that's a subtype of Definition), and make FunctionDefinition always > >>> >> have a Lambda node as its RHS. > >>> >> > >>> >> There is an example in the documentation that is like this, in the > >>> >> explanation of the `choice-filters-to-apply` property. But I'll > paste > >>> >> it here: > >>> >> > >>> >> ``` > >>> >> (add-choice-method > >>> >> my-component > >>> >> no-lambda-except-global-def > >>> >> [#f (? () #t)] > >>> >> [Lambda (? () > >>> >> (and (parent-node current-hole) > >>> >> (equal? (ast-node-type (parent-node current-hole)) > >>> >> 'Definition) > >>> >> (parent-node (parent-node current-hole)) > >>> >> (equal? (ast-node-type (parent-node (parent-node > >>> >> current-hole))) > >>> >> 'Program)))]) > >>> >> (add-property my-component > >>> >> choice-filters-to-apply > >>> >> [#f (no-lambda-except-global-def)]) > >>> >> ``` > >>> >> > >>> >> You can use something like that to allow Lambda nodes to only be > >>> >> generated on the RHS of FunctionDefinition nodes, and define > >>> >> FunctionDefinition like MyDefinition above to always generate Lambda > >>> >> as its RHS, and I think it should work for a first-order language. > >>> >> > >>> >> > >>> >> >(Also note that I tried unify! these unrestricted type variables > >>> >> >with some concrete non-function type variables, but down the road, > I > >>> get > >>> >> an > >>> >> >internal error about unification failure.). Perhaps I'm not > supposed > >>> to > >>> >> >unify! inside add-choice-method? > >>> >> > >>> >> Well, probably not. Generally the only place you want to `unify!` > is > >>> >> inside the type property in the function that determines child types > >>> >> based on the parent type. Inside that function you can `unify!`, > but > >>> >> the unification needs to be consistent (IE not unified based on the > >>> >> result of `random`) OR the choice needs to be saved in a way that it > >>> >> can be looked up when that function is run again. > >>> >> > >>> >> The problem is that the choice method gets its type information > based > >>> >> on a hole node that is later replaced, so the type is recomputed > >>> >> without necessarily consulting the type that the hole said it was. > >>> >> You should be able to constrain the type using unification if you > save > >>> >> that type to re-unify it consistently later. But without seeing the > >>> >> exact error and code that generated it, I'm not 100% certain. > >>> >> > >>> >> > >>> >> > In the last email, I asked if lift-type->ast-binder-type could be > >>> used > >>> >> > nondeterministically in this manner, and my experience with it so > far > >>> >> seems > >>> >> > to indicate no. > >>> >> > >>> >> I think this should work, barring the issues mentioned above about > the > >>> >> lifting vs referencing bug when you have the different definition > >>> >> types in different fields. Adjusting your example to have > >>> >> MyDefinition as a subtype, and then only having a Definition field > in > >>> >> MyProgram, random choice seems to work fine. But sometimes it > >>> >> generates gargantuan programs (that I kill before it finishes) > because > >>> >> it keeps choosing MyDefinition, requiring a function application, > >>> >> which likely lifts a reference to a function, which might choose > >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and > >>> >> bigger types. > >>> >> > >>> >> > >>> >> Anyway, does that all help? > >>> >> > >>> >> > >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase > >>> wrote: > >>> >> >> > >>> >> >> Hmm, it seems like first-order functions would be encoded as a > >>> >> >> function application node that only allows a reference as its RHS > >>> >> >> rather than allowing arbitrary expressions. But that said, I'm > not > >>> >> >> sure I fully understand what you're trying to do. > >>> >> >> > >>> >> >I?m not sure what you mean by that. But: > >>> >> > > >>> >> > 1. I have no problem with the definition site. That?s easy to > deal > >>> >> with. > >>> >> > 2. I also have no problem with restricting function application. > >>> That?s > >>> >> > easy to deal with as well. > >>> >> > 3. The problem is primarily with a bare variable reference bound > >>> to a > >>> >> > closure value, which must only occur in function application. > For > >>> >> example, > >>> >> > the following code is invalid. > >>> >> > > >>> >> >(define (f) (void)) > >>> >> > > >>> >> >(define g f) ;; this is invalid > >>> >> >(define h (g)) > >>> >> >(define a (list f)) ;; this is also invalid > >>> >> >(define b (f)) ;; this is ok > >>> >> > > >>> >> >I have been using choice-filters-to-apply to restrict variable > >>> references > >>> >> >whose type can-unify? with function-type that doesn't directly > appear > >>> >> under > >>> >> >function application. But this is obviously not ideal because the > >>> filtered > >>> >> >nodes are mostly those whose type is totally unrestricted. So > >>> filtering > >>> >> >them would prevent them from being concretized to other > non-function > >>> types > >>> >> >as well. (Also note that I tried unify! these unrestricted type > >>> variables > >>> >> >with some concrete non-function type variables, but down the road, > I > >>> get > >>> >> an > >>> >> >internal error about unification failure.). Perhaps I'm not > supposed > >>> to > >>> >> >unify! inside add-choice-method? > >>> >> > > >>> >> >That's probably a bug. ?Lifting? is what I've called the situation > >>> >> >> where a reference is generated that causes a new definition to be > >>> >> >> added such that the variable is in scope at the use site. IE the > >>> >> >> binding is lifted from the use site to some outer node that > contains > >>> >> >> definitions. `lift_X` is the name Xsmith gives to definitions > that > >>> >> >> are created automatically in this way (which is ultimately most > >>> >> >> definitions). The exception is caused because somehow the lift > >>> ended > >>> >> >> up in a place where it wasn't actually in scope. That shouldn't > >>> >> >> happen. Anyway, if you point me to a version of a fuzzer that > does > >>> >> >> this, I'll try to debug it. > >>> >> >> > >>> >> >#lang clotho > >>> >> > > >>> >> >(require > >>> >> > xsmith > >>> >> > racr > >>> >> > xsmith/racr-convenience > >>> >> > xsmith/canned-components > >>> >> > racket/string > >>> >> > racket/list > >>> >> > racket/pretty) > >>> >> > > >>> >> >;; We first define a basic component and add a bunch of > expressions. > >>> >> > > >>> >> >(define-basic-spec-component somelisp) > >>> >> > > >>> >> >(add-basic-expressions somelisp > >>> >> > #:ProgramWithSequence #t > >>> >> > #:VoidExpression #t > >>> >> > #:AssignmentExpression #t > >>> >> > #:VariableReference #t > >>> >> > #:ProcedureApplication #t > >>> >> > #:IfExpression #t > >>> >> > #:ExpressionSequence #t > >>> >> > #:LetSequential #t > >>> >> > #:LambdaWithExpression #t > >>> >> > #:Numbers #t > >>> >> > #:Booleans #t > >>> >> > #:Strings #t > >>> >> > #:ImmutableList #t) > >>> >> > > >>> >> >(add-loop-over-container > >>> >> > somelisp > >>> >> > #:name Map > >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type > >>> inner))) > >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >>> >> > > >>> >> >(add-to-grammar > >>> >> > somelisp > >>> >> > [MyProgram #f > >>> >> > ([decls : Definition *] > >>> >> > [declsTwo : MyDefinition *] > >>> >> > [body : Expression]) > >>> >> > #:prop type-info > >>> >> > [(fresh-type-variable) > >>> >> > (? (n t) > >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >> > 'declsTwo (? (c) (fresh-type-variable)) > >>> >> > 'body (fresh-type-variable)))]] > >>> >> > [MyDefinition #f ([type] > >>> >> > [name = (fresh-var-name "b_")] > >>> >> > [e : ProcedureApplication]) > >>> >> > #:prop binder-info () > >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) > (hash > >>> 'e > >>> >> t))]]) > >>> >> > > >>> >> >(add-property > >>> >> > somelisp > >>> >> > lift-type->ast-binder-type > >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >>> >> > > >>> >> >;; Sometimes we have a type variable that is not concrete but needs > >>> to be. > >>> >> >;; Here we provide a list of options we can pick for an > unconstrained > >>> >> >;; type variable. > >>> >> >(define (type-thunks-for-concretization) > >>> >> > (list > >>> >> > (?()int-type) > >>> >> > (?()bool-type) > >>> >> > (?()string-type) > >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >>> >> > > >>> >> >(define (somelisp-format-render s-exps) > >>> >> > (define out (open-output-string)) > >>> >> > (for ([symex s-exps]) > >>> >> > (pretty-print symex out 1)) > >>> >> > (get-output-string out)) > >>> >> > > >>> >> >(define-xsmith-interface-functions > >>> >> > [somelisp] > >>> >> > #:program-node MyProgram > >>> >> > #:type-thunks type-thunks-for-concretization > >>> >> > #:comment-wrap (? (lines) > >>> >> > (string-join > >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >>> >> > "\n")) > >>> >> > #:format-render somelisp-format-render) > >>> >> > > >>> >> >(module+ main (somelisp-command-line)) > >>> >> > > >>> >> > > >>> >> >> Probably something is going wrong in the `fresh` property. Have > you > >>> >> >> changed that? > >>> >> >> > >>> >> >I don?t think so. See the code at the end of the email. > >>> >> > > >>> >> >> I should change that error message. Anyway, it seems to think > there > >>> >> >> is no outer node that has a `Definition *` child that can be the > >>> >> >> target for lifting a definition. Generally this happens when you > >>> >> >> forget to have an outer Program node or something that can hold > >>> >> >> definitions. > >>> >> >> > >>> >> >I don?t think that?s the case, but I could be wrong. > >>> >> > > >>> >> >#lang clotho > >>> >> > > >>> >> >(require > >>> >> > xsmith > >>> >> > racr > >>> >> > xsmith/racr-convenience > >>> >> > xsmith/canned-components > >>> >> > racket/string > >>> >> > racket/list > >>> >> > racket/pretty) > >>> >> > > >>> >> >;; We first define a basic component and add a bunch of > expressions. > >>> >> > > >>> >> >(define-basic-spec-component somelisp) > >>> >> > > >>> >> >(add-basic-expressions somelisp > >>> >> > #:ProgramWithSequence #t > >>> >> > #:VoidExpression #t > >>> >> > #:AssignmentExpression #t > >>> >> > #:VariableReference #t > >>> >> > #:ProcedureApplication #t > >>> >> > #:IfExpression #t > >>> >> > #:ExpressionSequence #t > >>> >> > #:LetSequential #t > >>> >> > #:LambdaWithExpression #t > >>> >> > #:Numbers #t > >>> >> > #:Booleans #t > >>> >> > #:Strings #t > >>> >> > #:ImmutableList #t) > >>> >> > > >>> >> >(add-loop-over-container > >>> >> > somelisp > >>> >> > #:name Map > >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type > >>> inner))) > >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type inner)))) > >>> >> > > >>> >> >(add-to-grammar > >>> >> > somelisp > >>> >> > [MyProgram #f > >>> >> > ([decls : Definition *] > >>> >> > [body : SubProgram]) > >>> >> > #:prop type-info > >>> >> > [(fresh-type-variable) > >>> >> > (? (n t) > >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >> > 'body (fresh-type-variable)))]] > >>> >> > [SubProgram #f > >>> >> > ([decls : MyDefinition *] > >>> >> > [body : Expression]) > >>> >> > #:prop type-info > >>> >> > [(fresh-type-variable) > >>> >> > (? (n t) > >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >> > 'body (fresh-type-variable)))]] > >>> >> > [MyDefinition #f ([type] > >>> >> > [name = (fresh-var-name "b_")] > >>> >> > [e : ProcedureApplication]) > >>> >> > #:prop binder-info () > >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) > (hash > >>> 'e > >>> >> t))]]) > >>> >> > > >>> >> >(add-property > >>> >> > somelisp > >>> >> > lift-type->ast-binder-type > >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition 'Definition))]) > >>> >> > > >>> >> >;; Sometimes we have a type variable that is not concrete but needs > >>> to be. > >>> >> >;; Here we provide a list of options we can pick for an > unconstrained > >>> >> >;; type variable. > >>> >> >(define (type-thunks-for-concretization) > >>> >> > (list > >>> >> > (?()int-type) > >>> >> > (?()bool-type) > >>> >> > (?()string-type) > >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >>> >> > > >>> >> >(define (somelisp-format-render s-exps) > >>> >> > (define out (open-output-string)) > >>> >> > (for ([symex s-exps]) > >>> >> > (pretty-print symex out 1)) > >>> >> > (get-output-string out)) > >>> >> > > >>> >> >(define-xsmith-interface-functions > >>> >> > [somelisp] > >>> >> > #:program-node MyProgram > >>> >> > #:type-thunks type-thunks-for-concretization > >>> >> > #:comment-wrap (? (lines) > >>> >> > (string-join > >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >>> >> > "\n")) > >>> >> > #:format-render somelisp-format-render) > >>> >> > > >>> >> >(module+ main (somelisp-command-line)) > >>> >> > >>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Tue Jul 20 10:58:55 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 20 Jul 2021 09:58:55 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: By the way, here are the results so far: - https://github.com/dafny-lang/dafny/issues/1291 - https://github.com/dafny-lang/dafny/issues/1297 On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > We are primarily focused on different compiler backends for Dafny (Dafny > can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). > > On Tue, Jul 20, 2021 at 9:34 AM William G Hatch wrote: > >> >> Do you have a roadmap for implementing ... >> >> Not at the moment. Right now I'm focused on getting some of our >> mostly-finished fuzzers *really* finished and ready for fuzzing, >> running some fuzzing campaigns to find some bugs, and some features >> that are immediately helpful for running fuzzing (eg. some stuff to >> support feedback-directed fuzzing and choice weighting). >> >> >> >Also, I'm curious about the setup of various fuzzers that you are >> running. >> >How many programs are you able to generate and test per minute? Do you >> have >> >any particular parallelization setup that works well for you? >> >> There is a harness repository that we use that was mostly written by >> Guy Watson: >> >> https://gitlab.flux.utah.edu/xsmith/harness >> >> There is a `harness.py` script that takes a configuration file to run >> an xsmith fuzzer, running its output through various implementations >> of whatever language, comparing the output, and binning the result >> (eg. everything matching, diff, crash, everything crashing (IE bad >> program), xsmith itself crashing, ...). That harness script is >> basically for running a single job, and there are 2 ways of doing it >> in parallel. >> >> The first one is the `go-button`, which can run fuzzing on multiple >> machines at once. It is fairly dependent on Emulab/Cloudlab >> (https://www.emulab.net/) to have things set up just right, and >> requires at least 2 machines. It also generates a bunch of live >> graphs about fuzzing. It uses ansible to run setup scripts to prep >> each machine before starting the fuzz campaign. >> >> I mostly use `local-multi-harness` instead, which just runs on a >> single machine, but runs multiple harness processes. It doesn't >> automatically run the setup scripts like the `go-button` does, so >> eg. if you want to use the `racket-kernel-fuzzer` configuration you >> need to run the `racket-install.rkt` script first manually. >> >> As far as speed, xsmith is not particularly fast. There is a server >> mode that at least saves on startup time, but I think the harness >> might not run it particularly well (Guy correct me if I'm wrong -- at >> any rate I haven't been using it with server mode, but that might be >> because I never bothered changing my config after bugs were fixed). >> Aside from Racket's slow startup speed (particularly bad with Xsmith >> because it loads a lot of stuff), the runtime varies a lot based on >> the max-depth configuration, the size of the grammar, etc. >> >> I haven't focused a lot on performance, necessarily. Basically >> whenever I decide Xsmith is too slow for my taste I spend some time >> trying to optimize it, and generally make a few gains until I think >> ?that's enough optimization for now?. If you can find any ways to >> make improvements, that would be great! I might have taken all of the >> low-hanging fruit in my previous optimization passes, however. In >> previous optimization passes I got the most out of optimizing in the >> type system, but I made some major improvements to type checking speed >> last time so it may not dominate anymore. >> >> >> As an aside, a question for you: Are there multiple Dafny >> implementations? Or different optimization levels? Or are you >> running different versions of the compiler against each other? >> >> >> >> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase wrote: >> >Also, I'm curious about the setup of various fuzzers that you are >> running. >> >How many programs are you able to generate and test per minute? Do you >> have >> >any particular parallelization setup that works well for you? >> > >> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >> >sorawee.pwase at gmail.com> wrote: >> > >> >> Do you have a roadmap for implementing "Provide mechanism for >> precluding >> >> types in `fresh-type-variable` invocations"? ( >> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really >> >> want this feature. >> >> >> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >> wrote: >> >> >> >>> That would be great, thanks! >> >>> >> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase >> wrote: >> >>> >Yes, this is very helpful. I will let you know if I encounter any >> further >> >>> >issues after updating XSmith. >> >>> > >> >>> >By the way, do you want any help with Scribble? I notice a lot of >> typos >> >>> and >> >>> >link errors and can submit PRs to fix them. >> >>> > >> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >> >>> wrote: >> >>> > >> >>> >> Sorry for the delay. I wanted to do the debugging before >> responding, >> >>> >> but then I didn't get to it for a while. >> >>> >> >> >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >> >>> >> >> >>> >> First, that issue with the bud-node was a bug in the `edit` >> property >> >>> >> which the canned-components `ProcedureApplication` uses. It's >> fixed >> >>> >> now. >> >>> >> >> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >> >>> >> >> function application node that only allows a reference as its >> RHS >> >>> >> >> rather than allowing arbitrary expressions. But that said, I'm >> not >> >>> >> >> sure I fully understand what you're trying to do. >> >>> >> >> >> >>> >> >I?m not sure what you mean by that. But: >> >>> >> >> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be >> function >> >>> >> application that only allows a reference in the procedure >> position. I >> >>> >> don't know why I wrote RHS. >> >>> >> >> >>> >> Anyway, the strategy of having separate definition nodes for >> functions >> >>> >> and everything else is used in Cish. I don't recommend following >> >>> >> anything Cish does. However, it accomplishes this by having the >> >>> >> program node have non-function definitions first, and disallowing >> use >> >>> >> of functions on the RHS of those functions. >> >>> >> >> >>> >> The problem where it can't find a reference is a bug (that I >> haven't >> >>> >> fixed yet) caused by the lifting mechanics not really being >> completely >> >>> >> in sync with the reference mechanics with respect to nodes that >> have >> >>> >> multiple definition fields. When there are multiple definition >> >>> >> fields, definitions in the lower field can reference the >> definitions >> >>> >> in the higher field, but not vice versa. The lifting machinery >> >>> >> doesn't know about that ordering, it just stuffs the lifted >> definition >> >>> >> into the front of the definitions in whatever field it is >> targetting. >> >>> >> Thus it places a definition in a place that the reference machinery >> >>> >> thinks is out of scope. >> >>> >> >> >>> >> So... if you want to have different binders that get different >> fields >> >>> >> in the program node you can only make it work if you disallow >> stuff to >> >>> >> be sure you never need to reference something that will be lifted >> into >> >>> >> the lower field. >> >>> >> >> >>> >> That said, I've just fixed some issues to allow definition nodes >> to be >> >>> >> subtypable, so now you can have a definition node that is a >> subtype of >> >>> >> Definition, but has some different rules. Importantly, though, it >> can >> >>> >> be lifted to the same field. >> >>> >> >> >>> >> Eg. if I change the MyDefinition declaration in your example to >> look >> >>> >> like this: >> >>> >> >> >>> >> ``` >> >>> >> [MyDefinition Definition () >> >>> >> #:prop fresh (hash 'Expression (make-hole >> >>> >> 'ProcedureApplication))] >> >>> >> ``` >> >>> >> >> >>> >> It can now be lifted among normal definitions, but always have >> >>> >> ProcedureApplication as its RHS. >> >>> >> >> >>> >> However, I'm still not convinced that you actually want >> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict lambda >> >>> >> nodes to only be generated on the RHS of a FunctionDefinition node >> >>> >> (that's a subtype of Definition), and make FunctionDefinition >> always >> >>> >> have a Lambda node as its RHS. >> >>> >> >> >>> >> There is an example in the documentation that is like this, in the >> >>> >> explanation of the `choice-filters-to-apply` property. But I'll >> paste >> >>> >> it here: >> >>> >> >> >>> >> ``` >> >>> >> (add-choice-method >> >>> >> my-component >> >>> >> no-lambda-except-global-def >> >>> >> [#f (? () #t)] >> >>> >> [Lambda (? () >> >>> >> (and (parent-node current-hole) >> >>> >> (equal? (ast-node-type (parent-node current-hole)) >> >>> >> 'Definition) >> >>> >> (parent-node (parent-node current-hole)) >> >>> >> (equal? (ast-node-type (parent-node (parent-node >> >>> >> current-hole))) >> >>> >> 'Program)))]) >> >>> >> (add-property my-component >> >>> >> choice-filters-to-apply >> >>> >> [#f (no-lambda-except-global-def)]) >> >>> >> ``` >> >>> >> >> >>> >> You can use something like that to allow Lambda nodes to only be >> >>> >> generated on the RHS of FunctionDefinition nodes, and define >> >>> >> FunctionDefinition like MyDefinition above to always generate >> Lambda >> >>> >> as its RHS, and I think it should work for a first-order language. >> >>> >> >> >>> >> >> >>> >> >(Also note that I tried unify! these unrestricted type variables >> >>> >> >with some concrete non-function type variables, but down the >> road, I >> >>> get >> >>> >> an >> >>> >> >internal error about unification failure.). Perhaps I'm not >> supposed >> >>> to >> >>> >> >unify! inside add-choice-method? >> >>> >> >> >>> >> Well, probably not. Generally the only place you want to `unify!` >> is >> >>> >> inside the type property in the function that determines child >> types >> >>> >> based on the parent type. Inside that function you can `unify!`, >> but >> >>> >> the unification needs to be consistent (IE not unified based on the >> >>> >> result of `random`) OR the choice needs to be saved in a way that >> it >> >>> >> can be looked up when that function is run again. >> >>> >> >> >>> >> The problem is that the choice method gets its type information >> based >> >>> >> on a hole node that is later replaced, so the type is recomputed >> >>> >> without necessarily consulting the type that the hole said it was. >> >>> >> You should be able to constrain the type using unification if you >> save >> >>> >> that type to re-unify it consistently later. But without seeing >> the >> >>> >> exact error and code that generated it, I'm not 100% certain. >> >>> >> >> >>> >> >> >>> >> > In the last email, I asked if lift-type->ast-binder-type could be >> >>> used >> >>> >> > nondeterministically in this manner, and my experience with it >> so far >> >>> >> seems >> >>> >> > to indicate no. >> >>> >> >> >>> >> I think this should work, barring the issues mentioned above about >> the >> >>> >> lifting vs referencing bug when you have the different definition >> >>> >> types in different fields. Adjusting your example to have >> >>> >> MyDefinition as a subtype, and then only having a Definition field >> in >> >>> >> MyProgram, random choice seems to work fine. But sometimes it >> >>> >> generates gargantuan programs (that I kill before it finishes) >> because >> >>> >> it keeps choosing MyDefinition, requiring a function application, >> >>> >> which likely lifts a reference to a function, which might choose >> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and >> >>> >> bigger types. >> >>> >> >> >>> >> >> >>> >> Anyway, does that all help? >> >>> >> >> >>> >> >> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase >> >>> wrote: >> >>> >> >> >> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >> >>> >> >> function application node that only allows a reference as its >> RHS >> >>> >> >> rather than allowing arbitrary expressions. But that said, I'm >> not >> >>> >> >> sure I fully understand what you're trying to do. >> >>> >> >> >> >>> >> >I?m not sure what you mean by that. But: >> >>> >> > >> >>> >> > 1. I have no problem with the definition site. That?s easy to >> deal >> >>> >> with. >> >>> >> > 2. I also have no problem with restricting function >> application. >> >>> That?s >> >>> >> > easy to deal with as well. >> >>> >> > 3. The problem is primarily with a bare variable reference >> bound >> >>> to a >> >>> >> > closure value, which must only occur in function application. >> For >> >>> >> example, >> >>> >> > the following code is invalid. >> >>> >> > >> >>> >> >(define (f) (void)) >> >>> >> > >> >>> >> >(define g f) ;; this is invalid >> >>> >> >(define h (g)) >> >>> >> >(define a (list f)) ;; this is also invalid >> >>> >> >(define b (f)) ;; this is ok >> >>> >> > >> >>> >> >I have been using choice-filters-to-apply to restrict variable >> >>> references >> >>> >> >whose type can-unify? with function-type that doesn't directly >> appear >> >>> >> under >> >>> >> >function application. But this is obviously not ideal because the >> >>> filtered >> >>> >> >nodes are mostly those whose type is totally unrestricted. So >> >>> filtering >> >>> >> >them would prevent them from being concretized to other >> non-function >> >>> types >> >>> >> >as well. (Also note that I tried unify! these unrestricted type >> >>> variables >> >>> >> >with some concrete non-function type variables, but down the >> road, I >> >>> get >> >>> >> an >> >>> >> >internal error about unification failure.). Perhaps I'm not >> supposed >> >>> to >> >>> >> >unify! inside add-choice-method? >> >>> >> > >> >>> >> >That's probably a bug. ?Lifting? is what I've called the >> situation >> >>> >> >> where a reference is generated that causes a new definition to >> be >> >>> >> >> added such that the variable is in scope at the use site. IE >> the >> >>> >> >> binding is lifted from the use site to some outer node that >> contains >> >>> >> >> definitions. `lift_X` is the name Xsmith gives to definitions >> that >> >>> >> >> are created automatically in this way (which is ultimately most >> >>> >> >> definitions). The exception is caused because somehow the lift >> >>> ended >> >>> >> >> up in a place where it wasn't actually in scope. That shouldn't >> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer that >> does >> >>> >> >> this, I'll try to debug it. >> >>> >> >> >> >>> >> >#lang clotho >> >>> >> > >> >>> >> >(require >> >>> >> > xsmith >> >>> >> > racr >> >>> >> > xsmith/racr-convenience >> >>> >> > xsmith/canned-components >> >>> >> > racket/string >> >>> >> > racket/list >> >>> >> > racket/pretty) >> >>> >> > >> >>> >> >;; We first define a basic component and add a bunch of >> expressions. >> >>> >> > >> >>> >> >(define-basic-spec-component somelisp) >> >>> >> > >> >>> >> >(add-basic-expressions somelisp >> >>> >> > #:ProgramWithSequence #t >> >>> >> > #:VoidExpression #t >> >>> >> > #:AssignmentExpression #t >> >>> >> > #:VariableReference #t >> >>> >> > #:ProcedureApplication #t >> >>> >> > #:IfExpression #t >> >>> >> > #:ExpressionSequence #t >> >>> >> > #:LetSequential #t >> >>> >> > #:LambdaWithExpression #t >> >>> >> > #:Numbers #t >> >>> >> > #:Booleans #t >> >>> >> > #:Strings #t >> >>> >> > #:ImmutableList #t) >> >>> >> > >> >>> >> >(add-loop-over-container >> >>> >> > somelisp >> >>> >> > #:name Map >> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> >>> inner))) >> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> inner)))) >> >>> >> > >> >>> >> >(add-to-grammar >> >>> >> > somelisp >> >>> >> > [MyProgram #f >> >>> >> > ([decls : Definition *] >> >>> >> > [declsTwo : MyDefinition *] >> >>> >> > [body : Expression]) >> >>> >> > #:prop type-info >> >>> >> > [(fresh-type-variable) >> >>> >> > (? (n t) >> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) >> >>> >> > 'body (fresh-type-variable)))]] >> >>> >> > [MyDefinition #f ([type] >> >>> >> > [name = (fresh-var-name "b_")] >> >>> >> > [e : ProcedureApplication]) >> >>> >> > #:prop binder-info () >> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >> (hash >> >>> 'e >> >>> >> t))]]) >> >>> >> > >> >>> >> >(add-property >> >>> >> > somelisp >> >>> >> > lift-type->ast-binder-type >> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> 'Definition))]) >> >>> >> > >> >>> >> >;; Sometimes we have a type variable that is not concrete but >> needs >> >>> to be. >> >>> >> >;; Here we provide a list of options we can pick for an >> unconstrained >> >>> >> >;; type variable. >> >>> >> >(define (type-thunks-for-concretization) >> >>> >> > (list >> >>> >> > (?()int-type) >> >>> >> > (?()bool-type) >> >>> >> > (?()string-type) >> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >>> >> > >> >>> >> >(define (somelisp-format-render s-exps) >> >>> >> > (define out (open-output-string)) >> >>> >> > (for ([symex s-exps]) >> >>> >> > (pretty-print symex out 1)) >> >>> >> > (get-output-string out)) >> >>> >> > >> >>> >> >(define-xsmith-interface-functions >> >>> >> > [somelisp] >> >>> >> > #:program-node MyProgram >> >>> >> > #:type-thunks type-thunks-for-concretization >> >>> >> > #:comment-wrap (? (lines) >> >>> >> > (string-join >> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >>> >> > "\n")) >> >>> >> > #:format-render somelisp-format-render) >> >>> >> > >> >>> >> >(module+ main (somelisp-command-line)) >> >>> >> > >> >>> >> > >> >>> >> >> Probably something is going wrong in the `fresh` property. >> Have you >> >>> >> >> changed that? >> >>> >> >> >> >>> >> >I don?t think so. See the code at the end of the email. >> >>> >> > >> >>> >> >> I should change that error message. Anyway, it seems to think >> there >> >>> >> >> is no outer node that has a `Definition *` child that can be the >> >>> >> >> target for lifting a definition. Generally this happens when >> you >> >>> >> >> forget to have an outer Program node or something that can hold >> >>> >> >> definitions. >> >>> >> >> >> >>> >> >I don?t think that?s the case, but I could be wrong. >> >>> >> > >> >>> >> >#lang clotho >> >>> >> > >> >>> >> >(require >> >>> >> > xsmith >> >>> >> > racr >> >>> >> > xsmith/racr-convenience >> >>> >> > xsmith/canned-components >> >>> >> > racket/string >> >>> >> > racket/list >> >>> >> > racket/pretty) >> >>> >> > >> >>> >> >;; We first define a basic component and add a bunch of >> expressions. >> >>> >> > >> >>> >> >(define-basic-spec-component somelisp) >> >>> >> > >> >>> >> >(add-basic-expressions somelisp >> >>> >> > #:ProgramWithSequence #t >> >>> >> > #:VoidExpression #t >> >>> >> > #:AssignmentExpression #t >> >>> >> > #:VariableReference #t >> >>> >> > #:ProcedureApplication #t >> >>> >> > #:IfExpression #t >> >>> >> > #:ExpressionSequence #t >> >>> >> > #:LetSequential #t >> >>> >> > #:LambdaWithExpression #t >> >>> >> > #:Numbers #t >> >>> >> > #:Booleans #t >> >>> >> > #:Strings #t >> >>> >> > #:ImmutableList #t) >> >>> >> > >> >>> >> >(add-loop-over-container >> >>> >> > somelisp >> >>> >> > #:name Map >> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> >>> inner))) >> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> inner)))) >> >>> >> > >> >>> >> >(add-to-grammar >> >>> >> > somelisp >> >>> >> > [MyProgram #f >> >>> >> > ([decls : Definition *] >> >>> >> > [body : SubProgram]) >> >>> >> > #:prop type-info >> >>> >> > [(fresh-type-variable) >> >>> >> > (? (n t) >> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >> > 'body (fresh-type-variable)))]] >> >>> >> > [SubProgram #f >> >>> >> > ([decls : MyDefinition *] >> >>> >> > [body : Expression]) >> >>> >> > #:prop type-info >> >>> >> > [(fresh-type-variable) >> >>> >> > (? (n t) >> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >> > 'body (fresh-type-variable)))]] >> >>> >> > [MyDefinition #f ([type] >> >>> >> > [name = (fresh-var-name "b_")] >> >>> >> > [e : ProcedureApplication]) >> >>> >> > #:prop binder-info () >> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >> (hash >> >>> 'e >> >>> >> t))]]) >> >>> >> > >> >>> >> >(add-property >> >>> >> > somelisp >> >>> >> > lift-type->ast-binder-type >> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> 'Definition))]) >> >>> >> > >> >>> >> >;; Sometimes we have a type variable that is not concrete but >> needs >> >>> to be. >> >>> >> >;; Here we provide a list of options we can pick for an >> unconstrained >> >>> >> >;; type variable. >> >>> >> >(define (type-thunks-for-concretization) >> >>> >> > (list >> >>> >> > (?()int-type) >> >>> >> > (?()bool-type) >> >>> >> > (?()string-type) >> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >>> >> > >> >>> >> >(define (somelisp-format-render s-exps) >> >>> >> > (define out (open-output-string)) >> >>> >> > (for ([symex s-exps]) >> >>> >> > (pretty-print symex out 1)) >> >>> >> > (get-output-string out)) >> >>> >> > >> >>> >> >(define-xsmith-interface-functions >> >>> >> > [somelisp] >> >>> >> > #:program-node MyProgram >> >>> >> > #:type-thunks type-thunks-for-concretization >> >>> >> > #:comment-wrap (? (lines) >> >>> >> > (string-join >> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >>> >> > "\n")) >> >>> >> > #:format-render somelisp-format-render) >> >>> >> > >> >>> >> >(module+ main (somelisp-command-line)) >> >>> >> >> >>> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Tue Jul 20 11:44:57 2021 From: william at hatch.uno (William G Hatch) Date: Tue, 20 Jul 2021 11:44:57 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Awesome! We have a git repo where we are keeping track of bugs we find with Xsmith. If you want, we could add you to it and you can add any bugs you find to it. Otherwise I will probably put in anything you mention like this. On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase wrote: >By the way, here are the results so far: > >- https://github.com/dafny-lang/dafny/issues/1291 >- https://github.com/dafny-lang/dafny/issues/1297 > >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < >sorawee.pwase at gmail.com> wrote: > >> We are primarily focused on different compiler backends for Dafny (Dafny >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch wrote: >> >>> >> Do you have a roadmap for implementing ... >>> >>> Not at the moment. Right now I'm focused on getting some of our >>> mostly-finished fuzzers *really* finished and ready for fuzzing, >>> running some fuzzing campaigns to find some bugs, and some features >>> that are immediately helpful for running fuzzing (eg. some stuff to >>> support feedback-directed fuzzing and choice weighting). >>> >>> >>> >Also, I'm curious about the setup of various fuzzers that you are >>> running. >>> >How many programs are you able to generate and test per minute? Do you >>> have >>> >any particular parallelization setup that works well for you? >>> >>> There is a harness repository that we use that was mostly written by >>> Guy Watson: >>> >>> https://gitlab.flux.utah.edu/xsmith/harness >>> >>> There is a `harness.py` script that takes a configuration file to run >>> an xsmith fuzzer, running its output through various implementations >>> of whatever language, comparing the output, and binning the result >>> (eg. everything matching, diff, crash, everything crashing (IE bad >>> program), xsmith itself crashing, ...). That harness script is >>> basically for running a single job, and there are 2 ways of doing it >>> in parallel. >>> >>> The first one is the `go-button`, which can run fuzzing on multiple >>> machines at once. It is fairly dependent on Emulab/Cloudlab >>> (https://www.emulab.net/) to have things set up just right, and >>> requires at least 2 machines. It also generates a bunch of live >>> graphs about fuzzing. It uses ansible to run setup scripts to prep >>> each machine before starting the fuzz campaign. >>> >>> I mostly use `local-multi-harness` instead, which just runs on a >>> single machine, but runs multiple harness processes. It doesn't >>> automatically run the setup scripts like the `go-button` does, so >>> eg. if you want to use the `racket-kernel-fuzzer` configuration you >>> need to run the `racket-install.rkt` script first manually. >>> >>> As far as speed, xsmith is not particularly fast. There is a server >>> mode that at least saves on startup time, but I think the harness >>> might not run it particularly well (Guy correct me if I'm wrong -- at >>> any rate I haven't been using it with server mode, but that might be >>> because I never bothered changing my config after bugs were fixed). >>> Aside from Racket's slow startup speed (particularly bad with Xsmith >>> because it loads a lot of stuff), the runtime varies a lot based on >>> the max-depth configuration, the size of the grammar, etc. >>> >>> I haven't focused a lot on performance, necessarily. Basically >>> whenever I decide Xsmith is too slow for my taste I spend some time >>> trying to optimize it, and generally make a few gains until I think >>> ?that's enough optimization for now?. If you can find any ways to >>> make improvements, that would be great! I might have taken all of the >>> low-hanging fruit in my previous optimization passes, however. In >>> previous optimization passes I got the most out of optimizing in the >>> type system, but I made some major improvements to type checking speed >>> last time so it may not dominate anymore. >>> >>> >>> As an aside, a question for you: Are there multiple Dafny >>> implementations? Or different optimization levels? Or are you >>> running different versions of the compiler against each other? >>> >>> >>> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase wrote: >>> >Also, I'm curious about the setup of various fuzzers that you are >>> running. >>> >How many programs are you able to generate and test per minute? Do you >>> have >>> >any particular parallelization setup that works well for you? >>> > >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >>> >sorawee.pwase at gmail.com> wrote: >>> > >>> >> Do you have a roadmap for implementing "Provide mechanism for >>> precluding >>> >> types in `fresh-type-variable` invocations"? ( >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also really >>> >> want this feature. >>> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >>> wrote: >>> >> >>> >>> That would be great, thanks! >>> >>> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase >>> wrote: >>> >>> >Yes, this is very helpful. I will let you know if I encounter any >>> further >>> >>> >issues after updating XSmith. >>> >>> > >>> >>> >By the way, do you want any help with Scribble? I notice a lot of >>> typos >>> >>> and >>> >>> >link errors and can submit PRs to fix them. >>> >>> > >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >>> >>> wrote: >>> >>> > >>> >>> >> Sorry for the delay. I wanted to do the debugging before >>> responding, >>> >>> >> but then I didn't get to it for a while. >>> >>> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >>> >>> >> >>> >>> >> First, that issue with the bud-node was a bug in the `edit` >>> property >>> >>> >> which the canned-components `ProcedureApplication` uses. It's >>> fixed >>> >>> >> now. >>> >>> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >>> >>> >> >> function application node that only allows a reference as its >>> RHS >>> >>> >> >> rather than allowing arbitrary expressions. But that said, I'm >>> not >>> >>> >> >> sure I fully understand what you're trying to do. >>> >>> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >>> >>> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be >>> function >>> >>> >> application that only allows a reference in the procedure >>> position. I >>> >>> >> don't know why I wrote RHS. >>> >>> >> >>> >>> >> Anyway, the strategy of having separate definition nodes for >>> functions >>> >>> >> and everything else is used in Cish. I don't recommend following >>> >>> >> anything Cish does. However, it accomplishes this by having the >>> >>> >> program node have non-function definitions first, and disallowing >>> use >>> >>> >> of functions on the RHS of those functions. >>> >>> >> >>> >>> >> The problem where it can't find a reference is a bug (that I >>> haven't >>> >>> >> fixed yet) caused by the lifting mechanics not really being >>> completely >>> >>> >> in sync with the reference mechanics with respect to nodes that >>> have >>> >>> >> multiple definition fields. When there are multiple definition >>> >>> >> fields, definitions in the lower field can reference the >>> definitions >>> >>> >> in the higher field, but not vice versa. The lifting machinery >>> >>> >> doesn't know about that ordering, it just stuffs the lifted >>> definition >>> >>> >> into the front of the definitions in whatever field it is >>> targetting. >>> >>> >> Thus it places a definition in a place that the reference machinery >>> >>> >> thinks is out of scope. >>> >>> >> >>> >>> >> So... if you want to have different binders that get different >>> fields >>> >>> >> in the program node you can only make it work if you disallow >>> stuff to >>> >>> >> be sure you never need to reference something that will be lifted >>> into >>> >>> >> the lower field. >>> >>> >> >>> >>> >> That said, I've just fixed some issues to allow definition nodes >>> to be >>> >>> >> subtypable, so now you can have a definition node that is a >>> subtype of >>> >>> >> Definition, but has some different rules. Importantly, though, it >>> can >>> >>> >> be lifted to the same field. >>> >>> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your example to >>> look >>> >>> >> like this: >>> >>> >> >>> >>> >> ``` >>> >>> >> [MyDefinition Definition () >>> >>> >> #:prop fresh (hash 'Expression (make-hole >>> >>> >> 'ProcedureApplication))] >>> >>> >> ``` >>> >>> >> >>> >>> >> It can now be lifted among normal definitions, but always have >>> >>> >> ProcedureApplication as its RHS. >>> >>> >> >>> >>> >> However, I'm still not convinced that you actually want >>> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict lambda >>> >>> >> nodes to only be generated on the RHS of a FunctionDefinition node >>> >>> >> (that's a subtype of Definition), and make FunctionDefinition >>> always >>> >>> >> have a Lambda node as its RHS. >>> >>> >> >>> >>> >> There is an example in the documentation that is like this, in the >>> >>> >> explanation of the `choice-filters-to-apply` property. But I'll >>> paste >>> >>> >> it here: >>> >>> >> >>> >>> >> ``` >>> >>> >> (add-choice-method >>> >>> >> my-component >>> >>> >> no-lambda-except-global-def >>> >>> >> [#f (? () #t)] >>> >>> >> [Lambda (? () >>> >>> >> (and (parent-node current-hole) >>> >>> >> (equal? (ast-node-type (parent-node current-hole)) >>> >>> >> 'Definition) >>> >>> >> (parent-node (parent-node current-hole)) >>> >>> >> (equal? (ast-node-type (parent-node (parent-node >>> >>> >> current-hole))) >>> >>> >> 'Program)))]) >>> >>> >> (add-property my-component >>> >>> >> choice-filters-to-apply >>> >>> >> [#f (no-lambda-except-global-def)]) >>> >>> >> ``` >>> >>> >> >>> >>> >> You can use something like that to allow Lambda nodes to only be >>> >>> >> generated on the RHS of FunctionDefinition nodes, and define >>> >>> >> FunctionDefinition like MyDefinition above to always generate >>> Lambda >>> >>> >> as its RHS, and I think it should work for a first-order language. >>> >>> >> >>> >>> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type variables >>> >>> >> >with some concrete non-function type variables, but down the >>> road, I >>> >>> get >>> >>> >> an >>> >>> >> >internal error about unification failure.). Perhaps I'm not >>> supposed >>> >>> to >>> >>> >> >unify! inside add-choice-method? >>> >>> >> >>> >>> >> Well, probably not. Generally the only place you want to `unify!` >>> is >>> >>> >> inside the type property in the function that determines child >>> types >>> >>> >> based on the parent type. Inside that function you can `unify!`, >>> but >>> >>> >> the unification needs to be consistent (IE not unified based on the >>> >>> >> result of `random`) OR the choice needs to be saved in a way that >>> it >>> >>> >> can be looked up when that function is run again. >>> >>> >> >>> >>> >> The problem is that the choice method gets its type information >>> based >>> >>> >> on a hole node that is later replaced, so the type is recomputed >>> >>> >> without necessarily consulting the type that the hole said it was. >>> >>> >> You should be able to constrain the type using unification if you >>> save >>> >>> >> that type to re-unify it consistently later. But without seeing >>> the >>> >>> >> exact error and code that generated it, I'm not 100% certain. >>> >>> >> >>> >>> >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type could be >>> >>> used >>> >>> >> > nondeterministically in this manner, and my experience with it >>> so far >>> >>> >> seems >>> >>> >> > to indicate no. >>> >>> >> >>> >>> >> I think this should work, barring the issues mentioned above about >>> the >>> >>> >> lifting vs referencing bug when you have the different definition >>> >>> >> types in different fields. Adjusting your example to have >>> >>> >> MyDefinition as a subtype, and then only having a Definition field >>> in >>> >>> >> MyProgram, random choice seems to work fine. But sometimes it >>> >>> >> generates gargantuan programs (that I kill before it finishes) >>> because >>> >>> >> it keeps choosing MyDefinition, requiring a function application, >>> >>> >> which likely lifts a reference to a function, which might choose >>> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and >>> >>> >> bigger types. >>> >>> >> >>> >>> >> >>> >>> >> Anyway, does that all help? >>> >>> >> >>> >>> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee Porncharoenwase >>> >>> wrote: >>> >>> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as a >>> >>> >> >> function application node that only allows a reference as its >>> RHS >>> >>> >> >> rather than allowing arbitrary expressions. But that said, I'm >>> not >>> >>> >> >> sure I fully understand what you're trying to do. >>> >>> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >>> >>> >> > >>> >>> >> > 1. I have no problem with the definition site. That?s easy to >>> deal >>> >>> >> with. >>> >>> >> > 2. I also have no problem with restricting function >>> application. >>> >>> That?s >>> >>> >> > easy to deal with as well. >>> >>> >> > 3. The problem is primarily with a bare variable reference >>> bound >>> >>> to a >>> >>> >> > closure value, which must only occur in function application. >>> For >>> >>> >> example, >>> >>> >> > the following code is invalid. >>> >>> >> > >>> >>> >> >(define (f) (void)) >>> >>> >> > >>> >>> >> >(define g f) ;; this is invalid >>> >>> >> >(define h (g)) >>> >>> >> >(define a (list f)) ;; this is also invalid >>> >>> >> >(define b (f)) ;; this is ok >>> >>> >> > >>> >>> >> >I have been using choice-filters-to-apply to restrict variable >>> >>> references >>> >>> >> >whose type can-unify? with function-type that doesn't directly >>> appear >>> >>> >> under >>> >>> >> >function application. But this is obviously not ideal because the >>> >>> filtered >>> >>> >> >nodes are mostly those whose type is totally unrestricted. So >>> >>> filtering >>> >>> >> >them would prevent them from being concretized to other >>> non-function >>> >>> types >>> >>> >> >as well. (Also note that I tried unify! these unrestricted type >>> >>> variables >>> >>> >> >with some concrete non-function type variables, but down the >>> road, I >>> >>> get >>> >>> >> an >>> >>> >> >internal error about unification failure.). Perhaps I'm not >>> supposed >>> >>> to >>> >>> >> >unify! inside add-choice-method? >>> >>> >> > >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the >>> situation >>> >>> >> >> where a reference is generated that causes a new definition to >>> be >>> >>> >> >> added such that the variable is in scope at the use site. IE >>> the >>> >>> >> >> binding is lifted from the use site to some outer node that >>> contains >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to definitions >>> that >>> >>> >> >> are created automatically in this way (which is ultimately most >>> >>> >> >> definitions). The exception is caused because somehow the lift >>> >>> ended >>> >>> >> >> up in a place where it wasn't actually in scope. That shouldn't >>> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer that >>> does >>> >>> >> >> this, I'll try to debug it. >>> >>> >> >> >>> >>> >> >#lang clotho >>> >>> >> > >>> >>> >> >(require >>> >>> >> > xsmith >>> >>> >> > racr >>> >>> >> > xsmith/racr-convenience >>> >>> >> > xsmith/canned-components >>> >>> >> > racket/string >>> >>> >> > racket/list >>> >>> >> > racket/pretty) >>> >>> >> > >>> >>> >> >;; We first define a basic component and add a bunch of >>> expressions. >>> >>> >> > >>> >>> >> >(define-basic-spec-component somelisp) >>> >>> >> > >>> >>> >> >(add-basic-expressions somelisp >>> >>> >> > #:ProgramWithSequence #t >>> >>> >> > #:VoidExpression #t >>> >>> >> > #:AssignmentExpression #t >>> >>> >> > #:VariableReference #t >>> >>> >> > #:ProcedureApplication #t >>> >>> >> > #:IfExpression #t >>> >>> >> > #:ExpressionSequence #t >>> >>> >> > #:LetSequential #t >>> >>> >> > #:LambdaWithExpression #t >>> >>> >> > #:Numbers #t >>> >>> >> > #:Booleans #t >>> >>> >> > #:Strings #t >>> >>> >> > #:ImmutableList #t) >>> >>> >> > >>> >>> >> >(add-loop-over-container >>> >>> >> > somelisp >>> >>> >> > #:name Map >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >>> >>> inner))) >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >>> inner)))) >>> >>> >> > >>> >>> >> >(add-to-grammar >>> >>> >> > somelisp >>> >>> >> > [MyProgram #f >>> >>> >> > ([decls : Definition *] >>> >>> >> > [declsTwo : MyDefinition *] >>> >>> >> > [body : Expression]) >>> >>> >> > #:prop type-info >>> >>> >> > [(fresh-type-variable) >>> >>> >> > (? (n t) >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) >>> >>> >> > 'body (fresh-type-variable)))]] >>> >>> >> > [MyDefinition #f ([type] >>> >>> >> > [name = (fresh-var-name "b_")] >>> >>> >> > [e : ProcedureApplication]) >>> >>> >> > #:prop binder-info () >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >>> (hash >>> >>> 'e >>> >>> >> t))]]) >>> >>> >> > >>> >>> >> >(add-property >>> >>> >> > somelisp >>> >>> >> > lift-type->ast-binder-type >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >>> 'Definition))]) >>> >>> >> > >>> >>> >> >;; Sometimes we have a type variable that is not concrete but >>> needs >>> >>> to be. >>> >>> >> >;; Here we provide a list of options we can pick for an >>> unconstrained >>> >>> >> >;; type variable. >>> >>> >> >(define (type-thunks-for-concretization) >>> >>> >> > (list >>> >>> >> > (?()int-type) >>> >>> >> > (?()bool-type) >>> >>> >> > (?()string-type) >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >>> >>> >> > >>> >>> >> >(define (somelisp-format-render s-exps) >>> >>> >> > (define out (open-output-string)) >>> >>> >> > (for ([symex s-exps]) >>> >>> >> > (pretty-print symex out 1)) >>> >>> >> > (get-output-string out)) >>> >>> >> > >>> >>> >> >(define-xsmith-interface-functions >>> >>> >> > [somelisp] >>> >>> >> > #:program-node MyProgram >>> >>> >> > #:type-thunks type-thunks-for-concretization >>> >>> >> > #:comment-wrap (? (lines) >>> >>> >> > (string-join >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >>> >>> >> > "\n")) >>> >>> >> > #:format-render somelisp-format-render) >>> >>> >> > >>> >>> >> >(module+ main (somelisp-command-line)) >>> >>> >> > >>> >>> >> > >>> >>> >> >> Probably something is going wrong in the `fresh` property. >>> Have you >>> >>> >> >> changed that? >>> >>> >> >> >>> >>> >> >I don?t think so. See the code at the end of the email. >>> >>> >> > >>> >>> >> >> I should change that error message. Anyway, it seems to think >>> there >>> >>> >> >> is no outer node that has a `Definition *` child that can be the >>> >>> >> >> target for lifting a definition. Generally this happens when >>> you >>> >>> >> >> forget to have an outer Program node or something that can hold >>> >>> >> >> definitions. >>> >>> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. >>> >>> >> > >>> >>> >> >#lang clotho >>> >>> >> > >>> >>> >> >(require >>> >>> >> > xsmith >>> >>> >> > racr >>> >>> >> > xsmith/racr-convenience >>> >>> >> > xsmith/canned-components >>> >>> >> > racket/string >>> >>> >> > racket/list >>> >>> >> > racket/pretty) >>> >>> >> > >>> >>> >> >;; We first define a basic component and add a bunch of >>> expressions. >>> >>> >> > >>> >>> >> >(define-basic-spec-component somelisp) >>> >>> >> > >>> >>> >> >(add-basic-expressions somelisp >>> >>> >> > #:ProgramWithSequence #t >>> >>> >> > #:VoidExpression #t >>> >>> >> > #:AssignmentExpression #t >>> >>> >> > #:VariableReference #t >>> >>> >> > #:ProcedureApplication #t >>> >>> >> > #:IfExpression #t >>> >>> >> > #:ExpressionSequence #t >>> >>> >> > #:LetSequential #t >>> >>> >> > #:LambdaWithExpression #t >>> >>> >> > #:Numbers #t >>> >>> >> > #:Booleans #t >>> >>> >> > #:Strings #t >>> >>> >> > #:ImmutableList #t) >>> >>> >> > >>> >>> >> >(add-loop-over-container >>> >>> >> > somelisp >>> >>> >> > #:name Map >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >>> >>> inner))) >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >>> inner)))) >>> >>> >> > >>> >>> >> >(add-to-grammar >>> >>> >> > somelisp >>> >>> >> > [MyProgram #f >>> >>> >> > ([decls : Definition *] >>> >>> >> > [body : SubProgram]) >>> >>> >> > #:prop type-info >>> >>> >> > [(fresh-type-variable) >>> >>> >> > (? (n t) >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >>> >> > 'body (fresh-type-variable)))]] >>> >>> >> > [SubProgram #f >>> >>> >> > ([decls : MyDefinition *] >>> >>> >> > [body : Expression]) >>> >>> >> > #:prop type-info >>> >>> >> > [(fresh-type-variable) >>> >>> >> > (? (n t) >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >>> >>> >> > 'body (fresh-type-variable)))]] >>> >>> >> > [MyDefinition #f ([type] >>> >>> >> > [name = (fresh-var-name "b_")] >>> >>> >> > [e : ProcedureApplication]) >>> >>> >> > #:prop binder-info () >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >>> (hash >>> >>> 'e >>> >>> >> t))]]) >>> >>> >> > >>> >>> >> >(add-property >>> >>> >> > somelisp >>> >>> >> > lift-type->ast-binder-type >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >>> 'Definition))]) >>> >>> >> > >>> >>> >> >;; Sometimes we have a type variable that is not concrete but >>> needs >>> >>> to be. >>> >>> >> >;; Here we provide a list of options we can pick for an >>> unconstrained >>> >>> >> >;; type variable. >>> >>> >> >(define (type-thunks-for-concretization) >>> >>> >> > (list >>> >>> >> > (?()int-type) >>> >>> >> > (?()bool-type) >>> >>> >> > (?()string-type) >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >>> >>> >> > >>> >>> >> >(define (somelisp-format-render s-exps) >>> >>> >> > (define out (open-output-string)) >>> >>> >> > (for ([symex s-exps]) >>> >>> >> > (pretty-print symex out 1)) >>> >>> >> > (get-output-string out)) >>> >>> >> > >>> >>> >> >(define-xsmith-interface-functions >>> >>> >> > [somelisp] >>> >>> >> > #:program-node MyProgram >>> >>> >> > #:type-thunks type-thunks-for-concretization >>> >>> >> > #:comment-wrap (? (lines) >>> >>> >> > (string-join >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >>> >>> >> > "\n")) >>> >>> >> > #:format-render somelisp-format-render) >>> >>> >> > >>> >>> >> >(module+ main (somelisp-command-line)) >>> >>> >> >>> >>> >>> >> >>> >> From eeide at cs.utah.edu Tue Jul 20 12:01:17 2021 From: eeide at cs.utah.edu (Eric Eide) Date: Tue, 20 Jul 2021 12:01:17 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: (Sorawee Porncharoenwase's message of "Tue, 20 Jul 2021 09:58:55 -0700") References: Message-ID: Sorawee Porncharoenwase writes: > By the way, here are the results so far: > - https://github.com/dafny-lang/dafny/issues/1291 > - https://github.com/dafny-lang/dafny/issues/1297 Thank you! We are eager to keep track of these. -- ------------------------------------------------------------------------------- Eric Eide . University of Utah School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA From sorawee.pwase at gmail.com Wed Jul 21 12:22:07 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Wed, 21 Jul 2021 11:22:07 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Sure, I can do that. I have another question about type->string. I have been using type->string from standard-ml / cish examples. (define (type->string t) (let loop ([t* t]) (define t (concretize-type t*)) (unify! t t*) (cond [(can-unify? t (immutable (array-type (fresh-type-variable)))) (define inner (fresh-type-variable)) (unify! t (immutable (array-type inner))) (format "seq<~a>" (loop inner))] [(can-unify? t (product-type #f)) (match (product-type-inner-type-list t) [(list) "()"] [(list a) (loop a)] [inners (format "(~a)" (string-join (map loop inners) ", "))])] [(can-unify? t (function-type (fresh-type-variable) (fresh-type-variable))) (define ret (fresh-type-variable)) (define arg (fresh-type-variable)) (unify! t (function-type arg ret)) (format "(~a -> ~a)" (loop arg) (loop ret))] [(can-unify? t number-type) "int"] [(can-unify? t int-type) "int"] [(can-unify? t bool-type) "bool"] [(can-unify? t string-type) (if (zero? (random 2)) "string" "seq")] [else #;(error 'type->string "Type not implemented yet: ~v" t) "FOOBAR"]))) The first weird thing is that the number-type branch is necessary, despite the fact that I don?t have any float in the program. The second weird thing is that sometimes "FOOBAR" is reached because t is #. In particular, it generates the following code: safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) # roughly (vector-set (let ([theArray (vector) : (array (array FOOBAR))]) theArray) 1 (vector 1)) where you can see that ?FOOBAR? here should really be int. In my experiment, replacing default-base-type with int seems to work well, but it indicates something is obviously wrong. Here?s the call site of type->string FWIW: [ImmutableArrayLiteral (? (n) (match (ast-children (ast-child 'expressions n)) [(list) ;; sorry for convoluted logic, but I used to use inner-type for something and haven't refactored this. (define inner-type (fresh-type-variable)) (define the-array-type (immutable (array-type inner-type))) (define the-type (concretize-type ($xsmith_type n))) (unify! the-type the-array-type) (text (format "(var theArray : ~a := []; theArray)" (type->string the-type)))] [xs (h-append lbracket (comma-list (pretty-print-children xs)) rbracket)]))] Any ideas? On Tue, Jul 20, 2021 at 10:45 AM William G Hatch wrote: > Awesome! We have a git repo where we are keeping track of bugs we > find with Xsmith. If you want, we could add you to it and you can add > any bugs you find to it. Otherwise I will probably put in anything > you mention like this. > > On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase wrote: > >By the way, here are the results so far: > > > >- https://github.com/dafny-lang/dafny/issues/1291 > >- https://github.com/dafny-lang/dafny/issues/1297 > > > >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < > >sorawee.pwase at gmail.com> wrote: > > > >> We are primarily focused on different compiler backends for Dafny (Dafny > >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). > >> > >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch > wrote: > >> > >>> >> Do you have a roadmap for implementing ... > >>> > >>> Not at the moment. Right now I'm focused on getting some of our > >>> mostly-finished fuzzers *really* finished and ready for fuzzing, > >>> running some fuzzing campaigns to find some bugs, and some features > >>> that are immediately helpful for running fuzzing (eg. some stuff to > >>> support feedback-directed fuzzing and choice weighting). > >>> > >>> > >>> >Also, I'm curious about the setup of various fuzzers that you are > >>> running. > >>> >How many programs are you able to generate and test per minute? Do you > >>> have > >>> >any particular parallelization setup that works well for you? > >>> > >>> There is a harness repository that we use that was mostly written by > >>> Guy Watson: > >>> > >>> https://gitlab.flux.utah.edu/xsmith/harness > >>> > >>> There is a `harness.py` script that takes a configuration file to run > >>> an xsmith fuzzer, running its output through various implementations > >>> of whatever language, comparing the output, and binning the result > >>> (eg. everything matching, diff, crash, everything crashing (IE bad > >>> program), xsmith itself crashing, ...). That harness script is > >>> basically for running a single job, and there are 2 ways of doing it > >>> in parallel. > >>> > >>> The first one is the `go-button`, which can run fuzzing on multiple > >>> machines at once. It is fairly dependent on Emulab/Cloudlab > >>> (https://www.emulab.net/) to have things set up just right, and > >>> requires at least 2 machines. It also generates a bunch of live > >>> graphs about fuzzing. It uses ansible to run setup scripts to prep > >>> each machine before starting the fuzz campaign. > >>> > >>> I mostly use `local-multi-harness` instead, which just runs on a > >>> single machine, but runs multiple harness processes. It doesn't > >>> automatically run the setup scripts like the `go-button` does, so > >>> eg. if you want to use the `racket-kernel-fuzzer` configuration you > >>> need to run the `racket-install.rkt` script first manually. > >>> > >>> As far as speed, xsmith is not particularly fast. There is a server > >>> mode that at least saves on startup time, but I think the harness > >>> might not run it particularly well (Guy correct me if I'm wrong -- at > >>> any rate I haven't been using it with server mode, but that might be > >>> because I never bothered changing my config after bugs were fixed). > >>> Aside from Racket's slow startup speed (particularly bad with Xsmith > >>> because it loads a lot of stuff), the runtime varies a lot based on > >>> the max-depth configuration, the size of the grammar, etc. > >>> > >>> I haven't focused a lot on performance, necessarily. Basically > >>> whenever I decide Xsmith is too slow for my taste I spend some time > >>> trying to optimize it, and generally make a few gains until I think > >>> ?that's enough optimization for now?. If you can find any ways to > >>> make improvements, that would be great! I might have taken all of the > >>> low-hanging fruit in my previous optimization passes, however. In > >>> previous optimization passes I got the most out of optimizing in the > >>> type system, but I made some major improvements to type checking speed > >>> last time so it may not dominate anymore. > >>> > >>> > >>> As an aside, a question for you: Are there multiple Dafny > >>> implementations? Or different optimization levels? Or are you > >>> running different versions of the compiler against each other? > >>> > >>> > >>> > >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase > wrote: > >>> >Also, I'm curious about the setup of various fuzzers that you are > >>> running. > >>> >How many programs are you able to generate and test per minute? Do you > >>> have > >>> >any particular parallelization setup that works well for you? > >>> > > >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < > >>> >sorawee.pwase at gmail.com> wrote: > >>> > > >>> >> Do you have a roadmap for implementing "Provide mechanism for > >>> precluding > >>> >> types in `fresh-type-variable` invocations"? ( > >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also > really > >>> >> want this feature. > >>> >> > >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch > >>> wrote: > >>> >> > >>> >>> That would be great, thanks! > >>> >>> > >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase > >>> wrote: > >>> >>> >Yes, this is very helpful. I will let you know if I encounter any > >>> further > >>> >>> >issues after updating XSmith. > >>> >>> > > >>> >>> >By the way, do you want any help with Scribble? I notice a lot of > >>> typos > >>> >>> and > >>> >>> >link errors and can submit PRs to fix them. > >>> >>> > > >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > > >>> >>> wrote: > >>> >>> > > >>> >>> >> Sorry for the delay. I wanted to do the debugging before > >>> responding, > >>> >>> >> but then I didn't get to it for a while. > >>> >>> >> > >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. > >>> >>> >> > >>> >>> >> First, that issue with the bud-node was a bug in the `edit` > >>> property > >>> >>> >> which the canned-components `ProcedureApplication` uses. It's > >>> fixed > >>> >>> >> now. > >>> >>> >> > >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as > a > >>> >>> >> >> function application node that only allows a reference as its > >>> RHS > >>> >>> >> >> rather than allowing arbitrary expressions. But that said, > I'm > >>> not > >>> >>> >> >> sure I fully understand what you're trying to do. > >>> >>> >> >> > >>> >>> >> >I?m not sure what you mean by that. But: > >>> >>> >> > >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be > >>> function > >>> >>> >> application that only allows a reference in the procedure > >>> position. I > >>> >>> >> don't know why I wrote RHS. > >>> >>> >> > >>> >>> >> Anyway, the strategy of having separate definition nodes for > >>> functions > >>> >>> >> and everything else is used in Cish. I don't recommend > following > >>> >>> >> anything Cish does. However, it accomplishes this by having the > >>> >>> >> program node have non-function definitions first, and > disallowing > >>> use > >>> >>> >> of functions on the RHS of those functions. > >>> >>> >> > >>> >>> >> The problem where it can't find a reference is a bug (that I > >>> haven't > >>> >>> >> fixed yet) caused by the lifting mechanics not really being > >>> completely > >>> >>> >> in sync with the reference mechanics with respect to nodes that > >>> have > >>> >>> >> multiple definition fields. When there are multiple definition > >>> >>> >> fields, definitions in the lower field can reference the > >>> definitions > >>> >>> >> in the higher field, but not vice versa. The lifting machinery > >>> >>> >> doesn't know about that ordering, it just stuffs the lifted > >>> definition > >>> >>> >> into the front of the definitions in whatever field it is > >>> targetting. > >>> >>> >> Thus it places a definition in a place that the reference > machinery > >>> >>> >> thinks is out of scope. > >>> >>> >> > >>> >>> >> So... if you want to have different binders that get different > >>> fields > >>> >>> >> in the program node you can only make it work if you disallow > >>> stuff to > >>> >>> >> be sure you never need to reference something that will be > lifted > >>> into > >>> >>> >> the lower field. > >>> >>> >> > >>> >>> >> That said, I've just fixed some issues to allow definition nodes > >>> to be > >>> >>> >> subtypable, so now you can have a definition node that is a > >>> subtype of > >>> >>> >> Definition, but has some different rules. Importantly, though, > it > >>> can > >>> >>> >> be lifted to the same field. > >>> >>> >> > >>> >>> >> Eg. if I change the MyDefinition declaration in your example to > >>> look > >>> >>> >> like this: > >>> >>> >> > >>> >>> >> ``` > >>> >>> >> [MyDefinition Definition () > >>> >>> >> #:prop fresh (hash 'Expression (make-hole > >>> >>> >> 'ProcedureApplication))] > >>> >>> >> ``` > >>> >>> >> > >>> >>> >> It can now be lifted among normal definitions, but always have > >>> >>> >> ProcedureApplication as its RHS. > >>> >>> >> > >>> >>> >> However, I'm still not convinced that you actually want > >>> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict > lambda > >>> >>> >> nodes to only be generated on the RHS of a FunctionDefinition > node > >>> >>> >> (that's a subtype of Definition), and make FunctionDefinition > >>> always > >>> >>> >> have a Lambda node as its RHS. > >>> >>> >> > >>> >>> >> There is an example in the documentation that is like this, in > the > >>> >>> >> explanation of the `choice-filters-to-apply` property. But I'll > >>> paste > >>> >>> >> it here: > >>> >>> >> > >>> >>> >> ``` > >>> >>> >> (add-choice-method > >>> >>> >> my-component > >>> >>> >> no-lambda-except-global-def > >>> >>> >> [#f (? () #t)] > >>> >>> >> [Lambda (? () > >>> >>> >> (and (parent-node current-hole) > >>> >>> >> (equal? (ast-node-type (parent-node > current-hole)) > >>> >>> >> 'Definition) > >>> >>> >> (parent-node (parent-node current-hole)) > >>> >>> >> (equal? (ast-node-type (parent-node > (parent-node > >>> >>> >> current-hole))) > >>> >>> >> 'Program)))]) > >>> >>> >> (add-property my-component > >>> >>> >> choice-filters-to-apply > >>> >>> >> [#f (no-lambda-except-global-def)]) > >>> >>> >> ``` > >>> >>> >> > >>> >>> >> You can use something like that to allow Lambda nodes to only be > >>> >>> >> generated on the RHS of FunctionDefinition nodes, and define > >>> >>> >> FunctionDefinition like MyDefinition above to always generate > >>> Lambda > >>> >>> >> as its RHS, and I think it should work for a first-order > language. > >>> >>> >> > >>> >>> >> > >>> >>> >> >(Also note that I tried unify! these unrestricted type > variables > >>> >>> >> >with some concrete non-function type variables, but down the > >>> road, I > >>> >>> get > >>> >>> >> an > >>> >>> >> >internal error about unification failure.). Perhaps I'm not > >>> supposed > >>> >>> to > >>> >>> >> >unify! inside add-choice-method? > >>> >>> >> > >>> >>> >> Well, probably not. Generally the only place you want to > `unify!` > >>> is > >>> >>> >> inside the type property in the function that determines child > >>> types > >>> >>> >> based on the parent type. Inside that function you can > `unify!`, > >>> but > >>> >>> >> the unification needs to be consistent (IE not unified based on > the > >>> >>> >> result of `random`) OR the choice needs to be saved in a way > that > >>> it > >>> >>> >> can be looked up when that function is run again. > >>> >>> >> > >>> >>> >> The problem is that the choice method gets its type information > >>> based > >>> >>> >> on a hole node that is later replaced, so the type is recomputed > >>> >>> >> without necessarily consulting the type that the hole said it > was. > >>> >>> >> You should be able to constrain the type using unification if > you > >>> save > >>> >>> >> that type to re-unify it consistently later. But without seeing > >>> the > >>> >>> >> exact error and code that generated it, I'm not 100% certain. > >>> >>> >> > >>> >>> >> > >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type > could be > >>> >>> used > >>> >>> >> > nondeterministically in this manner, and my experience with it > >>> so far > >>> >>> >> seems > >>> >>> >> > to indicate no. > >>> >>> >> > >>> >>> >> I think this should work, barring the issues mentioned above > about > >>> the > >>> >>> >> lifting vs referencing bug when you have the different > definition > >>> >>> >> types in different fields. Adjusting your example to have > >>> >>> >> MyDefinition as a subtype, and then only having a Definition > field > >>> in > >>> >>> >> MyProgram, random choice seems to work fine. But sometimes it > >>> >>> >> generates gargantuan programs (that I kill before it finishes) > >>> because > >>> >>> >> it keeps choosing MyDefinition, requiring a function > application, > >>> >>> >> which likely lifts a reference to a function, which might choose > >>> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and > >>> >>> >> bigger types. > >>> >>> >> > >>> >>> >> > >>> >>> >> Anyway, does that all help? > >>> >>> >> > >>> >>> >> > >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee > Porncharoenwase > >>> >>> wrote: > >>> >>> >> >> > >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as > a > >>> >>> >> >> function application node that only allows a reference as its > >>> RHS > >>> >>> >> >> rather than allowing arbitrary expressions. But that said, > I'm > >>> not > >>> >>> >> >> sure I fully understand what you're trying to do. > >>> >>> >> >> > >>> >>> >> >I?m not sure what you mean by that. But: > >>> >>> >> > > >>> >>> >> > 1. I have no problem with the definition site. That?s easy > to > >>> deal > >>> >>> >> with. > >>> >>> >> > 2. I also have no problem with restricting function > >>> application. > >>> >>> That?s > >>> >>> >> > easy to deal with as well. > >>> >>> >> > 3. The problem is primarily with a bare variable reference > >>> bound > >>> >>> to a > >>> >>> >> > closure value, which must only occur in function > application. > >>> For > >>> >>> >> example, > >>> >>> >> > the following code is invalid. > >>> >>> >> > > >>> >>> >> >(define (f) (void)) > >>> >>> >> > > >>> >>> >> >(define g f) ;; this is invalid > >>> >>> >> >(define h (g)) > >>> >>> >> >(define a (list f)) ;; this is also invalid > >>> >>> >> >(define b (f)) ;; this is ok > >>> >>> >> > > >>> >>> >> >I have been using choice-filters-to-apply to restrict variable > >>> >>> references > >>> >>> >> >whose type can-unify? with function-type that doesn't directly > >>> appear > >>> >>> >> under > >>> >>> >> >function application. But this is obviously not ideal because > the > >>> >>> filtered > >>> >>> >> >nodes are mostly those whose type is totally unrestricted. So > >>> >>> filtering > >>> >>> >> >them would prevent them from being concretized to other > >>> non-function > >>> >>> types > >>> >>> >> >as well. (Also note that I tried unify! these unrestricted type > >>> >>> variables > >>> >>> >> >with some concrete non-function type variables, but down the > >>> road, I > >>> >>> get > >>> >>> >> an > >>> >>> >> >internal error about unification failure.). Perhaps I'm not > >>> supposed > >>> >>> to > >>> >>> >> >unify! inside add-choice-method? > >>> >>> >> > > >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the > >>> situation > >>> >>> >> >> where a reference is generated that causes a new definition > to > >>> be > >>> >>> >> >> added such that the variable is in scope at the use site. IE > >>> the > >>> >>> >> >> binding is lifted from the use site to some outer node that > >>> contains > >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to > definitions > >>> that > >>> >>> >> >> are created automatically in this way (which is ultimately > most > >>> >>> >> >> definitions). The exception is caused because somehow the > lift > >>> >>> ended > >>> >>> >> >> up in a place where it wasn't actually in scope. That > shouldn't > >>> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer > that > >>> does > >>> >>> >> >> this, I'll try to debug it. > >>> >>> >> >> > >>> >>> >> >#lang clotho > >>> >>> >> > > >>> >>> >> >(require > >>> >>> >> > xsmith > >>> >>> >> > racr > >>> >>> >> > xsmith/racr-convenience > >>> >>> >> > xsmith/canned-components > >>> >>> >> > racket/string > >>> >>> >> > racket/list > >>> >>> >> > racket/pretty) > >>> >>> >> > > >>> >>> >> >;; We first define a basic component and add a bunch of > >>> expressions. > >>> >>> >> > > >>> >>> >> >(define-basic-spec-component somelisp) > >>> >>> >> > > >>> >>> >> >(add-basic-expressions somelisp > >>> >>> >> > #:ProgramWithSequence #t > >>> >>> >> > #:VoidExpression #t > >>> >>> >> > #:AssignmentExpression #t > >>> >>> >> > #:VariableReference #t > >>> >>> >> > #:ProcedureApplication #t > >>> >>> >> > #:IfExpression #t > >>> >>> >> > #:ExpressionSequence #t > >>> >>> >> > #:LetSequential #t > >>> >>> >> > #:LambdaWithExpression #t > >>> >>> >> > #:Numbers #t > >>> >>> >> > #:Booleans #t > >>> >>> >> > #:Strings #t > >>> >>> >> > #:ImmutableList #t) > >>> >>> >> > > >>> >>> >> >(add-loop-over-container > >>> >>> >> > somelisp > >>> >>> >> > #:name Map > >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type > >>> >>> inner))) > >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >>> inner)))) > >>> >>> >> > > >>> >>> >> >(add-to-grammar > >>> >>> >> > somelisp > >>> >>> >> > [MyProgram #f > >>> >>> >> > ([decls : Definition *] > >>> >>> >> > [declsTwo : MyDefinition *] > >>> >>> >> > [body : Expression]) > >>> >>> >> > #:prop type-info > >>> >>> >> > [(fresh-type-variable) > >>> >>> >> > (? (n t) > >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) > >>> >>> >> > 'body (fresh-type-variable)))]] > >>> >>> >> > [MyDefinition #f ([type] > >>> >>> >> > [name = (fresh-var-name "b_")] > >>> >>> >> > [e : ProcedureApplication]) > >>> >>> >> > #:prop binder-info () > >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) > >>> (hash > >>> >>> 'e > >>> >>> >> t))]]) > >>> >>> >> > > >>> >>> >> >(add-property > >>> >>> >> > somelisp > >>> >>> >> > lift-type->ast-binder-type > >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >>> 'Definition))]) > >>> >>> >> > > >>> >>> >> >;; Sometimes we have a type variable that is not concrete but > >>> needs > >>> >>> to be. > >>> >>> >> >;; Here we provide a list of options we can pick for an > >>> unconstrained > >>> >>> >> >;; type variable. > >>> >>> >> >(define (type-thunks-for-concretization) > >>> >>> >> > (list > >>> >>> >> > (?()int-type) > >>> >>> >> > (?()bool-type) > >>> >>> >> > (?()string-type) > >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >>> >>> >> > > >>> >>> >> >(define (somelisp-format-render s-exps) > >>> >>> >> > (define out (open-output-string)) > >>> >>> >> > (for ([symex s-exps]) > >>> >>> >> > (pretty-print symex out 1)) > >>> >>> >> > (get-output-string out)) > >>> >>> >> > > >>> >>> >> >(define-xsmith-interface-functions > >>> >>> >> > [somelisp] > >>> >>> >> > #:program-node MyProgram > >>> >>> >> > #:type-thunks type-thunks-for-concretization > >>> >>> >> > #:comment-wrap (? (lines) > >>> >>> >> > (string-join > >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >>> >>> >> > "\n")) > >>> >>> >> > #:format-render somelisp-format-render) > >>> >>> >> > > >>> >>> >> >(module+ main (somelisp-command-line)) > >>> >>> >> > > >>> >>> >> > > >>> >>> >> >> Probably something is going wrong in the `fresh` property. > >>> Have you > >>> >>> >> >> changed that? > >>> >>> >> >> > >>> >>> >> >I don?t think so. See the code at the end of the email. > >>> >>> >> > > >>> >>> >> >> I should change that error message. Anyway, it seems to > think > >>> there > >>> >>> >> >> is no outer node that has a `Definition *` child that can be > the > >>> >>> >> >> target for lifting a definition. Generally this happens when > >>> you > >>> >>> >> >> forget to have an outer Program node or something that can > hold > >>> >>> >> >> definitions. > >>> >>> >> >> > >>> >>> >> >I don?t think that?s the case, but I could be wrong. > >>> >>> >> > > >>> >>> >> >#lang clotho > >>> >>> >> > > >>> >>> >> >(require > >>> >>> >> > xsmith > >>> >>> >> > racr > >>> >>> >> > xsmith/racr-convenience > >>> >>> >> > xsmith/canned-components > >>> >>> >> > racket/string > >>> >>> >> > racket/list > >>> >>> >> > racket/pretty) > >>> >>> >> > > >>> >>> >> >;; We first define a basic component and add a bunch of > >>> expressions. > >>> >>> >> > > >>> >>> >> >(define-basic-spec-component somelisp) > >>> >>> >> > > >>> >>> >> >(add-basic-expressions somelisp > >>> >>> >> > #:ProgramWithSequence #t > >>> >>> >> > #:VoidExpression #t > >>> >>> >> > #:AssignmentExpression #t > >>> >>> >> > #:VariableReference #t > >>> >>> >> > #:ProcedureApplication #t > >>> >>> >> > #:IfExpression #t > >>> >>> >> > #:ExpressionSequence #t > >>> >>> >> > #:LetSequential #t > >>> >>> >> > #:LambdaWithExpression #t > >>> >>> >> > #:Numbers #t > >>> >>> >> > #:Booleans #t > >>> >>> >> > #:Strings #t > >>> >>> >> > #:ImmutableList #t) > >>> >>> >> > > >>> >>> >> >(add-loop-over-container > >>> >>> >> > somelisp > >>> >>> >> > #:name Map > >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type > >>> >>> inner))) > >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >>> inner)))) > >>> >>> >> > > >>> >>> >> >(add-to-grammar > >>> >>> >> > somelisp > >>> >>> >> > [MyProgram #f > >>> >>> >> > ([decls : Definition *] > >>> >>> >> > [body : SubProgram]) > >>> >>> >> > #:prop type-info > >>> >>> >> > [(fresh-type-variable) > >>> >>> >> > (? (n t) > >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >>> >> > 'body (fresh-type-variable)))]] > >>> >>> >> > [SubProgram #f > >>> >>> >> > ([decls : MyDefinition *] > >>> >>> >> > [body : Expression]) > >>> >>> >> > #:prop type-info > >>> >>> >> > [(fresh-type-variable) > >>> >>> >> > (? (n t) > >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >>> >>> >> > 'body (fresh-type-variable)))]] > >>> >>> >> > [MyDefinition #f ([type] > >>> >>> >> > [name = (fresh-var-name "b_")] > >>> >>> >> > [e : ProcedureApplication]) > >>> >>> >> > #:prop binder-info () > >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) > >>> (hash > >>> >>> 'e > >>> >>> >> t))]]) > >>> >>> >> > > >>> >>> >> >(add-property > >>> >>> >> > somelisp > >>> >>> >> > lift-type->ast-binder-type > >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >>> 'Definition))]) > >>> >>> >> > > >>> >>> >> >;; Sometimes we have a type variable that is not concrete but > >>> needs > >>> >>> to be. > >>> >>> >> >;; Here we provide a list of options we can pick for an > >>> unconstrained > >>> >>> >> >;; type variable. > >>> >>> >> >(define (type-thunks-for-concretization) > >>> >>> >> > (list > >>> >>> >> > (?()int-type) > >>> >>> >> > (?()bool-type) > >>> >>> >> > (?()string-type) > >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >>> >>> >> > > >>> >>> >> >(define (somelisp-format-render s-exps) > >>> >>> >> > (define out (open-output-string)) > >>> >>> >> > (for ([symex s-exps]) > >>> >>> >> > (pretty-print symex out 1)) > >>> >>> >> > (get-output-string out)) > >>> >>> >> > > >>> >>> >> >(define-xsmith-interface-functions > >>> >>> >> > [somelisp] > >>> >>> >> > #:program-node MyProgram > >>> >>> >> > #:type-thunks type-thunks-for-concretization > >>> >>> >> > #:comment-wrap (? (lines) > >>> >>> >> > (string-join > >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >>> >>> >> > "\n")) > >>> >>> >> > #:format-render somelisp-format-render) > >>> >>> >> > > >>> >>> >> >(module+ main (somelisp-command-line)) > >>> >>> >> > >>> >>> > >>> >> > >>> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Wed Jul 21 16:08:16 2021 From: william at hatch.uno (William G Hatch) Date: Wed, 21 Jul 2021 16:08:16 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: The default-base-type should never show up in a fuzzer that has defined the type property for any node. So this is definitely a bug. I had the default-base-type show up once before, but some other change made me stop seeing it, and I never got to the bottom of it. It's probably because of the #:type-thunks argument that you need for concretizing types of lifted definitions. The default was a list with the default base type. This was always a bad idea. I've just pushed a change that instead errors if the concretization list is used without being parameterized with something new first. At any rate, I think this is an xsmith bug, not a bug in your code. Somewhere the `current-xsmith-type-constructor-thunks` parameter is being called where it hasn't been parameterized. If the new exception that will be raised gives any useful information about where this is happening, let me know. On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase wrote: >Sure, I can do that. > >I have another question about type->string. I have been using type->string >from standard-ml / cish examples. > >(define (type->string t) > (let loop ([t* t]) > (define t (concretize-type t*)) > (unify! t t*) > (cond > [(can-unify? t (immutable (array-type (fresh-type-variable)))) > (define inner (fresh-type-variable)) > (unify! t (immutable (array-type inner))) > (format "seq<~a>" (loop inner))] > [(can-unify? t (product-type #f)) > (match (product-type-inner-type-list t) > [(list) "()"] > [(list a) (loop a)] > [inners (format "(~a)" (string-join (map loop inners) ", "))])] > [(can-unify? t (function-type (fresh-type-variable) >(fresh-type-variable))) > (define ret (fresh-type-variable)) > (define arg (fresh-type-variable)) > (unify! t (function-type arg ret)) > (format "(~a -> ~a)" (loop arg) (loop ret))] > [(can-unify? t number-type) "int"] > [(can-unify? t int-type) "int"] > [(can-unify? t bool-type) "bool"] > [(can-unify? t string-type) (if (zero? (random 2)) "string" "seq")] > [else #;(error 'type->string "Type not implemented yet: ~v" t) > "FOOBAR"]))) > >The first weird thing is that the number-type branch is necessary, despite >the fact that I don?t have any float in the program. > >The second weird thing is that sometimes "FOOBAR" is reached because t is >#. In particular, it generates the following code: > >safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) ># roughly (vector-set (let ([theArray (vector) : (array (array >FOOBAR))]) theArray) 1 (vector 1)) > >where you can see that ?FOOBAR? here should really be int. > >In my experiment, replacing default-base-type with int seems to work well, >but it indicates something is obviously wrong. Here?s the call site of >type->string FWIW: > >[ImmutableArrayLiteral > (? (n) > (match (ast-children (ast-child 'expressions n)) > [(list) > ;; sorry for convoluted logic, but I used to use inner-type for >something and haven't refactored this. > (define inner-type (fresh-type-variable)) > (define the-array-type (immutable (array-type inner-type))) > (define the-type (concretize-type ($xsmith_type n))) > (unify! the-type the-array-type) > (text (format "(var theArray : ~a := []; theArray)" >(type->string the-type)))] > [xs (h-append lbracket > (comma-list (pretty-print-children xs)) > rbracket)]))] > >Any ideas? > > >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch wrote: > >> Awesome! We have a git repo where we are keeping track of bugs we >> find with Xsmith. If you want, we could add you to it and you can add >> any bugs you find to it. Otherwise I will probably put in anything >> you mention like this. >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase wrote: >> >By the way, here are the results so far: >> > >> >- https://github.com/dafny-lang/dafny/issues/1291 >> >- https://github.com/dafny-lang/dafny/issues/1297 >> > >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < >> >sorawee.pwase at gmail.com> wrote: >> > >> >> We are primarily focused on different compiler backends for Dafny (Dafny >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). >> >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch >> wrote: >> >> >> >>> >> Do you have a roadmap for implementing ... >> >>> >> >>> Not at the moment. Right now I'm focused on getting some of our >> >>> mostly-finished fuzzers *really* finished and ready for fuzzing, >> >>> running some fuzzing campaigns to find some bugs, and some features >> >>> that are immediately helpful for running fuzzing (eg. some stuff to >> >>> support feedback-directed fuzzing and choice weighting). >> >>> >> >>> >> >>> >Also, I'm curious about the setup of various fuzzers that you are >> >>> running. >> >>> >How many programs are you able to generate and test per minute? Do you >> >>> have >> >>> >any particular parallelization setup that works well for you? >> >>> >> >>> There is a harness repository that we use that was mostly written by >> >>> Guy Watson: >> >>> >> >>> https://gitlab.flux.utah.edu/xsmith/harness >> >>> >> >>> There is a `harness.py` script that takes a configuration file to run >> >>> an xsmith fuzzer, running its output through various implementations >> >>> of whatever language, comparing the output, and binning the result >> >>> (eg. everything matching, diff, crash, everything crashing (IE bad >> >>> program), xsmith itself crashing, ...). That harness script is >> >>> basically for running a single job, and there are 2 ways of doing it >> >>> in parallel. >> >>> >> >>> The first one is the `go-button`, which can run fuzzing on multiple >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab >> >>> (https://www.emulab.net/) to have things set up just right, and >> >>> requires at least 2 machines. It also generates a bunch of live >> >>> graphs about fuzzing. It uses ansible to run setup scripts to prep >> >>> each machine before starting the fuzz campaign. >> >>> >> >>> I mostly use `local-multi-harness` instead, which just runs on a >> >>> single machine, but runs multiple harness processes. It doesn't >> >>> automatically run the setup scripts like the `go-button` does, so >> >>> eg. if you want to use the `racket-kernel-fuzzer` configuration you >> >>> need to run the `racket-install.rkt` script first manually. >> >>> >> >>> As far as speed, xsmith is not particularly fast. There is a server >> >>> mode that at least saves on startup time, but I think the harness >> >>> might not run it particularly well (Guy correct me if I'm wrong -- at >> >>> any rate I haven't been using it with server mode, but that might be >> >>> because I never bothered changing my config after bugs were fixed). >> >>> Aside from Racket's slow startup speed (particularly bad with Xsmith >> >>> because it loads a lot of stuff), the runtime varies a lot based on >> >>> the max-depth configuration, the size of the grammar, etc. >> >>> >> >>> I haven't focused a lot on performance, necessarily. Basically >> >>> whenever I decide Xsmith is too slow for my taste I spend some time >> >>> trying to optimize it, and generally make a few gains until I think >> >>> ?that's enough optimization for now?. If you can find any ways to >> >>> make improvements, that would be great! I might have taken all of the >> >>> low-hanging fruit in my previous optimization passes, however. In >> >>> previous optimization passes I got the most out of optimizing in the >> >>> type system, but I made some major improvements to type checking speed >> >>> last time so it may not dominate anymore. >> >>> >> >>> >> >>> As an aside, a question for you: Are there multiple Dafny >> >>> implementations? Or different optimization levels? Or are you >> >>> running different versions of the compiler against each other? >> >>> >> >>> >> >>> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase >> wrote: >> >>> >Also, I'm curious about the setup of various fuzzers that you are >> >>> running. >> >>> >How many programs are you able to generate and test per minute? Do you >> >>> have >> >>> >any particular parallelization setup that works well for you? >> >>> > >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >> >>> >sorawee.pwase at gmail.com> wrote: >> >>> > >> >>> >> Do you have a roadmap for implementing "Provide mechanism for >> >>> precluding >> >>> >> types in `fresh-type-variable` invocations"? ( >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also >> really >> >>> >> want this feature. >> >>> >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >> >>> wrote: >> >>> >> >> >>> >>> That would be great, thanks! >> >>> >>> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee Porncharoenwase >> >>> wrote: >> >>> >>> >Yes, this is very helpful. I will let you know if I encounter any >> >>> further >> >>> >>> >issues after updating XSmith. >> >>> >>> > >> >>> >>> >By the way, do you want any help with Scribble? I notice a lot of >> >>> typos >> >>> >>> and >> >>> >>> >link errors and can submit PRs to fix them. >> >>> >>> > >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > > >> >>> >>> wrote: >> >>> >>> > >> >>> >>> >> Sorry for the delay. I wanted to do the debugging before >> >>> responding, >> >>> >>> >> but then I didn't get to it for a while. >> >>> >>> >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some bugs. >> >>> >>> >> >> >>> >>> >> First, that issue with the bud-node was a bug in the `edit` >> >>> property >> >>> >>> >> which the canned-components `ProcedureApplication` uses. It's >> >>> fixed >> >>> >>> >> now. >> >>> >>> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as >> a >> >>> >>> >> >> function application node that only allows a reference as its >> >>> RHS >> >>> >>> >> >> rather than allowing arbitrary expressions. But that said, >> I'm >> >>> not >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >>> >>> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >>> >>> >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be >> >>> function >> >>> >>> >> application that only allows a reference in the procedure >> >>> position. I >> >>> >>> >> don't know why I wrote RHS. >> >>> >>> >> >> >>> >>> >> Anyway, the strategy of having separate definition nodes for >> >>> functions >> >>> >>> >> and everything else is used in Cish. I don't recommend >> following >> >>> >>> >> anything Cish does. However, it accomplishes this by having the >> >>> >>> >> program node have non-function definitions first, and >> disallowing >> >>> use >> >>> >>> >> of functions on the RHS of those functions. >> >>> >>> >> >> >>> >>> >> The problem where it can't find a reference is a bug (that I >> >>> haven't >> >>> >>> >> fixed yet) caused by the lifting mechanics not really being >> >>> completely >> >>> >>> >> in sync with the reference mechanics with respect to nodes that >> >>> have >> >>> >>> >> multiple definition fields. When there are multiple definition >> >>> >>> >> fields, definitions in the lower field can reference the >> >>> definitions >> >>> >>> >> in the higher field, but not vice versa. The lifting machinery >> >>> >>> >> doesn't know about that ordering, it just stuffs the lifted >> >>> definition >> >>> >>> >> into the front of the definitions in whatever field it is >> >>> targetting. >> >>> >>> >> Thus it places a definition in a place that the reference >> machinery >> >>> >>> >> thinks is out of scope. >> >>> >>> >> >> >>> >>> >> So... if you want to have different binders that get different >> >>> fields >> >>> >>> >> in the program node you can only make it work if you disallow >> >>> stuff to >> >>> >>> >> be sure you never need to reference something that will be >> lifted >> >>> into >> >>> >>> >> the lower field. >> >>> >>> >> >> >>> >>> >> That said, I've just fixed some issues to allow definition nodes >> >>> to be >> >>> >>> >> subtypable, so now you can have a definition node that is a >> >>> subtype of >> >>> >>> >> Definition, but has some different rules. Importantly, though, >> it >> >>> can >> >>> >>> >> be lifted to the same field. >> >>> >>> >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your example to >> >>> look >> >>> >>> >> like this: >> >>> >>> >> >> >>> >>> >> ``` >> >>> >>> >> [MyDefinition Definition () >> >>> >>> >> #:prop fresh (hash 'Expression (make-hole >> >>> >>> >> 'ProcedureApplication))] >> >>> >>> >> ``` >> >>> >>> >> >> >>> >>> >> It can now be lifted among normal definitions, but always have >> >>> >>> >> ProcedureApplication as its RHS. >> >>> >>> >> >> >>> >>> >> However, I'm still not convinced that you actually want >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict >> lambda >> >>> >>> >> nodes to only be generated on the RHS of a FunctionDefinition >> node >> >>> >>> >> (that's a subtype of Definition), and make FunctionDefinition >> >>> always >> >>> >>> >> have a Lambda node as its RHS. >> >>> >>> >> >> >>> >>> >> There is an example in the documentation that is like this, in >> the >> >>> >>> >> explanation of the `choice-filters-to-apply` property. But I'll >> >>> paste >> >>> >>> >> it here: >> >>> >>> >> >> >>> >>> >> ``` >> >>> >>> >> (add-choice-method >> >>> >>> >> my-component >> >>> >>> >> no-lambda-except-global-def >> >>> >>> >> [#f (? () #t)] >> >>> >>> >> [Lambda (? () >> >>> >>> >> (and (parent-node current-hole) >> >>> >>> >> (equal? (ast-node-type (parent-node >> current-hole)) >> >>> >>> >> 'Definition) >> >>> >>> >> (parent-node (parent-node current-hole)) >> >>> >>> >> (equal? (ast-node-type (parent-node >> (parent-node >> >>> >>> >> current-hole))) >> >>> >>> >> 'Program)))]) >> >>> >>> >> (add-property my-component >> >>> >>> >> choice-filters-to-apply >> >>> >>> >> [#f (no-lambda-except-global-def)]) >> >>> >>> >> ``` >> >>> >>> >> >> >>> >>> >> You can use something like that to allow Lambda nodes to only be >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and define >> >>> >>> >> FunctionDefinition like MyDefinition above to always generate >> >>> Lambda >> >>> >>> >> as its RHS, and I think it should work for a first-order >> language. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type >> variables >> >>> >>> >> >with some concrete non-function type variables, but down the >> >>> road, I >> >>> >>> get >> >>> >>> >> an >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not >> >>> supposed >> >>> >>> to >> >>> >>> >> >unify! inside add-choice-method? >> >>> >>> >> >> >>> >>> >> Well, probably not. Generally the only place you want to >> `unify!` >> >>> is >> >>> >>> >> inside the type property in the function that determines child >> >>> types >> >>> >>> >> based on the parent type. Inside that function you can >> `unify!`, >> >>> but >> >>> >>> >> the unification needs to be consistent (IE not unified based on >> the >> >>> >>> >> result of `random`) OR the choice needs to be saved in a way >> that >> >>> it >> >>> >>> >> can be looked up when that function is run again. >> >>> >>> >> >> >>> >>> >> The problem is that the choice method gets its type information >> >>> based >> >>> >>> >> on a hole node that is later replaced, so the type is recomputed >> >>> >>> >> without necessarily consulting the type that the hole said it >> was. >> >>> >>> >> You should be able to constrain the type using unification if >> you >> >>> save >> >>> >>> >> that type to re-unify it consistently later. But without seeing >> >>> the >> >>> >>> >> exact error and code that generated it, I'm not 100% certain. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type >> could be >> >>> >>> used >> >>> >>> >> > nondeterministically in this manner, and my experience with it >> >>> so far >> >>> >>> >> seems >> >>> >>> >> > to indicate no. >> >>> >>> >> >> >>> >>> >> I think this should work, barring the issues mentioned above >> about >> >>> the >> >>> >>> >> lifting vs referencing bug when you have the different >> definition >> >>> >>> >> types in different fields. Adjusting your example to have >> >>> >>> >> MyDefinition as a subtype, and then only having a Definition >> field >> >>> in >> >>> >>> >> MyProgram, random choice seems to work fine. But sometimes it >> >>> >>> >> generates gargantuan programs (that I kill before it finishes) >> >>> because >> >>> >>> >> it keeps choosing MyDefinition, requiring a function >> application, >> >>> >>> >> which likely lifts a reference to a function, which might choose >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger and >> >>> >>> >> bigger types. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Anyway, does that all help? >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee >> Porncharoenwase >> >>> >>> wrote: >> >>> >>> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded as >> a >> >>> >>> >> >> function application node that only allows a reference as its >> >>> RHS >> >>> >>> >> >> rather than allowing arbitrary expressions. But that said, >> I'm >> >>> not >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >>> >>> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >>> >>> >> > >> >>> >>> >> > 1. I have no problem with the definition site. That?s easy >> to >> >>> deal >> >>> >>> >> with. >> >>> >>> >> > 2. I also have no problem with restricting function >> >>> application. >> >>> >>> That?s >> >>> >>> >> > easy to deal with as well. >> >>> >>> >> > 3. The problem is primarily with a bare variable reference >> >>> bound >> >>> >>> to a >> >>> >>> >> > closure value, which must only occur in function >> application. >> >>> For >> >>> >>> >> example, >> >>> >>> >> > the following code is invalid. >> >>> >>> >> > >> >>> >>> >> >(define (f) (void)) >> >>> >>> >> > >> >>> >>> >> >(define g f) ;; this is invalid >> >>> >>> >> >(define h (g)) >> >>> >>> >> >(define a (list f)) ;; this is also invalid >> >>> >>> >> >(define b (f)) ;; this is ok >> >>> >>> >> > >> >>> >>> >> >I have been using choice-filters-to-apply to restrict variable >> >>> >>> references >> >>> >>> >> >whose type can-unify? with function-type that doesn't directly >> >>> appear >> >>> >>> >> under >> >>> >>> >> >function application. But this is obviously not ideal because >> the >> >>> >>> filtered >> >>> >>> >> >nodes are mostly those whose type is totally unrestricted. So >> >>> >>> filtering >> >>> >>> >> >them would prevent them from being concretized to other >> >>> non-function >> >>> >>> types >> >>> >>> >> >as well. (Also note that I tried unify! these unrestricted type >> >>> >>> variables >> >>> >>> >> >with some concrete non-function type variables, but down the >> >>> road, I >> >>> >>> get >> >>> >>> >> an >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not >> >>> supposed >> >>> >>> to >> >>> >>> >> >unify! inside add-choice-method? >> >>> >>> >> > >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the >> >>> situation >> >>> >>> >> >> where a reference is generated that causes a new definition >> to >> >>> be >> >>> >>> >> >> added such that the variable is in scope at the use site. IE >> >>> the >> >>> >>> >> >> binding is lifted from the use site to some outer node that >> >>> contains >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to >> definitions >> >>> that >> >>> >>> >> >> are created automatically in this way (which is ultimately >> most >> >>> >>> >> >> definitions). The exception is caused because somehow the >> lift >> >>> >>> ended >> >>> >>> >> >> up in a place where it wasn't actually in scope. That >> shouldn't >> >>> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer >> that >> >>> does >> >>> >>> >> >> this, I'll try to debug it. >> >>> >>> >> >> >> >>> >>> >> >#lang clotho >> >>> >>> >> > >> >>> >>> >> >(require >> >>> >>> >> > xsmith >> >>> >>> >> > racr >> >>> >>> >> > xsmith/racr-convenience >> >>> >>> >> > xsmith/canned-components >> >>> >>> >> > racket/string >> >>> >>> >> > racket/list >> >>> >>> >> > racket/pretty) >> >>> >>> >> > >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >>> expressions. >> >>> >>> >> > >> >>> >>> >> >(define-basic-spec-component somelisp) >> >>> >>> >> > >> >>> >>> >> >(add-basic-expressions somelisp >> >>> >>> >> > #:ProgramWithSequence #t >> >>> >>> >> > #:VoidExpression #t >> >>> >>> >> > #:AssignmentExpression #t >> >>> >>> >> > #:VariableReference #t >> >>> >>> >> > #:ProcedureApplication #t >> >>> >>> >> > #:IfExpression #t >> >>> >>> >> > #:ExpressionSequence #t >> >>> >>> >> > #:LetSequential #t >> >>> >>> >> > #:LambdaWithExpression #t >> >>> >>> >> > #:Numbers #t >> >>> >>> >> > #:Booleans #t >> >>> >>> >> > #:Strings #t >> >>> >>> >> > #:ImmutableList #t) >> >>> >>> >> > >> >>> >>> >> >(add-loop-over-container >> >>> >>> >> > somelisp >> >>> >>> >> > #:name Map >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> >>> >>> inner))) >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >>> inner)))) >> >>> >>> >> > >> >>> >>> >> >(add-to-grammar >> >>> >>> >> > somelisp >> >>> >>> >> > [MyProgram #f >> >>> >>> >> > ([decls : Definition *] >> >>> >>> >> > [declsTwo : MyDefinition *] >> >>> >>> >> > [body : Expression]) >> >>> >>> >> > #:prop type-info >> >>> >>> >> > [(fresh-type-variable) >> >>> >>> >> > (? (n t) >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >>> >>> >> > [MyDefinition #f ([type] >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >>> >>> >> > [e : ProcedureApplication]) >> >>> >>> >> > #:prop binder-info () >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >> >>> (hash >> >>> >>> 'e >> >>> >>> >> t))]]) >> >>> >>> >> > >> >>> >>> >> >(add-property >> >>> >>> >> > somelisp >> >>> >>> >> > lift-type->ast-binder-type >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >>> 'Definition))]) >> >>> >>> >> > >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete but >> >>> needs >> >>> >>> to be. >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >>> unconstrained >> >>> >>> >> >;; type variable. >> >>> >>> >> >(define (type-thunks-for-concretization) >> >>> >>> >> > (list >> >>> >>> >> > (?()int-type) >> >>> >>> >> > (?()bool-type) >> >>> >>> >> > (?()string-type) >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >>> >>> >> > >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >>> >>> >> > (define out (open-output-string)) >> >>> >>> >> > (for ([symex s-exps]) >> >>> >>> >> > (pretty-print symex out 1)) >> >>> >>> >> > (get-output-string out)) >> >>> >>> >> > >> >>> >>> >> >(define-xsmith-interface-functions >> >>> >>> >> > [somelisp] >> >>> >>> >> > #:program-node MyProgram >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >>> >>> >> > #:comment-wrap (? (lines) >> >>> >>> >> > (string-join >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >>> >>> >> > "\n")) >> >>> >>> >> > #:format-render somelisp-format-render) >> >>> >>> >> > >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> >> >> Probably something is going wrong in the `fresh` property. >> >>> Have you >> >>> >>> >> >> changed that? >> >>> >>> >> >> >> >>> >>> >> >I don?t think so. See the code at the end of the email. >> >>> >>> >> > >> >>> >>> >> >> I should change that error message. Anyway, it seems to >> think >> >>> there >> >>> >>> >> >> is no outer node that has a `Definition *` child that can be >> the >> >>> >>> >> >> target for lifting a definition. Generally this happens when >> >>> you >> >>> >>> >> >> forget to have an outer Program node or something that can >> hold >> >>> >>> >> >> definitions. >> >>> >>> >> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. >> >>> >>> >> > >> >>> >>> >> >#lang clotho >> >>> >>> >> > >> >>> >>> >> >(require >> >>> >>> >> > xsmith >> >>> >>> >> > racr >> >>> >>> >> > xsmith/racr-convenience >> >>> >>> >> > xsmith/canned-components >> >>> >>> >> > racket/string >> >>> >>> >> > racket/list >> >>> >>> >> > racket/pretty) >> >>> >>> >> > >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >>> expressions. >> >>> >>> >> > >> >>> >>> >> >(define-basic-spec-component somelisp) >> >>> >>> >> > >> >>> >>> >> >(add-basic-expressions somelisp >> >>> >>> >> > #:ProgramWithSequence #t >> >>> >>> >> > #:VoidExpression #t >> >>> >>> >> > #:AssignmentExpression #t >> >>> >>> >> > #:VariableReference #t >> >>> >>> >> > #:ProcedureApplication #t >> >>> >>> >> > #:IfExpression #t >> >>> >>> >> > #:ExpressionSequence #t >> >>> >>> >> > #:LetSequential #t >> >>> >>> >> > #:LambdaWithExpression #t >> >>> >>> >> > #:Numbers #t >> >>> >>> >> > #:Booleans #t >> >>> >>> >> > #:Strings #t >> >>> >>> >> > #:ImmutableList #t) >> >>> >>> >> > >> >>> >>> >> >(add-loop-over-container >> >>> >>> >> > somelisp >> >>> >>> >> > #:name Map >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable (list-type >> >>> >>> inner))) >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >>> inner)))) >> >>> >>> >> > >> >>> >>> >> >(add-to-grammar >> >>> >>> >> > somelisp >> >>> >>> >> > [MyProgram #f >> >>> >>> >> > ([decls : Definition *] >> >>> >>> >> > [body : SubProgram]) >> >>> >>> >> > #:prop type-info >> >>> >>> >> > [(fresh-type-variable) >> >>> >>> >> > (? (n t) >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >>> >>> >> > [SubProgram #f >> >>> >>> >> > ([decls : MyDefinition *] >> >>> >>> >> > [body : Expression]) >> >>> >>> >> > #:prop type-info >> >>> >>> >> > [(fresh-type-variable) >> >>> >>> >> > (? (n t) >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >>> >>> >> > [MyDefinition #f ([type] >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >>> >>> >> > [e : ProcedureApplication]) >> >>> >>> >> > #:prop binder-info () >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? (n t) >> >>> (hash >> >>> >>> 'e >> >>> >>> >> t))]]) >> >>> >>> >> > >> >>> >>> >> >(add-property >> >>> >>> >> > somelisp >> >>> >>> >> > lift-type->ast-binder-type >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >>> 'Definition))]) >> >>> >>> >> > >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete but >> >>> needs >> >>> >>> to be. >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >>> unconstrained >> >>> >>> >> >;; type variable. >> >>> >>> >> >(define (type-thunks-for-concretization) >> >>> >>> >> > (list >> >>> >>> >> > (?()int-type) >> >>> >>> >> > (?()bool-type) >> >>> >>> >> > (?()string-type) >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >>> >>> >> > >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >>> >>> >> > (define out (open-output-string)) >> >>> >>> >> > (for ([symex s-exps]) >> >>> >>> >> > (pretty-print symex out 1)) >> >>> >>> >> > (get-output-string out)) >> >>> >>> >> > >> >>> >>> >> >(define-xsmith-interface-functions >> >>> >>> >> > [somelisp] >> >>> >>> >> > #:program-node MyProgram >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >>> >>> >> > #:comment-wrap (? (lines) >> >>> >>> >> > (string-join >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >>> >>> >> > "\n")) >> >>> >>> >> > #:format-render somelisp-format-render) >> >>> >>> >> > >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >>> >>> >> >> >>> >>> >> >>> >> >> >>> >> >> >> From sorawee.pwase at gmail.com Wed Jul 21 16:45:38 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Wed, 21 Jul 2021 15:45:38 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: I submitted a PR that fixes the problem. current-xsmith-type-constructor-thunks needs to be parameterized for printing. Note that this is not only required by my fuzzer, but also by several example fuzzers that use type->string. I.e., standard-ml and cish. On Wed, Jul 21, 2021 at 3:08 PM William G Hatch wrote: > The default-base-type should never show up in a fuzzer that has > defined the type property for any node. So this is definitely a bug. > I had the default-base-type show up once before, but some other change > made me stop seeing it, and I never got to the bottom of it. It's > probably because of the #:type-thunks argument that you need for > concretizing types of lifted definitions. The default was a list with > the default base type. This was always a bad idea. I've just pushed > a change that instead errors if the concretization list is used > without being parameterized with something new first. > > At any rate, I think this is an xsmith bug, not a bug in your code. > Somewhere the `current-xsmith-type-constructor-thunks` parameter is > being called where it hasn't been parameterized. If the new exception > that will be raised gives any useful information about where this is > happening, let me know. > > On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase wrote: > >Sure, I can do that. > > > >I have another question about type->string. I have been using type->string > >from standard-ml / cish examples. > > > >(define (type->string t) > > (let loop ([t* t]) > > (define t (concretize-type t*)) > > (unify! t t*) > > (cond > > [(can-unify? t (immutable (array-type (fresh-type-variable)))) > > (define inner (fresh-type-variable)) > > (unify! t (immutable (array-type inner))) > > (format "seq<~a>" (loop inner))] > > [(can-unify? t (product-type #f)) > > (match (product-type-inner-type-list t) > > [(list) "()"] > > [(list a) (loop a)] > > [inners (format "(~a)" (string-join (map loop inners) ", "))])] > > [(can-unify? t (function-type (fresh-type-variable) > >(fresh-type-variable))) > > (define ret (fresh-type-variable)) > > (define arg (fresh-type-variable)) > > (unify! t (function-type arg ret)) > > (format "(~a -> ~a)" (loop arg) (loop ret))] > > [(can-unify? t number-type) "int"] > > [(can-unify? t int-type) "int"] > > [(can-unify? t bool-type) "bool"] > > [(can-unify? t string-type) (if (zero? (random 2)) "string" > "seq")] > > [else #;(error 'type->string "Type not implemented yet: ~v" t) > > "FOOBAR"]))) > > > >The first weird thing is that the number-type branch is necessary, despite > >the fact that I don?t have any float in the program. > > > >The second weird thing is that sometimes "FOOBAR" is reached because t is > >#. In particular, it generates the following > code: > > > >safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) > ># roughly (vector-set (let ([theArray (vector) : (array (array > >FOOBAR))]) theArray) 1 (vector 1)) > > > >where you can see that ?FOOBAR? here should really be int. > > > >In my experiment, replacing default-base-type with int seems to work well, > >but it indicates something is obviously wrong. Here?s the call site of > >type->string FWIW: > > > >[ImmutableArrayLiteral > > (? (n) > > (match (ast-children (ast-child 'expressions n)) > > [(list) > > ;; sorry for convoluted logic, but I used to use inner-type for > >something and haven't refactored this. > > (define inner-type (fresh-type-variable)) > > (define the-array-type (immutable (array-type inner-type))) > > (define the-type (concretize-type ($xsmith_type n))) > > (unify! the-type the-array-type) > > (text (format "(var theArray : ~a := []; theArray)" > >(type->string the-type)))] > > [xs (h-append lbracket > > (comma-list (pretty-print-children xs)) > > rbracket)]))] > > > >Any ideas? > > > > > >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch > wrote: > > > >> Awesome! We have a git repo where we are keeping track of bugs we > >> find with Xsmith. If you want, we could add you to it and you can add > >> any bugs you find to it. Otherwise I will probably put in anything > >> you mention like this. > >> > >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase wrote: > >> >By the way, here are the results so far: > >> > > >> >- https://github.com/dafny-lang/dafny/issues/1291 > >> >- https://github.com/dafny-lang/dafny/issues/1297 > >> > > >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < > >> >sorawee.pwase at gmail.com> wrote: > >> > > >> >> We are primarily focused on different compiler backends for Dafny > (Dafny > >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). > >> >> > >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch > >> wrote: > >> >> > >> >>> >> Do you have a roadmap for implementing ... > >> >>> > >> >>> Not at the moment. Right now I'm focused on getting some of our > >> >>> mostly-finished fuzzers *really* finished and ready for fuzzing, > >> >>> running some fuzzing campaigns to find some bugs, and some features > >> >>> that are immediately helpful for running fuzzing (eg. some stuff to > >> >>> support feedback-directed fuzzing and choice weighting). > >> >>> > >> >>> > >> >>> >Also, I'm curious about the setup of various fuzzers that you are > >> >>> running. > >> >>> >How many programs are you able to generate and test per minute? Do > you > >> >>> have > >> >>> >any particular parallelization setup that works well for you? > >> >>> > >> >>> There is a harness repository that we use that was mostly written by > >> >>> Guy Watson: > >> >>> > >> >>> https://gitlab.flux.utah.edu/xsmith/harness > >> >>> > >> >>> There is a `harness.py` script that takes a configuration file to > run > >> >>> an xsmith fuzzer, running its output through various implementations > >> >>> of whatever language, comparing the output, and binning the result > >> >>> (eg. everything matching, diff, crash, everything crashing (IE bad > >> >>> program), xsmith itself crashing, ...). That harness script is > >> >>> basically for running a single job, and there are 2 ways of doing it > >> >>> in parallel. > >> >>> > >> >>> The first one is the `go-button`, which can run fuzzing on multiple > >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab > >> >>> (https://www.emulab.net/) to have things set up just right, and > >> >>> requires at least 2 machines. It also generates a bunch of live > >> >>> graphs about fuzzing. It uses ansible to run setup scripts to prep > >> >>> each machine before starting the fuzz campaign. > >> >>> > >> >>> I mostly use `local-multi-harness` instead, which just runs on a > >> >>> single machine, but runs multiple harness processes. It doesn't > >> >>> automatically run the setup scripts like the `go-button` does, so > >> >>> eg. if you want to use the `racket-kernel-fuzzer` configuration you > >> >>> need to run the `racket-install.rkt` script first manually. > >> >>> > >> >>> As far as speed, xsmith is not particularly fast. There is a server > >> >>> mode that at least saves on startup time, but I think the harness > >> >>> might not run it particularly well (Guy correct me if I'm wrong -- > at > >> >>> any rate I haven't been using it with server mode, but that might be > >> >>> because I never bothered changing my config after bugs were fixed). > >> >>> Aside from Racket's slow startup speed (particularly bad with Xsmith > >> >>> because it loads a lot of stuff), the runtime varies a lot based on > >> >>> the max-depth configuration, the size of the grammar, etc. > >> >>> > >> >>> I haven't focused a lot on performance, necessarily. Basically > >> >>> whenever I decide Xsmith is too slow for my taste I spend some time > >> >>> trying to optimize it, and generally make a few gains until I think > >> >>> ?that's enough optimization for now?. If you can find any ways to > >> >>> make improvements, that would be great! I might have taken all of > the > >> >>> low-hanging fruit in my previous optimization passes, however. In > >> >>> previous optimization passes I got the most out of optimizing in the > >> >>> type system, but I made some major improvements to type checking > speed > >> >>> last time so it may not dominate anymore. > >> >>> > >> >>> > >> >>> As an aside, a question for you: Are there multiple Dafny > >> >>> implementations? Or different optimization levels? Or are you > >> >>> running different versions of the compiler against each other? > >> >>> > >> >>> > >> >>> > >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase > >> wrote: > >> >>> >Also, I'm curious about the setup of various fuzzers that you are > >> >>> running. > >> >>> >How many programs are you able to generate and test per minute? Do > you > >> >>> have > >> >>> >any particular parallelization setup that works well for you? > >> >>> > > >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < > >> >>> >sorawee.pwase at gmail.com> wrote: > >> >>> > > >> >>> >> Do you have a roadmap for implementing "Provide mechanism for > >> >>> precluding > >> >>> >> types in `fresh-type-variable` invocations"? ( > >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also > >> really > >> >>> >> want this feature. > >> >>> >> > >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch > > >> >>> wrote: > >> >>> >> > >> >>> >>> That would be great, thanks! > >> >>> >>> > >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee > Porncharoenwase > >> >>> wrote: > >> >>> >>> >Yes, this is very helpful. I will let you know if I encounter > any > >> >>> further > >> >>> >>> >issues after updating XSmith. > >> >>> >>> > > >> >>> >>> >By the way, do you want any help with Scribble? I notice a lot > of > >> >>> typos > >> >>> >>> and > >> >>> >>> >link errors and can submit PRs to fix them. > >> >>> >>> > > >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > >> > > >> >>> >>> wrote: > >> >>> >>> > > >> >>> >>> >> Sorry for the delay. I wanted to do the debugging before > >> >>> responding, > >> >>> >>> >> but then I didn't get to it for a while. > >> >>> >>> >> > >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some > bugs. > >> >>> >>> >> > >> >>> >>> >> First, that issue with the bud-node was a bug in the `edit` > >> >>> property > >> >>> >>> >> which the canned-components `ProcedureApplication` uses. > It's > >> >>> fixed > >> >>> >>> >> now. > >> >>> >>> >> > >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded > as > >> a > >> >>> >>> >> >> function application node that only allows a reference as > its > >> >>> RHS > >> >>> >>> >> >> rather than allowing arbitrary expressions. But that > said, > >> I'm > >> >>> not > >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >>> >>> >> >> > >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >>> >>> >> > >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be > >> >>> function > >> >>> >>> >> application that only allows a reference in the procedure > >> >>> position. I > >> >>> >>> >> don't know why I wrote RHS. > >> >>> >>> >> > >> >>> >>> >> Anyway, the strategy of having separate definition nodes for > >> >>> functions > >> >>> >>> >> and everything else is used in Cish. I don't recommend > >> following > >> >>> >>> >> anything Cish does. However, it accomplishes this by having > the > >> >>> >>> >> program node have non-function definitions first, and > >> disallowing > >> >>> use > >> >>> >>> >> of functions on the RHS of those functions. > >> >>> >>> >> > >> >>> >>> >> The problem where it can't find a reference is a bug (that I > >> >>> haven't > >> >>> >>> >> fixed yet) caused by the lifting mechanics not really being > >> >>> completely > >> >>> >>> >> in sync with the reference mechanics with respect to nodes > that > >> >>> have > >> >>> >>> >> multiple definition fields. When there are multiple > definition > >> >>> >>> >> fields, definitions in the lower field can reference the > >> >>> definitions > >> >>> >>> >> in the higher field, but not vice versa. The lifting > machinery > >> >>> >>> >> doesn't know about that ordering, it just stuffs the lifted > >> >>> definition > >> >>> >>> >> into the front of the definitions in whatever field it is > >> >>> targetting. > >> >>> >>> >> Thus it places a definition in a place that the reference > >> machinery > >> >>> >>> >> thinks is out of scope. > >> >>> >>> >> > >> >>> >>> >> So... if you want to have different binders that get > different > >> >>> fields > >> >>> >>> >> in the program node you can only make it work if you disallow > >> >>> stuff to > >> >>> >>> >> be sure you never need to reference something that will be > >> lifted > >> >>> into > >> >>> >>> >> the lower field. > >> >>> >>> >> > >> >>> >>> >> That said, I've just fixed some issues to allow definition > nodes > >> >>> to be > >> >>> >>> >> subtypable, so now you can have a definition node that is a > >> >>> subtype of > >> >>> >>> >> Definition, but has some different rules. Importantly, > though, > >> it > >> >>> can > >> >>> >>> >> be lifted to the same field. > >> >>> >>> >> > >> >>> >>> >> Eg. if I change the MyDefinition declaration in your example > to > >> >>> look > >> >>> >>> >> like this: > >> >>> >>> >> > >> >>> >>> >> ``` > >> >>> >>> >> [MyDefinition Definition () > >> >>> >>> >> #:prop fresh (hash 'Expression (make-hole > >> >>> >>> >> 'ProcedureApplication))] > >> >>> >>> >> ``` > >> >>> >>> >> > >> >>> >>> >> It can now be lifted among normal definitions, but always > have > >> >>> >>> >> ProcedureApplication as its RHS. > >> >>> >>> >> > >> >>> >>> >> However, I'm still not convinced that you actually want > >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict > >> lambda > >> >>> >>> >> nodes to only be generated on the RHS of a FunctionDefinition > >> node > >> >>> >>> >> (that's a subtype of Definition), and make FunctionDefinition > >> >>> always > >> >>> >>> >> have a Lambda node as its RHS. > >> >>> >>> >> > >> >>> >>> >> There is an example in the documentation that is like this, > in > >> the > >> >>> >>> >> explanation of the `choice-filters-to-apply` property. But > I'll > >> >>> paste > >> >>> >>> >> it here: > >> >>> >>> >> > >> >>> >>> >> ``` > >> >>> >>> >> (add-choice-method > >> >>> >>> >> my-component > >> >>> >>> >> no-lambda-except-global-def > >> >>> >>> >> [#f (? () #t)] > >> >>> >>> >> [Lambda (? () > >> >>> >>> >> (and (parent-node current-hole) > >> >>> >>> >> (equal? (ast-node-type (parent-node > >> current-hole)) > >> >>> >>> >> 'Definition) > >> >>> >>> >> (parent-node (parent-node current-hole)) > >> >>> >>> >> (equal? (ast-node-type (parent-node > >> (parent-node > >> >>> >>> >> current-hole))) > >> >>> >>> >> 'Program)))]) > >> >>> >>> >> (add-property my-component > >> >>> >>> >> choice-filters-to-apply > >> >>> >>> >> [#f (no-lambda-except-global-def)]) > >> >>> >>> >> ``` > >> >>> >>> >> > >> >>> >>> >> You can use something like that to allow Lambda nodes to > only be > >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and define > >> >>> >>> >> FunctionDefinition like MyDefinition above to always generate > >> >>> Lambda > >> >>> >>> >> as its RHS, and I think it should work for a first-order > >> language. > >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> >> >(Also note that I tried unify! these unrestricted type > >> variables > >> >>> >>> >> >with some concrete non-function type variables, but down the > >> >>> road, I > >> >>> >>> get > >> >>> >>> >> an > >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not > >> >>> supposed > >> >>> >>> to > >> >>> >>> >> >unify! inside add-choice-method? > >> >>> >>> >> > >> >>> >>> >> Well, probably not. Generally the only place you want to > >> `unify!` > >> >>> is > >> >>> >>> >> inside the type property in the function that determines > child > >> >>> types > >> >>> >>> >> based on the parent type. Inside that function you can > >> `unify!`, > >> >>> but > >> >>> >>> >> the unification needs to be consistent (IE not unified based > on > >> the > >> >>> >>> >> result of `random`) OR the choice needs to be saved in a way > >> that > >> >>> it > >> >>> >>> >> can be looked up when that function is run again. > >> >>> >>> >> > >> >>> >>> >> The problem is that the choice method gets its type > information > >> >>> based > >> >>> >>> >> on a hole node that is later replaced, so the type is > recomputed > >> >>> >>> >> without necessarily consulting the type that the hole said it > >> was. > >> >>> >>> >> You should be able to constrain the type using unification if > >> you > >> >>> save > >> >>> >>> >> that type to re-unify it consistently later. But without > seeing > >> >>> the > >> >>> >>> >> exact error and code that generated it, I'm not 100% certain. > >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type > >> could be > >> >>> >>> used > >> >>> >>> >> > nondeterministically in this manner, and my experience > with it > >> >>> so far > >> >>> >>> >> seems > >> >>> >>> >> > to indicate no. > >> >>> >>> >> > >> >>> >>> >> I think this should work, barring the issues mentioned above > >> about > >> >>> the > >> >>> >>> >> lifting vs referencing bug when you have the different > >> definition > >> >>> >>> >> types in different fields. Adjusting your example to have > >> >>> >>> >> MyDefinition as a subtype, and then only having a Definition > >> field > >> >>> in > >> >>> >>> >> MyProgram, random choice seems to work fine. But sometimes > it > >> >>> >>> >> generates gargantuan programs (that I kill before it > finishes) > >> >>> because > >> >>> >>> >> it keeps choosing MyDefinition, requiring a function > >> application, > >> >>> >>> >> which likely lifts a reference to a function, which might > choose > >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger > and > >> >>> >>> >> bigger types. > >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> >> Anyway, does that all help? > >> >>> >>> >> > >> >>> >>> >> > >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee > >> Porncharoenwase > >> >>> >>> wrote: > >> >>> >>> >> >> > >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded > as > >> a > >> >>> >>> >> >> function application node that only allows a reference as > its > >> >>> RHS > >> >>> >>> >> >> rather than allowing arbitrary expressions. But that > said, > >> I'm > >> >>> not > >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >>> >>> >> >> > >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >>> >>> >> > > >> >>> >>> >> > 1. I have no problem with the definition site. That?s > easy > >> to > >> >>> deal > >> >>> >>> >> with. > >> >>> >>> >> > 2. I also have no problem with restricting function > >> >>> application. > >> >>> >>> That?s > >> >>> >>> >> > easy to deal with as well. > >> >>> >>> >> > 3. The problem is primarily with a bare variable > reference > >> >>> bound > >> >>> >>> to a > >> >>> >>> >> > closure value, which must only occur in function > >> application. > >> >>> For > >> >>> >>> >> example, > >> >>> >>> >> > the following code is invalid. > >> >>> >>> >> > > >> >>> >>> >> >(define (f) (void)) > >> >>> >>> >> > > >> >>> >>> >> >(define g f) ;; this is invalid > >> >>> >>> >> >(define h (g)) > >> >>> >>> >> >(define a (list f)) ;; this is also invalid > >> >>> >>> >> >(define b (f)) ;; this is ok > >> >>> >>> >> > > >> >>> >>> >> >I have been using choice-filters-to-apply to restrict > variable > >> >>> >>> references > >> >>> >>> >> >whose type can-unify? with function-type that doesn't > directly > >> >>> appear > >> >>> >>> >> under > >> >>> >>> >> >function application. But this is obviously not ideal > because > >> the > >> >>> >>> filtered > >> >>> >>> >> >nodes are mostly those whose type is totally unrestricted. > So > >> >>> >>> filtering > >> >>> >>> >> >them would prevent them from being concretized to other > >> >>> non-function > >> >>> >>> types > >> >>> >>> >> >as well. (Also note that I tried unify! these unrestricted > type > >> >>> >>> variables > >> >>> >>> >> >with some concrete non-function type variables, but down the > >> >>> road, I > >> >>> >>> get > >> >>> >>> >> an > >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not > >> >>> supposed > >> >>> >>> to > >> >>> >>> >> >unify! inside add-choice-method? > >> >>> >>> >> > > >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the > >> >>> situation > >> >>> >>> >> >> where a reference is generated that causes a new > definition > >> to > >> >>> be > >> >>> >>> >> >> added such that the variable is in scope at the use > site. IE > >> >>> the > >> >>> >>> >> >> binding is lifted from the use site to some outer node > that > >> >>> contains > >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to > >> definitions > >> >>> that > >> >>> >>> >> >> are created automatically in this way (which is ultimately > >> most > >> >>> >>> >> >> definitions). The exception is caused because somehow the > >> lift > >> >>> >>> ended > >> >>> >>> >> >> up in a place where it wasn't actually in scope. That > >> shouldn't > >> >>> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer > >> that > >> >>> does > >> >>> >>> >> >> this, I'll try to debug it. > >> >>> >>> >> >> > >> >>> >>> >> >#lang clotho > >> >>> >>> >> > > >> >>> >>> >> >(require > >> >>> >>> >> > xsmith > >> >>> >>> >> > racr > >> >>> >>> >> > xsmith/racr-convenience > >> >>> >>> >> > xsmith/canned-components > >> >>> >>> >> > racket/string > >> >>> >>> >> > racket/list > >> >>> >>> >> > racket/pretty) > >> >>> >>> >> > > >> >>> >>> >> >;; We first define a basic component and add a bunch of > >> >>> expressions. > >> >>> >>> >> > > >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >>> >>> >> > > >> >>> >>> >> >(add-basic-expressions somelisp > >> >>> >>> >> > #:ProgramWithSequence #t > >> >>> >>> >> > #:VoidExpression #t > >> >>> >>> >> > #:AssignmentExpression #t > >> >>> >>> >> > #:VariableReference #t > >> >>> >>> >> > #:ProcedureApplication #t > >> >>> >>> >> > #:IfExpression #t > >> >>> >>> >> > #:ExpressionSequence #t > >> >>> >>> >> > #:LetSequential #t > >> >>> >>> >> > #:LambdaWithExpression #t > >> >>> >>> >> > #:Numbers #t > >> >>> >>> >> > #:Booleans #t > >> >>> >>> >> > #:Strings #t > >> >>> >>> >> > #:ImmutableList #t) > >> >>> >>> >> > > >> >>> >>> >> >(add-loop-over-container > >> >>> >>> >> > somelisp > >> >>> >>> >> > #:name Map > >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > (list-type > >> >>> >>> inner))) > >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >> >>> inner)))) > >> >>> >>> >> > > >> >>> >>> >> >(add-to-grammar > >> >>> >>> >> > somelisp > >> >>> >>> >> > [MyProgram #f > >> >>> >>> >> > ([decls : Definition *] > >> >>> >>> >> > [declsTwo : MyDefinition *] > >> >>> >>> >> > [body : Expression]) > >> >>> >>> >> > #:prop type-info > >> >>> >>> >> > [(fresh-type-variable) > >> >>> >>> >> > (? (n t) > >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >> >>> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) > >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >>> >>> >> > [MyDefinition #f ([type] > >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >>> >>> >> > [e : ProcedureApplication]) > >> >>> >>> >> > #:prop binder-info () > >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? > (n t) > >> >>> (hash > >> >>> >>> 'e > >> >>> >>> >> t))]]) > >> >>> >>> >> > > >> >>> >>> >> >(add-property > >> >>> >>> >> > somelisp > >> >>> >>> >> > lift-type->ast-binder-type > >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >>> 'Definition))]) > >> >>> >>> >> > > >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete > but > >> >>> needs > >> >>> >>> to be. > >> >>> >>> >> >;; Here we provide a list of options we can pick for an > >> >>> unconstrained > >> >>> >>> >> >;; type variable. > >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >>> >>> >> > (list > >> >>> >>> >> > (?()int-type) > >> >>> >>> >> > (?()bool-type) > >> >>> >>> >> > (?()string-type) > >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> >>> >>> >> > > >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >>> >>> >> > (define out (open-output-string)) > >> >>> >>> >> > (for ([symex s-exps]) > >> >>> >>> >> > (pretty-print symex out 1)) > >> >>> >>> >> > (get-output-string out)) > >> >>> >>> >> > > >> >>> >>> >> >(define-xsmith-interface-functions > >> >>> >>> >> > [somelisp] > >> >>> >>> >> > #:program-node MyProgram > >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >>> >>> >> > #:comment-wrap (? (lines) > >> >>> >>> >> > (string-join > >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >> >>> >>> >> > "\n")) > >> >>> >>> >> > #:format-render somelisp-format-render) > >> >>> >>> >> > > >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> >> Probably something is going wrong in the `fresh` property. > >> >>> Have you > >> >>> >>> >> >> changed that? > >> >>> >>> >> >> > >> >>> >>> >> >I don?t think so. See the code at the end of the email. > >> >>> >>> >> > > >> >>> >>> >> >> I should change that error message. Anyway, it seems to > >> think > >> >>> there > >> >>> >>> >> >> is no outer node that has a `Definition *` child that can > be > >> the > >> >>> >>> >> >> target for lifting a definition. Generally this happens > when > >> >>> you > >> >>> >>> >> >> forget to have an outer Program node or something that can > >> hold > >> >>> >>> >> >> definitions. > >> >>> >>> >> >> > >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. > >> >>> >>> >> > > >> >>> >>> >> >#lang clotho > >> >>> >>> >> > > >> >>> >>> >> >(require > >> >>> >>> >> > xsmith > >> >>> >>> >> > racr > >> >>> >>> >> > xsmith/racr-convenience > >> >>> >>> >> > xsmith/canned-components > >> >>> >>> >> > racket/string > >> >>> >>> >> > racket/list > >> >>> >>> >> > racket/pretty) > >> >>> >>> >> > > >> >>> >>> >> >;; We first define a basic component and add a bunch of > >> >>> expressions. > >> >>> >>> >> > > >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >>> >>> >> > > >> >>> >>> >> >(add-basic-expressions somelisp > >> >>> >>> >> > #:ProgramWithSequence #t > >> >>> >>> >> > #:VoidExpression #t > >> >>> >>> >> > #:AssignmentExpression #t > >> >>> >>> >> > #:VariableReference #t > >> >>> >>> >> > #:ProcedureApplication #t > >> >>> >>> >> > #:IfExpression #t > >> >>> >>> >> > #:ExpressionSequence #t > >> >>> >>> >> > #:LetSequential #t > >> >>> >>> >> > #:LambdaWithExpression #t > >> >>> >>> >> > #:Numbers #t > >> >>> >>> >> > #:Booleans #t > >> >>> >>> >> > #:Strings #t > >> >>> >>> >> > #:ImmutableList #t) > >> >>> >>> >> > > >> >>> >>> >> >(add-loop-over-container > >> >>> >>> >> > somelisp > >> >>> >>> >> > #:name Map > >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > (list-type > >> >>> >>> inner))) > >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >> >>> inner)))) > >> >>> >>> >> > > >> >>> >>> >> >(add-to-grammar > >> >>> >>> >> > somelisp > >> >>> >>> >> > [MyProgram #f > >> >>> >>> >> > ([decls : Definition *] > >> >>> >>> >> > [body : SubProgram]) > >> >>> >>> >> > #:prop type-info > >> >>> >>> >> > [(fresh-type-variable) > >> >>> >>> >> > (? (n t) > >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >>> >>> >> > [SubProgram #f > >> >>> >>> >> > ([decls : MyDefinition *] > >> >>> >>> >> > [body : Expression]) > >> >>> >>> >> > #:prop type-info > >> >>> >>> >> > [(fresh-type-variable) > >> >>> >>> >> > (? (n t) > >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >>> >>> >> > [MyDefinition #f ([type] > >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >>> >>> >> > [e : ProcedureApplication]) > >> >>> >>> >> > #:prop binder-info () > >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? > (n t) > >> >>> (hash > >> >>> >>> 'e > >> >>> >>> >> t))]]) > >> >>> >>> >> > > >> >>> >>> >> >(add-property > >> >>> >>> >> > somelisp > >> >>> >>> >> > lift-type->ast-binder-type > >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >>> 'Definition))]) > >> >>> >>> >> > > >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete > but > >> >>> needs > >> >>> >>> to be. > >> >>> >>> >> >;; Here we provide a list of options we can pick for an > >> >>> unconstrained > >> >>> >>> >> >;; type variable. > >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >>> >>> >> > (list > >> >>> >>> >> > (?()int-type) > >> >>> >>> >> > (?()bool-type) > >> >>> >>> >> > (?()string-type) > >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> >>> >>> >> > > >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >>> >>> >> > (define out (open-output-string)) > >> >>> >>> >> > (for ([symex s-exps]) > >> >>> >>> >> > (pretty-print symex out 1)) > >> >>> >>> >> > (get-output-string out)) > >> >>> >>> >> > > >> >>> >>> >> >(define-xsmith-interface-functions > >> >>> >>> >> > [somelisp] > >> >>> >>> >> > #:program-node MyProgram > >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >>> >>> >> > #:comment-wrap (? (lines) > >> >>> >>> >> > (string-join > >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) > >> >>> >>> >> > "\n")) > >> >>> >>> >> > #:format-render somelisp-format-render) > >> >>> >>> >> > > >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >>> >>> >> > >> >>> >>> > >> >>> >> > >> >>> > >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Wed Jul 21 19:26:06 2021 From: william at hatch.uno (William G Hatch) Date: Wed, 21 Jul 2021 19:26:06 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Thanks! Note that if concretization needs to call up these thunks during printing it is because you still have unconstrained type variables at printing time. That's not necessarily a problem, exactly, but it might mean that thing is not really used. On Wed, Jul 21, 2021 at 03:45:38PM -0700, Sorawee Porncharoenwase wrote: >I submitted a PR that fixes the problem. >current-xsmith-type-constructor-thunks needs to be parameterized for >printing. Note that this is not only required by my fuzzer, but also by >several example fuzzers that use type->string. I.e., standard-ml and cish. > >On Wed, Jul 21, 2021 at 3:08 PM William G Hatch wrote: > >> The default-base-type should never show up in a fuzzer that has >> defined the type property for any node. So this is definitely a bug. >> I had the default-base-type show up once before, but some other change >> made me stop seeing it, and I never got to the bottom of it. It's >> probably because of the #:type-thunks argument that you need for >> concretizing types of lifted definitions. The default was a list with >> the default base type. This was always a bad idea. I've just pushed >> a change that instead errors if the concretization list is used >> without being parameterized with something new first. >> >> At any rate, I think this is an xsmith bug, not a bug in your code. >> Somewhere the `current-xsmith-type-constructor-thunks` parameter is >> being called where it hasn't been parameterized. If the new exception >> that will be raised gives any useful information about where this is >> happening, let me know. >> >> On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase wrote: >> >Sure, I can do that. >> > >> >I have another question about type->string. I have been using type->string >> >from standard-ml / cish examples. >> > >> >(define (type->string t) >> > (let loop ([t* t]) >> > (define t (concretize-type t*)) >> > (unify! t t*) >> > (cond >> > [(can-unify? t (immutable (array-type (fresh-type-variable)))) >> > (define inner (fresh-type-variable)) >> > (unify! t (immutable (array-type inner))) >> > (format "seq<~a>" (loop inner))] >> > [(can-unify? t (product-type #f)) >> > (match (product-type-inner-type-list t) >> > [(list) "()"] >> > [(list a) (loop a)] >> > [inners (format "(~a)" (string-join (map loop inners) ", "))])] >> > [(can-unify? t (function-type (fresh-type-variable) >> >(fresh-type-variable))) >> > (define ret (fresh-type-variable)) >> > (define arg (fresh-type-variable)) >> > (unify! t (function-type arg ret)) >> > (format "(~a -> ~a)" (loop arg) (loop ret))] >> > [(can-unify? t number-type) "int"] >> > [(can-unify? t int-type) "int"] >> > [(can-unify? t bool-type) "bool"] >> > [(can-unify? t string-type) (if (zero? (random 2)) "string" >> "seq")] >> > [else #;(error 'type->string "Type not implemented yet: ~v" t) >> > "FOOBAR"]))) >> > >> >The first weird thing is that the number-type branch is necessary, despite >> >the fact that I don?t have any float in the program. >> > >> >The second weird thing is that sometimes "FOOBAR" is reached because t is >> >#. In particular, it generates the following >> code: >> > >> >safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) >> ># roughly (vector-set (let ([theArray (vector) : (array (array >> >FOOBAR))]) theArray) 1 (vector 1)) >> > >> >where you can see that ?FOOBAR? here should really be int. >> > >> >In my experiment, replacing default-base-type with int seems to work well, >> >but it indicates something is obviously wrong. Here?s the call site of >> >type->string FWIW: >> > >> >[ImmutableArrayLiteral >> > (? (n) >> > (match (ast-children (ast-child 'expressions n)) >> > [(list) >> > ;; sorry for convoluted logic, but I used to use inner-type for >> >something and haven't refactored this. >> > (define inner-type (fresh-type-variable)) >> > (define the-array-type (immutable (array-type inner-type))) >> > (define the-type (concretize-type ($xsmith_type n))) >> > (unify! the-type the-array-type) >> > (text (format "(var theArray : ~a := []; theArray)" >> >(type->string the-type)))] >> > [xs (h-append lbracket >> > (comma-list (pretty-print-children xs)) >> > rbracket)]))] >> > >> >Any ideas? >> > >> > >> >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch >> wrote: >> > >> >> Awesome! We have a git repo where we are keeping track of bugs we >> >> find with Xsmith. If you want, we could add you to it and you can add >> >> any bugs you find to it. Otherwise I will probably put in anything >> >> you mention like this. >> >> >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase wrote: >> >> >By the way, here are the results so far: >> >> > >> >> >- https://github.com/dafny-lang/dafny/issues/1291 >> >> >- https://github.com/dafny-lang/dafny/issues/1297 >> >> > >> >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < >> >> >sorawee.pwase at gmail.com> wrote: >> >> > >> >> >> We are primarily focused on different compiler backends for Dafny >> (Dafny >> >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). >> >> >> >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch >> >> wrote: >> >> >> >> >> >>> >> Do you have a roadmap for implementing ... >> >> >>> >> >> >>> Not at the moment. Right now I'm focused on getting some of our >> >> >>> mostly-finished fuzzers *really* finished and ready for fuzzing, >> >> >>> running some fuzzing campaigns to find some bugs, and some features >> >> >>> that are immediately helpful for running fuzzing (eg. some stuff to >> >> >>> support feedback-directed fuzzing and choice weighting). >> >> >>> >> >> >>> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you are >> >> >>> running. >> >> >>> >How many programs are you able to generate and test per minute? Do >> you >> >> >>> have >> >> >>> >any particular parallelization setup that works well for you? >> >> >>> >> >> >>> There is a harness repository that we use that was mostly written by >> >> >>> Guy Watson: >> >> >>> >> >> >>> https://gitlab.flux.utah.edu/xsmith/harness >> >> >>> >> >> >>> There is a `harness.py` script that takes a configuration file to >> run >> >> >>> an xsmith fuzzer, running its output through various implementations >> >> >>> of whatever language, comparing the output, and binning the result >> >> >>> (eg. everything matching, diff, crash, everything crashing (IE bad >> >> >>> program), xsmith itself crashing, ...). That harness script is >> >> >>> basically for running a single job, and there are 2 ways of doing it >> >> >>> in parallel. >> >> >>> >> >> >>> The first one is the `go-button`, which can run fuzzing on multiple >> >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab >> >> >>> (https://www.emulab.net/) to have things set up just right, and >> >> >>> requires at least 2 machines. It also generates a bunch of live >> >> >>> graphs about fuzzing. It uses ansible to run setup scripts to prep >> >> >>> each machine before starting the fuzz campaign. >> >> >>> >> >> >>> I mostly use `local-multi-harness` instead, which just runs on a >> >> >>> single machine, but runs multiple harness processes. It doesn't >> >> >>> automatically run the setup scripts like the `go-button` does, so >> >> >>> eg. if you want to use the `racket-kernel-fuzzer` configuration you >> >> >>> need to run the `racket-install.rkt` script first manually. >> >> >>> >> >> >>> As far as speed, xsmith is not particularly fast. There is a server >> >> >>> mode that at least saves on startup time, but I think the harness >> >> >>> might not run it particularly well (Guy correct me if I'm wrong -- >> at >> >> >>> any rate I haven't been using it with server mode, but that might be >> >> >>> because I never bothered changing my config after bugs were fixed). >> >> >>> Aside from Racket's slow startup speed (particularly bad with Xsmith >> >> >>> because it loads a lot of stuff), the runtime varies a lot based on >> >> >>> the max-depth configuration, the size of the grammar, etc. >> >> >>> >> >> >>> I haven't focused a lot on performance, necessarily. Basically >> >> >>> whenever I decide Xsmith is too slow for my taste I spend some time >> >> >>> trying to optimize it, and generally make a few gains until I think >> >> >>> ?that's enough optimization for now?. If you can find any ways to >> >> >>> make improvements, that would be great! I might have taken all of >> the >> >> >>> low-hanging fruit in my previous optimization passes, however. In >> >> >>> previous optimization passes I got the most out of optimizing in the >> >> >>> type system, but I made some major improvements to type checking >> speed >> >> >>> last time so it may not dominate anymore. >> >> >>> >> >> >>> >> >> >>> As an aside, a question for you: Are there multiple Dafny >> >> >>> implementations? Or different optimization levels? Or are you >> >> >>> running different versions of the compiler against each other? >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase >> >> wrote: >> >> >>> >Also, I'm curious about the setup of various fuzzers that you are >> >> >>> running. >> >> >>> >How many programs are you able to generate and test per minute? Do >> you >> >> >>> have >> >> >>> >any particular parallelization setup that works well for you? >> >> >>> > >> >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >> >> >>> >sorawee.pwase at gmail.com> wrote: >> >> >>> > >> >> >>> >> Do you have a roadmap for implementing "Provide mechanism for >> >> >>> precluding >> >> >>> >> types in `fresh-type-variable` invocations"? ( >> >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I also >> >> really >> >> >>> >> want this feature. >> >> >>> >> >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >> >> >> >>> wrote: >> >> >>> >> >> >> >>> >>> That would be great, thanks! >> >> >>> >>> >> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee >> Porncharoenwase >> >> >>> wrote: >> >> >>> >>> >Yes, this is very helpful. I will let you know if I encounter >> any >> >> >>> further >> >> >>> >>> >issues after updating XSmith. >> >> >>> >>> > >> >> >>> >>> >By the way, do you want any help with Scribble? I notice a lot >> of >> >> >>> typos >> >> >>> >>> and >> >> >>> >>> >link errors and can submit PRs to fix them. >> >> >>> >>> > >> >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >> > >> > >> >> >>> >>> wrote: >> >> >>> >>> > >> >> >>> >>> >> Sorry for the delay. I wanted to do the debugging before >> >> >>> responding, >> >> >>> >>> >> but then I didn't get to it for a while. >> >> >>> >>> >> >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some >> bugs. >> >> >>> >>> >> >> >> >>> >>> >> First, that issue with the bud-node was a bug in the `edit` >> >> >>> property >> >> >>> >>> >> which the canned-components `ProcedureApplication` uses. >> It's >> >> >>> fixed >> >> >>> >>> >> now. >> >> >>> >>> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded >> as >> >> a >> >> >>> >>> >> >> function application node that only allows a reference as >> its >> >> >>> RHS >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that >> said, >> >> I'm >> >> >>> not >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >>> >>> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >>> >>> >> >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would be >> >> >>> function >> >> >>> >>> >> application that only allows a reference in the procedure >> >> >>> position. I >> >> >>> >>> >> don't know why I wrote RHS. >> >> >>> >>> >> >> >> >>> >>> >> Anyway, the strategy of having separate definition nodes for >> >> >>> functions >> >> >>> >>> >> and everything else is used in Cish. I don't recommend >> >> following >> >> >>> >>> >> anything Cish does. However, it accomplishes this by having >> the >> >> >>> >>> >> program node have non-function definitions first, and >> >> disallowing >> >> >>> use >> >> >>> >>> >> of functions on the RHS of those functions. >> >> >>> >>> >> >> >> >>> >>> >> The problem where it can't find a reference is a bug (that I >> >> >>> haven't >> >> >>> >>> >> fixed yet) caused by the lifting mechanics not really being >> >> >>> completely >> >> >>> >>> >> in sync with the reference mechanics with respect to nodes >> that >> >> >>> have >> >> >>> >>> >> multiple definition fields. When there are multiple >> definition >> >> >>> >>> >> fields, definitions in the lower field can reference the >> >> >>> definitions >> >> >>> >>> >> in the higher field, but not vice versa. The lifting >> machinery >> >> >>> >>> >> doesn't know about that ordering, it just stuffs the lifted >> >> >>> definition >> >> >>> >>> >> into the front of the definitions in whatever field it is >> >> >>> targetting. >> >> >>> >>> >> Thus it places a definition in a place that the reference >> >> machinery >> >> >>> >>> >> thinks is out of scope. >> >> >>> >>> >> >> >> >>> >>> >> So... if you want to have different binders that get >> different >> >> >>> fields >> >> >>> >>> >> in the program node you can only make it work if you disallow >> >> >>> stuff to >> >> >>> >>> >> be sure you never need to reference something that will be >> >> lifted >> >> >>> into >> >> >>> >>> >> the lower field. >> >> >>> >>> >> >> >> >>> >>> >> That said, I've just fixed some issues to allow definition >> nodes >> >> >>> to be >> >> >>> >>> >> subtypable, so now you can have a definition node that is a >> >> >>> subtype of >> >> >>> >>> >> Definition, but has some different rules. Importantly, >> though, >> >> it >> >> >>> can >> >> >>> >>> >> be lifted to the same field. >> >> >>> >>> >> >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your example >> to >> >> >>> look >> >> >>> >>> >> like this: >> >> >>> >>> >> >> >> >>> >>> >> ``` >> >> >>> >>> >> [MyDefinition Definition () >> >> >>> >>> >> #:prop fresh (hash 'Expression (make-hole >> >> >>> >>> >> 'ProcedureApplication))] >> >> >>> >>> >> ``` >> >> >>> >>> >> >> >> >>> >>> >> It can now be lifted among normal definitions, but always >> have >> >> >>> >>> >> ProcedureApplication as its RHS. >> >> >>> >>> >> >> >> >>> >>> >> However, I'm still not convinced that you actually want >> >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to restrict >> >> lambda >> >> >>> >>> >> nodes to only be generated on the RHS of a FunctionDefinition >> >> node >> >> >>> >>> >> (that's a subtype of Definition), and make FunctionDefinition >> >> >>> always >> >> >>> >>> >> have a Lambda node as its RHS. >> >> >>> >>> >> >> >> >>> >>> >> There is an example in the documentation that is like this, >> in >> >> the >> >> >>> >>> >> explanation of the `choice-filters-to-apply` property. But >> I'll >> >> >>> paste >> >> >>> >>> >> it here: >> >> >>> >>> >> >> >> >>> >>> >> ``` >> >> >>> >>> >> (add-choice-method >> >> >>> >>> >> my-component >> >> >>> >>> >> no-lambda-except-global-def >> >> >>> >>> >> [#f (? () #t)] >> >> >>> >>> >> [Lambda (? () >> >> >>> >>> >> (and (parent-node current-hole) >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> current-hole)) >> >> >>> >>> >> 'Definition) >> >> >>> >>> >> (parent-node (parent-node current-hole)) >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> (parent-node >> >> >>> >>> >> current-hole))) >> >> >>> >>> >> 'Program)))]) >> >> >>> >>> >> (add-property my-component >> >> >>> >>> >> choice-filters-to-apply >> >> >>> >>> >> [#f (no-lambda-except-global-def)]) >> >> >>> >>> >> ``` >> >> >>> >>> >> >> >> >>> >>> >> You can use something like that to allow Lambda nodes to >> only be >> >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and define >> >> >>> >>> >> FunctionDefinition like MyDefinition above to always generate >> >> >>> Lambda >> >> >>> >>> >> as its RHS, and I think it should work for a first-order >> >> language. >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type >> >> variables >> >> >>> >>> >> >with some concrete non-function type variables, but down the >> >> >>> road, I >> >> >>> >>> get >> >> >>> >>> >> an >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not >> >> >>> supposed >> >> >>> >>> to >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >>> >>> >> >> >> >>> >>> >> Well, probably not. Generally the only place you want to >> >> `unify!` >> >> >>> is >> >> >>> >>> >> inside the type property in the function that determines >> child >> >> >>> types >> >> >>> >>> >> based on the parent type. Inside that function you can >> >> `unify!`, >> >> >>> but >> >> >>> >>> >> the unification needs to be consistent (IE not unified based >> on >> >> the >> >> >>> >>> >> result of `random`) OR the choice needs to be saved in a way >> >> that >> >> >>> it >> >> >>> >>> >> can be looked up when that function is run again. >> >> >>> >>> >> >> >> >>> >>> >> The problem is that the choice method gets its type >> information >> >> >>> based >> >> >>> >>> >> on a hole node that is later replaced, so the type is >> recomputed >> >> >>> >>> >> without necessarily consulting the type that the hole said it >> >> was. >> >> >>> >>> >> You should be able to constrain the type using unification if >> >> you >> >> >>> save >> >> >>> >>> >> that type to re-unify it consistently later. But without >> seeing >> >> >>> the >> >> >>> >>> >> exact error and code that generated it, I'm not 100% certain. >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type >> >> could be >> >> >>> >>> used >> >> >>> >>> >> > nondeterministically in this manner, and my experience >> with it >> >> >>> so far >> >> >>> >>> >> seems >> >> >>> >>> >> > to indicate no. >> >> >>> >>> >> >> >> >>> >>> >> I think this should work, barring the issues mentioned above >> >> about >> >> >>> the >> >> >>> >>> >> lifting vs referencing bug when you have the different >> >> definition >> >> >>> >>> >> types in different fields. Adjusting your example to have >> >> >>> >>> >> MyDefinition as a subtype, and then only having a Definition >> >> field >> >> >>> in >> >> >>> >>> >> MyProgram, random choice seems to work fine. But sometimes >> it >> >> >>> >>> >> generates gargantuan programs (that I kill before it >> finishes) >> >> >>> because >> >> >>> >>> >> it keeps choosing MyDefinition, requiring a function >> >> application, >> >> >>> >>> >> which likely lifts a reference to a function, which might >> choose >> >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates bigger >> and >> >> >>> >>> >> bigger types. >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> Anyway, does that all help? >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee >> >> Porncharoenwase >> >> >>> >>> wrote: >> >> >>> >>> >> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be encoded >> as >> >> a >> >> >>> >>> >> >> function application node that only allows a reference as >> its >> >> >>> RHS >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that >> said, >> >> I'm >> >> >>> not >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >>> >>> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >>> >>> >> > >> >> >>> >>> >> > 1. I have no problem with the definition site. That?s >> easy >> >> to >> >> >>> deal >> >> >>> >>> >> with. >> >> >>> >>> >> > 2. I also have no problem with restricting function >> >> >>> application. >> >> >>> >>> That?s >> >> >>> >>> >> > easy to deal with as well. >> >> >>> >>> >> > 3. The problem is primarily with a bare variable >> reference >> >> >>> bound >> >> >>> >>> to a >> >> >>> >>> >> > closure value, which must only occur in function >> >> application. >> >> >>> For >> >> >>> >>> >> example, >> >> >>> >>> >> > the following code is invalid. >> >> >>> >>> >> > >> >> >>> >>> >> >(define (f) (void)) >> >> >>> >>> >> > >> >> >>> >>> >> >(define g f) ;; this is invalid >> >> >>> >>> >> >(define h (g)) >> >> >>> >>> >> >(define a (list f)) ;; this is also invalid >> >> >>> >>> >> >(define b (f)) ;; this is ok >> >> >>> >>> >> > >> >> >>> >>> >> >I have been using choice-filters-to-apply to restrict >> variable >> >> >>> >>> references >> >> >>> >>> >> >whose type can-unify? with function-type that doesn't >> directly >> >> >>> appear >> >> >>> >>> >> under >> >> >>> >>> >> >function application. But this is obviously not ideal >> because >> >> the >> >> >>> >>> filtered >> >> >>> >>> >> >nodes are mostly those whose type is totally unrestricted. >> So >> >> >>> >>> filtering >> >> >>> >>> >> >them would prevent them from being concretized to other >> >> >>> non-function >> >> >>> >>> types >> >> >>> >>> >> >as well. (Also note that I tried unify! these unrestricted >> type >> >> >>> >>> variables >> >> >>> >>> >> >with some concrete non-function type variables, but down the >> >> >>> road, I >> >> >>> >>> get >> >> >>> >>> >> an >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm not >> >> >>> supposed >> >> >>> >>> to >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >>> >>> >> > >> >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the >> >> >>> situation >> >> >>> >>> >> >> where a reference is generated that causes a new >> definition >> >> to >> >> >>> be >> >> >>> >>> >> >> added such that the variable is in scope at the use >> site. IE >> >> >>> the >> >> >>> >>> >> >> binding is lifted from the use site to some outer node >> that >> >> >>> contains >> >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to >> >> definitions >> >> >>> that >> >> >>> >>> >> >> are created automatically in this way (which is ultimately >> >> most >> >> >>> >>> >> >> definitions). The exception is caused because somehow the >> >> lift >> >> >>> >>> ended >> >> >>> >>> >> >> up in a place where it wasn't actually in scope. That >> >> shouldn't >> >> >>> >>> >> >> happen. Anyway, if you point me to a version of a fuzzer >> >> that >> >> >>> does >> >> >>> >>> >> >> this, I'll try to debug it. >> >> >>> >>> >> >> >> >> >>> >>> >> >#lang clotho >> >> >>> >>> >> > >> >> >>> >>> >> >(require >> >> >>> >>> >> > xsmith >> >> >>> >>> >> > racr >> >> >>> >>> >> > xsmith/racr-convenience >> >> >>> >>> >> > xsmith/canned-components >> >> >>> >>> >> > racket/string >> >> >>> >>> >> > racket/list >> >> >>> >>> >> > racket/pretty) >> >> >>> >>> >> > >> >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >> >>> expressions. >> >> >>> >>> >> > >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >>> >>> >> > #:VoidExpression #t >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >>> >>> >> > #:VariableReference #t >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >>> >>> >> > #:IfExpression #t >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >>> >>> >> > #:LetSequential #t >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >>> >>> >> > #:Numbers #t >> >> >>> >>> >> > #:Booleans #t >> >> >>> >>> >> > #:Strings #t >> >> >>> >>> >> > #:ImmutableList #t) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-loop-over-container >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > #:name Map >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> (list-type >> >> >>> >>> inner))) >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >> >>> inner)))) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-to-grammar >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > [MyProgram #f >> >> >>> >>> >> > ([decls : Definition *] >> >> >>> >>> >> > [declsTwo : MyDefinition *] >> >> >>> >>> >> > [body : Expression]) >> >> >>> >>> >> > #:prop type-info >> >> >>> >>> >> > [(fresh-type-variable) >> >> >>> >>> >> > (? (n t) >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> >>> >>> >> > 'declsTwo (? (c) (fresh-type-variable)) >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >>> >>> >> > #:prop binder-info () >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? >> (n t) >> >> >>> (hash >> >> >>> >>> 'e >> >> >>> >>> >> t))]]) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-property >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >>> 'Definition))]) >> >> >>> >>> >> > >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete >> but >> >> >>> needs >> >> >>> >>> to be. >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >> >>> unconstrained >> >> >>> >>> >> >;; type variable. >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >>> >>> >> > (list >> >> >>> >>> >> > (?()int-type) >> >> >>> >>> >> > (?()bool-type) >> >> >>> >>> >> > (?()string-type) >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> >>> >>> >> > >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >>> >>> >> > (define out (open-output-string)) >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >>> >>> >> > (get-output-string out)) >> >> >>> >>> >> > >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >>> >>> >> > [somelisp] >> >> >>> >>> >> > #:program-node MyProgram >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >>> >>> >> > (string-join >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >> >>> >>> >> > "\n")) >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >>> >>> >> > >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> >> Probably something is going wrong in the `fresh` property. >> >> >>> Have you >> >> >>> >>> >> >> changed that? >> >> >>> >>> >> >> >> >> >>> >>> >> >I don?t think so. See the code at the end of the email. >> >> >>> >>> >> > >> >> >>> >>> >> >> I should change that error message. Anyway, it seems to >> >> think >> >> >>> there >> >> >>> >>> >> >> is no outer node that has a `Definition *` child that can >> be >> >> the >> >> >>> >>> >> >> target for lifting a definition. Generally this happens >> when >> >> >>> you >> >> >>> >>> >> >> forget to have an outer Program node or something that can >> >> hold >> >> >>> >>> >> >> definitions. >> >> >>> >>> >> >> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. >> >> >>> >>> >> > >> >> >>> >>> >> >#lang clotho >> >> >>> >>> >> > >> >> >>> >>> >> >(require >> >> >>> >>> >> > xsmith >> >> >>> >>> >> > racr >> >> >>> >>> >> > xsmith/racr-convenience >> >> >>> >>> >> > xsmith/canned-components >> >> >>> >>> >> > racket/string >> >> >>> >>> >> > racket/list >> >> >>> >>> >> > racket/pretty) >> >> >>> >>> >> > >> >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >> >>> expressions. >> >> >>> >>> >> > >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >>> >>> >> > #:VoidExpression #t >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >>> >>> >> > #:VariableReference #t >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >>> >>> >> > #:IfExpression #t >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >>> >>> >> > #:LetSequential #t >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >>> >>> >> > #:Numbers #t >> >> >>> >>> >> > #:Booleans #t >> >> >>> >>> >> > #:Strings #t >> >> >>> >>> >> > #:ImmutableList #t) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-loop-over-container >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > #:name Map >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> (list-type >> >> >>> >>> inner))) >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >> >>> inner)))) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-to-grammar >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > [MyProgram #f >> >> >>> >>> >> > ([decls : Definition *] >> >> >>> >>> >> > [body : SubProgram]) >> >> >>> >>> >> > #:prop type-info >> >> >>> >>> >> > [(fresh-type-variable) >> >> >>> >>> >> > (? (n t) >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >>> >>> >> > [SubProgram #f >> >> >>> >>> >> > ([decls : MyDefinition *] >> >> >>> >>> >> > [body : Expression]) >> >> >>> >>> >> > #:prop type-info >> >> >>> >>> >> > [(fresh-type-variable) >> >> >>> >>> >> > (? (n t) >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >>> >>> >> > #:prop binder-info () >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? >> (n t) >> >> >>> (hash >> >> >>> >>> 'e >> >> >>> >>> >> t))]]) >> >> >>> >>> >> > >> >> >>> >>> >> >(add-property >> >> >>> >>> >> > somelisp >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >>> 'Definition))]) >> >> >>> >>> >> > >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete >> but >> >> >>> needs >> >> >>> >>> to be. >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >> >>> unconstrained >> >> >>> >>> >> >;; type variable. >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >>> >>> >> > (list >> >> >>> >>> >> > (?()int-type) >> >> >>> >>> >> > (?()bool-type) >> >> >>> >>> >> > (?()string-type) >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> >>> >>> >> > >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >>> >>> >> > (define out (open-output-string)) >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >>> >>> >> > (get-output-string out)) >> >> >>> >>> >> > >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >>> >>> >> > [somelisp] >> >> >>> >>> >> > #:program-node MyProgram >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >>> >>> >> > (string-join >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) lines) >> >> >>> >>> >> > "\n")) >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >>> >>> >> > >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >>> >>> >> >> >> >>> >>> >> >> >>> >> >> >> >>> >> >> >> >> >> >> From sorawee.pwase at gmail.com Wed Jul 21 19:47:12 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Wed, 21 Jul 2021 18:47:12 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Are you suggesting that this call is not necessary? https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 One instance where concretize-type is used for my fuzzer is when the AST is: (PrintStmt (Array (list (Array (list)) (Array (list))))) - PrintStmt doesn?t constrain the type of the child node. - The inner type of (Array (list)) is unconstrained, and indeed, by itself it can have any type. E.g., seq, seq, seq> - There?s a constraint that the type of the left (Array (list)) must be the same as the type of the right (Array (list)). Using concretize-type + unify!, I can communicate across both empty array printing that they will have the same type like seq, making the outer list have type seq>. If this is not the right way to do things, I would appreciate it if you can point to me the right way. On Wed, Jul 21, 2021 at 6:26 PM William G Hatch wrote: > Thanks! Note that if concretization needs to call up these thunks > during printing it is because you still have unconstrained type > variables at printing time. That's not necessarily a problem, > exactly, but it might mean that thing is not really used. > > On Wed, Jul 21, 2021 at 03:45:38PM -0700, Sorawee Porncharoenwase wrote: > >I submitted a PR that fixes the problem. > >current-xsmith-type-constructor-thunks needs to be parameterized for > >printing. Note that this is not only required by my fuzzer, but also by > >several example fuzzers that use type->string. I.e., standard-ml and cish. > > > >On Wed, Jul 21, 2021 at 3:08 PM William G Hatch > wrote: > > > >> The default-base-type should never show up in a fuzzer that has > >> defined the type property for any node. So this is definitely a bug. > >> I had the default-base-type show up once before, but some other change > >> made me stop seeing it, and I never got to the bottom of it. It's > >> probably because of the #:type-thunks argument that you need for > >> concretizing types of lifted definitions. The default was a list with > >> the default base type. This was always a bad idea. I've just pushed > >> a change that instead errors if the concretization list is used > >> without being parameterized with something new first. > >> > >> At any rate, I think this is an xsmith bug, not a bug in your code. > >> Somewhere the `current-xsmith-type-constructor-thunks` parameter is > >> being called where it hasn't been parameterized. If the new exception > >> that will be raised gives any useful information about where this is > >> happening, let me know. > >> > >> On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase wrote: > >> >Sure, I can do that. > >> > > >> >I have another question about type->string. I have been using > type->string > >> >from standard-ml / cish examples. > >> > > >> >(define (type->string t) > >> > (let loop ([t* t]) > >> > (define t (concretize-type t*)) > >> > (unify! t t*) > >> > (cond > >> > [(can-unify? t (immutable (array-type (fresh-type-variable)))) > >> > (define inner (fresh-type-variable)) > >> > (unify! t (immutable (array-type inner))) > >> > (format "seq<~a>" (loop inner))] > >> > [(can-unify? t (product-type #f)) > >> > (match (product-type-inner-type-list t) > >> > [(list) "()"] > >> > [(list a) (loop a)] > >> > [inners (format "(~a)" (string-join (map loop inners) ", > "))])] > >> > [(can-unify? t (function-type (fresh-type-variable) > >> >(fresh-type-variable))) > >> > (define ret (fresh-type-variable)) > >> > (define arg (fresh-type-variable)) > >> > (unify! t (function-type arg ret)) > >> > (format "(~a -> ~a)" (loop arg) (loop ret))] > >> > [(can-unify? t number-type) "int"] > >> > [(can-unify? t int-type) "int"] > >> > [(can-unify? t bool-type) "bool"] > >> > [(can-unify? t string-type) (if (zero? (random 2)) "string" > >> "seq")] > >> > [else #;(error 'type->string "Type not implemented yet: ~v" t) > >> > "FOOBAR"]))) > >> > > >> >The first weird thing is that the number-type branch is necessary, > despite > >> >the fact that I don?t have any float in the program. > >> > > >> >The second weird thing is that sometimes "FOOBAR" is reached because t > is > >> >#. In particular, it generates the following > >> code: > >> > > >> >safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) > >> ># roughly (vector-set (let ([theArray (vector) : (array (array > >> >FOOBAR))]) theArray) 1 (vector 1)) > >> > > >> >where you can see that ?FOOBAR? here should really be int. > >> > > >> >In my experiment, replacing default-base-type with int seems to work > well, > >> >but it indicates something is obviously wrong. Here?s the call site of > >> >type->string FWIW: > >> > > >> >[ImmutableArrayLiteral > >> > (? (n) > >> > (match (ast-children (ast-child 'expressions n)) > >> > [(list) > >> > ;; sorry for convoluted logic, but I used to use inner-type for > >> >something and haven't refactored this. > >> > (define inner-type (fresh-type-variable)) > >> > (define the-array-type (immutable (array-type inner-type))) > >> > (define the-type (concretize-type ($xsmith_type n))) > >> > (unify! the-type the-array-type) > >> > (text (format "(var theArray : ~a := []; theArray)" > >> >(type->string the-type)))] > >> > [xs (h-append lbracket > >> > (comma-list (pretty-print-children xs)) > >> > rbracket)]))] > >> > > >> >Any ideas? > >> > > >> > > >> >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch > >> wrote: > >> > > >> >> Awesome! We have a git repo where we are keeping track of bugs we > >> >> find with Xsmith. If you want, we could add you to it and you can > add > >> >> any bugs you find to it. Otherwise I will probably put in anything > >> >> you mention like this. > >> >> > >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase > wrote: > >> >> >By the way, here are the results so far: > >> >> > > >> >> >- https://github.com/dafny-lang/dafny/issues/1291 > >> >> >- https://github.com/dafny-lang/dafny/issues/1297 > >> >> > > >> >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < > >> >> >sorawee.pwase at gmail.com> wrote: > >> >> > > >> >> >> We are primarily focused on different compiler backends for Dafny > >> (Dafny > >> >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). > >> >> >> > >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch > > >> >> wrote: > >> >> >> > >> >> >>> >> Do you have a roadmap for implementing ... > >> >> >>> > >> >> >>> Not at the moment. Right now I'm focused on getting some of our > >> >> >>> mostly-finished fuzzers *really* finished and ready for fuzzing, > >> >> >>> running some fuzzing campaigns to find some bugs, and some > features > >> >> >>> that are immediately helpful for running fuzzing (eg. some stuff > to > >> >> >>> support feedback-directed fuzzing and choice weighting). > >> >> >>> > >> >> >>> > >> >> >>> >Also, I'm curious about the setup of various fuzzers that you > are > >> >> >>> running. > >> >> >>> >How many programs are you able to generate and test per minute? > Do > >> you > >> >> >>> have > >> >> >>> >any particular parallelization setup that works well for you? > >> >> >>> > >> >> >>> There is a harness repository that we use that was mostly > written by > >> >> >>> Guy Watson: > >> >> >>> > >> >> >>> https://gitlab.flux.utah.edu/xsmith/harness > >> >> >>> > >> >> >>> There is a `harness.py` script that takes a configuration file to > >> run > >> >> >>> an xsmith fuzzer, running its output through various > implementations > >> >> >>> of whatever language, comparing the output, and binning the > result > >> >> >>> (eg. everything matching, diff, crash, everything crashing (IE > bad > >> >> >>> program), xsmith itself crashing, ...). That harness script is > >> >> >>> basically for running a single job, and there are 2 ways of > doing it > >> >> >>> in parallel. > >> >> >>> > >> >> >>> The first one is the `go-button`, which can run fuzzing on > multiple > >> >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab > >> >> >>> (https://www.emulab.net/) to have things set up just right, and > >> >> >>> requires at least 2 machines. It also generates a bunch of live > >> >> >>> graphs about fuzzing. It uses ansible to run setup scripts to > prep > >> >> >>> each machine before starting the fuzz campaign. > >> >> >>> > >> >> >>> I mostly use `local-multi-harness` instead, which just runs on a > >> >> >>> single machine, but runs multiple harness processes. It doesn't > >> >> >>> automatically run the setup scripts like the `go-button` does, so > >> >> >>> eg. if you want to use the `racket-kernel-fuzzer` configuration > you > >> >> >>> need to run the `racket-install.rkt` script first manually. > >> >> >>> > >> >> >>> As far as speed, xsmith is not particularly fast. There is a > server > >> >> >>> mode that at least saves on startup time, but I think the harness > >> >> >>> might not run it particularly well (Guy correct me if I'm wrong > -- > >> at > >> >> >>> any rate I haven't been using it with server mode, but that > might be > >> >> >>> because I never bothered changing my config after bugs were > fixed). > >> >> >>> Aside from Racket's slow startup speed (particularly bad with > Xsmith > >> >> >>> because it loads a lot of stuff), the runtime varies a lot based > on > >> >> >>> the max-depth configuration, the size of the grammar, etc. > >> >> >>> > >> >> >>> I haven't focused a lot on performance, necessarily. Basically > >> >> >>> whenever I decide Xsmith is too slow for my taste I spend some > time > >> >> >>> trying to optimize it, and generally make a few gains until I > think > >> >> >>> ?that's enough optimization for now?. If you can find any ways > to > >> >> >>> make improvements, that would be great! I might have taken all > of > >> the > >> >> >>> low-hanging fruit in my previous optimization passes, however. > In > >> >> >>> previous optimization passes I got the most out of optimizing in > the > >> >> >>> type system, but I made some major improvements to type checking > >> speed > >> >> >>> last time so it may not dominate anymore. > >> >> >>> > >> >> >>> > >> >> >>> As an aside, a question for you: Are there multiple Dafny > >> >> >>> implementations? Or different optimization levels? Or are you > >> >> >>> running different versions of the compiler against each other? > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase > >> >> wrote: > >> >> >>> >Also, I'm curious about the setup of various fuzzers that you > are > >> >> >>> running. > >> >> >>> >How many programs are you able to generate and test per minute? > Do > >> you > >> >> >>> have > >> >> >>> >any particular parallelization setup that works well for you? > >> >> >>> > > >> >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < > >> >> >>> >sorawee.pwase at gmail.com> wrote: > >> >> >>> > > >> >> >>> >> Do you have a roadmap for implementing "Provide mechanism for > >> >> >>> precluding > >> >> >>> >> types in `fresh-type-variable` invocations"? ( > >> >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I > also > >> >> really > >> >> >>> >> want this feature. > >> >> >>> >> > >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch > >> > >> >> >>> wrote: > >> >> >>> >> > >> >> >>> >>> That would be great, thanks! > >> >> >>> >>> > >> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee > >> Porncharoenwase > >> >> >>> wrote: > >> >> >>> >>> >Yes, this is very helpful. I will let you know if I > encounter > >> any > >> >> >>> further > >> >> >>> >>> >issues after updating XSmith. > >> >> >>> >>> > > >> >> >>> >>> >By the way, do you want any help with Scribble? I notice a > lot > >> of > >> >> >>> typos > >> >> >>> >>> and > >> >> >>> >>> >link errors and can submit PRs to fix them. > >> >> >>> >>> > > >> >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > >> >> >> > > >> >> >>> >>> wrote: > >> >> >>> >>> > > >> >> >>> >>> >> Sorry for the delay. I wanted to do the debugging before > >> >> >>> responding, > >> >> >>> >>> >> but then I didn't get to it for a while. > >> >> >>> >>> >> > >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some > >> bugs. > >> >> >>> >>> >> > >> >> >>> >>> >> First, that issue with the bud-node was a bug in the > `edit` > >> >> >>> property > >> >> >>> >>> >> which the canned-components `ProcedureApplication` uses. > >> It's > >> >> >>> fixed > >> >> >>> >>> >> now. > >> >> >>> >>> >> > >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be > encoded > >> as > >> >> a > >> >> >>> >>> >> >> function application node that only allows a reference > as > >> its > >> >> >>> RHS > >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that > >> said, > >> >> I'm > >> >> >>> not > >> >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >> >>> >>> >> >> > >> >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >> >>> >>> >> > >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would > be > >> >> >>> function > >> >> >>> >>> >> application that only allows a reference in the procedure > >> >> >>> position. I > >> >> >>> >>> >> don't know why I wrote RHS. > >> >> >>> >>> >> > >> >> >>> >>> >> Anyway, the strategy of having separate definition nodes > for > >> >> >>> functions > >> >> >>> >>> >> and everything else is used in Cish. I don't recommend > >> >> following > >> >> >>> >>> >> anything Cish does. However, it accomplishes this by > having > >> the > >> >> >>> >>> >> program node have non-function definitions first, and > >> >> disallowing > >> >> >>> use > >> >> >>> >>> >> of functions on the RHS of those functions. > >> >> >>> >>> >> > >> >> >>> >>> >> The problem where it can't find a reference is a bug > (that I > >> >> >>> haven't > >> >> >>> >>> >> fixed yet) caused by the lifting mechanics not really > being > >> >> >>> completely > >> >> >>> >>> >> in sync with the reference mechanics with respect to nodes > >> that > >> >> >>> have > >> >> >>> >>> >> multiple definition fields. When there are multiple > >> definition > >> >> >>> >>> >> fields, definitions in the lower field can reference the > >> >> >>> definitions > >> >> >>> >>> >> in the higher field, but not vice versa. The lifting > >> machinery > >> >> >>> >>> >> doesn't know about that ordering, it just stuffs the > lifted > >> >> >>> definition > >> >> >>> >>> >> into the front of the definitions in whatever field it is > >> >> >>> targetting. > >> >> >>> >>> >> Thus it places a definition in a place that the reference > >> >> machinery > >> >> >>> >>> >> thinks is out of scope. > >> >> >>> >>> >> > >> >> >>> >>> >> So... if you want to have different binders that get > >> different > >> >> >>> fields > >> >> >>> >>> >> in the program node you can only make it work if you > disallow > >> >> >>> stuff to > >> >> >>> >>> >> be sure you never need to reference something that will be > >> >> lifted > >> >> >>> into > >> >> >>> >>> >> the lower field. > >> >> >>> >>> >> > >> >> >>> >>> >> That said, I've just fixed some issues to allow definition > >> nodes > >> >> >>> to be > >> >> >>> >>> >> subtypable, so now you can have a definition node that is > a > >> >> >>> subtype of > >> >> >>> >>> >> Definition, but has some different rules. Importantly, > >> though, > >> >> it > >> >> >>> can > >> >> >>> >>> >> be lifted to the same field. > >> >> >>> >>> >> > >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your > example > >> to > >> >> >>> look > >> >> >>> >>> >> like this: > >> >> >>> >>> >> > >> >> >>> >>> >> ``` > >> >> >>> >>> >> [MyDefinition Definition () > >> >> >>> >>> >> #:prop fresh (hash 'Expression (make-hole > >> >> >>> >>> >> 'ProcedureApplication))] > >> >> >>> >>> >> ``` > >> >> >>> >>> >> > >> >> >>> >>> >> It can now be lifted among normal definitions, but always > >> have > >> >> >>> >>> >> ProcedureApplication as its RHS. > >> >> >>> >>> >> > >> >> >>> >>> >> However, I'm still not convinced that you actually want > >> >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to > restrict > >> >> lambda > >> >> >>> >>> >> nodes to only be generated on the RHS of a > FunctionDefinition > >> >> node > >> >> >>> >>> >> (that's a subtype of Definition), and make > FunctionDefinition > >> >> >>> always > >> >> >>> >>> >> have a Lambda node as its RHS. > >> >> >>> >>> >> > >> >> >>> >>> >> There is an example in the documentation that is like > this, > >> in > >> >> the > >> >> >>> >>> >> explanation of the `choice-filters-to-apply` property. > But > >> I'll > >> >> >>> paste > >> >> >>> >>> >> it here: > >> >> >>> >>> >> > >> >> >>> >>> >> ``` > >> >> >>> >>> >> (add-choice-method > >> >> >>> >>> >> my-component > >> >> >>> >>> >> no-lambda-except-global-def > >> >> >>> >>> >> [#f (? () #t)] > >> >> >>> >>> >> [Lambda (? () > >> >> >>> >>> >> (and (parent-node current-hole) > >> >> >>> >>> >> (equal? (ast-node-type (parent-node > >> >> current-hole)) > >> >> >>> >>> >> 'Definition) > >> >> >>> >>> >> (parent-node (parent-node current-hole)) > >> >> >>> >>> >> (equal? (ast-node-type (parent-node > >> >> (parent-node > >> >> >>> >>> >> current-hole))) > >> >> >>> >>> >> 'Program)))]) > >> >> >>> >>> >> (add-property my-component > >> >> >>> >>> >> choice-filters-to-apply > >> >> >>> >>> >> [#f (no-lambda-except-global-def)]) > >> >> >>> >>> >> ``` > >> >> >>> >>> >> > >> >> >>> >>> >> You can use something like that to allow Lambda nodes to > >> only be > >> >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and > define > >> >> >>> >>> >> FunctionDefinition like MyDefinition above to always > generate > >> >> >>> Lambda > >> >> >>> >>> >> as its RHS, and I think it should work for a first-order > >> >> language. > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type > >> >> variables > >> >> >>> >>> >> >with some concrete non-function type variables, but down > the > >> >> >>> road, I > >> >> >>> >>> get > >> >> >>> >>> >> an > >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm > not > >> >> >>> supposed > >> >> >>> >>> to > >> >> >>> >>> >> >unify! inside add-choice-method? > >> >> >>> >>> >> > >> >> >>> >>> >> Well, probably not. Generally the only place you want to > >> >> `unify!` > >> >> >>> is > >> >> >>> >>> >> inside the type property in the function that determines > >> child > >> >> >>> types > >> >> >>> >>> >> based on the parent type. Inside that function you can > >> >> `unify!`, > >> >> >>> but > >> >> >>> >>> >> the unification needs to be consistent (IE not unified > based > >> on > >> >> the > >> >> >>> >>> >> result of `random`) OR the choice needs to be saved in a > way > >> >> that > >> >> >>> it > >> >> >>> >>> >> can be looked up when that function is run again. > >> >> >>> >>> >> > >> >> >>> >>> >> The problem is that the choice method gets its type > >> information > >> >> >>> based > >> >> >>> >>> >> on a hole node that is later replaced, so the type is > >> recomputed > >> >> >>> >>> >> without necessarily consulting the type that the hole > said it > >> >> was. > >> >> >>> >>> >> You should be able to constrain the type using > unification if > >> >> you > >> >> >>> save > >> >> >>> >>> >> that type to re-unify it consistently later. But without > >> seeing > >> >> >>> the > >> >> >>> >>> >> exact error and code that generated it, I'm not 100% > certain. > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type > >> >> could be > >> >> >>> >>> used > >> >> >>> >>> >> > nondeterministically in this manner, and my experience > >> with it > >> >> >>> so far > >> >> >>> >>> >> seems > >> >> >>> >>> >> > to indicate no. > >> >> >>> >>> >> > >> >> >>> >>> >> I think this should work, barring the issues mentioned > above > >> >> about > >> >> >>> the > >> >> >>> >>> >> lifting vs referencing bug when you have the different > >> >> definition > >> >> >>> >>> >> types in different fields. Adjusting your example to have > >> >> >>> >>> >> MyDefinition as a subtype, and then only having a > Definition > >> >> field > >> >> >>> in > >> >> >>> >>> >> MyProgram, random choice seems to work fine. But > sometimes > >> it > >> >> >>> >>> >> generates gargantuan programs (that I kill before it > >> finishes) > >> >> >>> because > >> >> >>> >>> >> it keeps choosing MyDefinition, requiring a function > >> >> application, > >> >> >>> >>> >> which likely lifts a reference to a function, which might > >> choose > >> >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates > bigger > >> and > >> >> >>> >>> >> bigger types. > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> Anyway, does that all help? > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee > >> >> Porncharoenwase > >> >> >>> >>> wrote: > >> >> >>> >>> >> >> > >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be > encoded > >> as > >> >> a > >> >> >>> >>> >> >> function application node that only allows a reference > as > >> its > >> >> >>> RHS > >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that > >> said, > >> >> I'm > >> >> >>> not > >> >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >> >>> >>> >> >> > >> >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >> >>> >>> >> > > >> >> >>> >>> >> > 1. I have no problem with the definition site. That?s > >> easy > >> >> to > >> >> >>> deal > >> >> >>> >>> >> with. > >> >> >>> >>> >> > 2. I also have no problem with restricting function > >> >> >>> application. > >> >> >>> >>> That?s > >> >> >>> >>> >> > easy to deal with as well. > >> >> >>> >>> >> > 3. The problem is primarily with a bare variable > >> reference > >> >> >>> bound > >> >> >>> >>> to a > >> >> >>> >>> >> > closure value, which must only occur in function > >> >> application. > >> >> >>> For > >> >> >>> >>> >> example, > >> >> >>> >>> >> > the following code is invalid. > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define (f) (void)) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define g f) ;; this is invalid > >> >> >>> >>> >> >(define h (g)) > >> >> >>> >>> >> >(define a (list f)) ;; this is also invalid > >> >> >>> >>> >> >(define b (f)) ;; this is ok > >> >> >>> >>> >> > > >> >> >>> >>> >> >I have been using choice-filters-to-apply to restrict > >> variable > >> >> >>> >>> references > >> >> >>> >>> >> >whose type can-unify? with function-type that doesn't > >> directly > >> >> >>> appear > >> >> >>> >>> >> under > >> >> >>> >>> >> >function application. But this is obviously not ideal > >> because > >> >> the > >> >> >>> >>> filtered > >> >> >>> >>> >> >nodes are mostly those whose type is totally > unrestricted. > >> So > >> >> >>> >>> filtering > >> >> >>> >>> >> >them would prevent them from being concretized to other > >> >> >>> non-function > >> >> >>> >>> types > >> >> >>> >>> >> >as well. (Also note that I tried unify! these > unrestricted > >> type > >> >> >>> >>> variables > >> >> >>> >>> >> >with some concrete non-function type variables, but down > the > >> >> >>> road, I > >> >> >>> >>> get > >> >> >>> >>> >> an > >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm > not > >> >> >>> supposed > >> >> >>> >>> to > >> >> >>> >>> >> >unify! inside add-choice-method? > >> >> >>> >>> >> > > >> >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the > >> >> >>> situation > >> >> >>> >>> >> >> where a reference is generated that causes a new > >> definition > >> >> to > >> >> >>> be > >> >> >>> >>> >> >> added such that the variable is in scope at the use > >> site. IE > >> >> >>> the > >> >> >>> >>> >> >> binding is lifted from the use site to some outer node > >> that > >> >> >>> contains > >> >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to > >> >> definitions > >> >> >>> that > >> >> >>> >>> >> >> are created automatically in this way (which is > ultimately > >> >> most > >> >> >>> >>> >> >> definitions). The exception is caused because somehow > the > >> >> lift > >> >> >>> >>> ended > >> >> >>> >>> >> >> up in a place where it wasn't actually in scope. That > >> >> shouldn't > >> >> >>> >>> >> >> happen. Anyway, if you point me to a version of a > fuzzer > >> >> that > >> >> >>> does > >> >> >>> >>> >> >> this, I'll try to debug it. > >> >> >>> >>> >> >> > >> >> >>> >>> >> >#lang clotho > >> >> >>> >>> >> > > >> >> >>> >>> >> >(require > >> >> >>> >>> >> > xsmith > >> >> >>> >>> >> > racr > >> >> >>> >>> >> > xsmith/racr-convenience > >> >> >>> >>> >> > xsmith/canned-components > >> >> >>> >>> >> > racket/string > >> >> >>> >>> >> > racket/list > >> >> >>> >>> >> > racket/pretty) > >> >> >>> >>> >> > > >> >> >>> >>> >> >;; We first define a basic component and add a bunch of > >> >> >>> expressions. > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-basic-expressions somelisp > >> >> >>> >>> >> > #:ProgramWithSequence #t > >> >> >>> >>> >> > #:VoidExpression #t > >> >> >>> >>> >> > #:AssignmentExpression #t > >> >> >>> >>> >> > #:VariableReference #t > >> >> >>> >>> >> > #:ProcedureApplication #t > >> >> >>> >>> >> > #:IfExpression #t > >> >> >>> >>> >> > #:ExpressionSequence #t > >> >> >>> >>> >> > #:LetSequential #t > >> >> >>> >>> >> > #:LambdaWithExpression #t > >> >> >>> >>> >> > #:Numbers #t > >> >> >>> >>> >> > #:Booleans #t > >> >> >>> >>> >> > #:Strings #t > >> >> >>> >>> >> > #:ImmutableList #t) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-loop-over-container > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > #:name Map > >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > >> (list-type > >> >> >>> >>> inner))) > >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >> >> >>> inner)))) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-to-grammar > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > [MyProgram #f > >> >> >>> >>> >> > ([decls : Definition *] > >> >> >>> >>> >> > [declsTwo : MyDefinition *] > >> >> >>> >>> >> > [body : Expression]) > >> >> >>> >>> >> > #:prop type-info > >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >>> >>> >> > (? (n t) > >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >> >> >>> >>> >> > 'declsTwo (? (c) > (fresh-type-variable)) > >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >>> >>> >> > [MyDefinition #f ([type] > >> >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >> >>> >>> >> > [e : ProcedureApplication]) > >> >> >>> >>> >> > #:prop binder-info () > >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? > >> (n t) > >> >> >>> (hash > >> >> >>> >>> 'e > >> >> >>> >>> >> t))]]) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-property > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > lift-type->ast-binder-type > >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >> >>> 'Definition))]) > >> >> >>> >>> >> > > >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete > >> but > >> >> >>> needs > >> >> >>> >>> to be. > >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an > >> >> >>> unconstrained > >> >> >>> >>> >> >;; type variable. > >> >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >> >>> >>> >> > (list > >> >> >>> >>> >> > (?()int-type) > >> >> >>> >>> >> > (?()bool-type) > >> >> >>> >>> >> > (?()string-type) > >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >> >>> >>> >> > (define out (open-output-string)) > >> >> >>> >>> >> > (for ([symex s-exps]) > >> >> >>> >>> >> > (pretty-print symex out 1)) > >> >> >>> >>> >> > (get-output-string out)) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define-xsmith-interface-functions > >> >> >>> >>> >> > [somelisp] > >> >> >>> >>> >> > #:program-node MyProgram > >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >> >>> >>> >> > #:comment-wrap (? (lines) > >> >> >>> >>> >> > (string-join > >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) > lines) > >> >> >>> >>> >> > "\n")) > >> >> >>> >>> >> > #:format-render somelisp-format-render) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >> >>> >>> >> > > >> >> >>> >>> >> > > >> >> >>> >>> >> >> Probably something is going wrong in the `fresh` > property. > >> >> >>> Have you > >> >> >>> >>> >> >> changed that? > >> >> >>> >>> >> >> > >> >> >>> >>> >> >I don?t think so. See the code at the end of the email. > >> >> >>> >>> >> > > >> >> >>> >>> >> >> I should change that error message. Anyway, it seems > to > >> >> think > >> >> >>> there > >> >> >>> >>> >> >> is no outer node that has a `Definition *` child that > can > >> be > >> >> the > >> >> >>> >>> >> >> target for lifting a definition. Generally this > happens > >> when > >> >> >>> you > >> >> >>> >>> >> >> forget to have an outer Program node or something that > can > >> >> hold > >> >> >>> >>> >> >> definitions. > >> >> >>> >>> >> >> > >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. > >> >> >>> >>> >> > > >> >> >>> >>> >> >#lang clotho > >> >> >>> >>> >> > > >> >> >>> >>> >> >(require > >> >> >>> >>> >> > xsmith > >> >> >>> >>> >> > racr > >> >> >>> >>> >> > xsmith/racr-convenience > >> >> >>> >>> >> > xsmith/canned-components > >> >> >>> >>> >> > racket/string > >> >> >>> >>> >> > racket/list > >> >> >>> >>> >> > racket/pretty) > >> >> >>> >>> >> > > >> >> >>> >>> >> >;; We first define a basic component and add a bunch of > >> >> >>> expressions. > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-basic-expressions somelisp > >> >> >>> >>> >> > #:ProgramWithSequence #t > >> >> >>> >>> >> > #:VoidExpression #t > >> >> >>> >>> >> > #:AssignmentExpression #t > >> >> >>> >>> >> > #:VariableReference #t > >> >> >>> >>> >> > #:ProcedureApplication #t > >> >> >>> >>> >> > #:IfExpression #t > >> >> >>> >>> >> > #:ExpressionSequence #t > >> >> >>> >>> >> > #:LetSequential #t > >> >> >>> >>> >> > #:LambdaWithExpression #t > >> >> >>> >>> >> > #:Numbers #t > >> >> >>> >>> >> > #:Booleans #t > >> >> >>> >>> >> > #:Strings #t > >> >> >>> >>> >> > #:ImmutableList #t) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-loop-over-container > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > #:name Map > >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > >> (list-type > >> >> >>> >>> inner))) > >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type > >> >> >>> inner)))) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-to-grammar > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > [MyProgram #f > >> >> >>> >>> >> > ([decls : Definition *] > >> >> >>> >>> >> > [body : SubProgram]) > >> >> >>> >>> >> > #:prop type-info > >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >>> >>> >> > (? (n t) > >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) > >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >>> >>> >> > [SubProgram #f > >> >> >>> >>> >> > ([decls : MyDefinition *] > >> >> >>> >>> >> > [body : Expression]) > >> >> >>> >>> >> > #:prop type-info > >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >>> >>> >> > (? (n t) > >> >> >>> >>> >> > (hash 'decls (? (c) > (fresh-type-variable)) > >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >>> >>> >> > [MyDefinition #f ([type] > >> >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >> >>> >>> >> > [e : ProcedureApplication]) > >> >> >>> >>> >> > #:prop binder-info () > >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? > >> (n t) > >> >> >>> (hash > >> >> >>> >>> 'e > >> >> >>> >>> >> t))]]) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(add-property > >> >> >>> >>> >> > somelisp > >> >> >>> >>> >> > lift-type->ast-binder-type > >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >> >>> 'Definition))]) > >> >> >>> >>> >> > > >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete > >> but > >> >> >>> needs > >> >> >>> >>> to be. > >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an > >> >> >>> unconstrained > >> >> >>> >>> >> >;; type variable. > >> >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >> >>> >>> >> > (list > >> >> >>> >>> >> > (?()int-type) > >> >> >>> >>> >> > (?()bool-type) > >> >> >>> >>> >> > (?()string-type) > >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >> >>> >>> >> > (define out (open-output-string)) > >> >> >>> >>> >> > (for ([symex s-exps]) > >> >> >>> >>> >> > (pretty-print symex out 1)) > >> >> >>> >>> >> > (get-output-string out)) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(define-xsmith-interface-functions > >> >> >>> >>> >> > [somelisp] > >> >> >>> >>> >> > #:program-node MyProgram > >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >> >>> >>> >> > #:comment-wrap (? (lines) > >> >> >>> >>> >> > (string-join > >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) > lines) > >> >> >>> >>> >> > "\n")) > >> >> >>> >>> >> > #:format-render somelisp-format-render) > >> >> >>> >>> >> > > >> >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >> >>> >>> >> > >> >> >>> >>> > >> >> >>> >> > >> >> >>> > >> >> >> > >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Thu Jul 22 10:33:14 2021 From: william at hatch.uno (William G Hatch) Date: Thu, 22 Jul 2021 10:33:14 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: >If this is not the right way to do things, I would appreciate it if you can >point to me the right way. No, it is the right way. >Are you suggesting that this call is not necessary? > >https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 It is necessary for two things: (1) In case of unconstrained types like you found. Your list example is a good point. I think the SML fuzzer was actually where I was seeing this bug myself, probably for similar reasons. But I haven't been looking at the Xsmith crashes with the SML fuzzer lately, only the wrong code generation issues. (2) Even if the types were fully constrained everywhere, they still might have type-variable wrappers that make it so you can't directly use the accessor functions. Of course, I should write functions like `function-type-argument!` that unify and unwrap themselves. But as I haven't, the result of `concretize-type` has no type variable wrappers. On Wed, Jul 21, 2021 at 06:47:12PM -0700, Sorawee Porncharoenwase wrote: >Are you suggesting that this call is not necessary? > >https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 > >One instance where concretize-type is used for my fuzzer is when the AST is: > >(PrintStmt (Array (list (Array (list)) (Array (list))))) > > > - PrintStmt doesn?t constrain the type of the child node. > - The inner type of (Array (list)) is unconstrained, and indeed, by > itself it can have any type. E.g., seq, seq, seq> > - There?s a constraint that the type of the left (Array (list)) must be > the same as the type of the right (Array (list)). > >Using concretize-type + unify!, I can communicate across both empty array >printing that they will have the same type like seq, making the outer >list have type seq>. > >If this is not the right way to do things, I would appreciate it if you can >point to me the right way. > >On Wed, Jul 21, 2021 at 6:26 PM William G Hatch wrote: > >> Thanks! Note that if concretization needs to call up these thunks >> during printing it is because you still have unconstrained type >> variables at printing time. That's not necessarily a problem, >> exactly, but it might mean that thing is not really used. >> >> On Wed, Jul 21, 2021 at 03:45:38PM -0700, Sorawee Porncharoenwase wrote: >> >I submitted a PR that fixes the problem. >> >current-xsmith-type-constructor-thunks needs to be parameterized for >> >printing. Note that this is not only required by my fuzzer, but also by >> >several example fuzzers that use type->string. I.e., standard-ml and cish. >> > >> >On Wed, Jul 21, 2021 at 3:08 PM William G Hatch >> wrote: >> > >> >> The default-base-type should never show up in a fuzzer that has >> >> defined the type property for any node. So this is definitely a bug. >> >> I had the default-base-type show up once before, but some other change >> >> made me stop seeing it, and I never got to the bottom of it. It's >> >> probably because of the #:type-thunks argument that you need for >> >> concretizing types of lifted definitions. The default was a list with >> >> the default base type. This was always a bad idea. I've just pushed >> >> a change that instead errors if the concretization list is used >> >> without being parameterized with something new first. >> >> >> >> At any rate, I think this is an xsmith bug, not a bug in your code. >> >> Somewhere the `current-xsmith-type-constructor-thunks` parameter is >> >> being called where it hasn't been parameterized. If the new exception >> >> that will be raised gives any useful information about where this is >> >> happening, let me know. >> >> >> >> On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase wrote: >> >> >Sure, I can do that. >> >> > >> >> >I have another question about type->string. I have been using >> type->string >> >> >from standard-ml / cish examples. >> >> > >> >> >(define (type->string t) >> >> > (let loop ([t* t]) >> >> > (define t (concretize-type t*)) >> >> > (unify! t t*) >> >> > (cond >> >> > [(can-unify? t (immutable (array-type (fresh-type-variable)))) >> >> > (define inner (fresh-type-variable)) >> >> > (unify! t (immutable (array-type inner))) >> >> > (format "seq<~a>" (loop inner))] >> >> > [(can-unify? t (product-type #f)) >> >> > (match (product-type-inner-type-list t) >> >> > [(list) "()"] >> >> > [(list a) (loop a)] >> >> > [inners (format "(~a)" (string-join (map loop inners) ", >> "))])] >> >> > [(can-unify? t (function-type (fresh-type-variable) >> >> >(fresh-type-variable))) >> >> > (define ret (fresh-type-variable)) >> >> > (define arg (fresh-type-variable)) >> >> > (unify! t (function-type arg ret)) >> >> > (format "(~a -> ~a)" (loop arg) (loop ret))] >> >> > [(can-unify? t number-type) "int"] >> >> > [(can-unify? t int-type) "int"] >> >> > [(can-unify? t bool-type) "bool"] >> >> > [(can-unify? t string-type) (if (zero? (random 2)) "string" >> >> "seq")] >> >> > [else #;(error 'type->string "Type not implemented yet: ~v" t) >> >> > "FOOBAR"]))) >> >> > >> >> >The first weird thing is that the number-type branch is necessary, >> despite >> >> >the fact that I don?t have any float in the program. >> >> > >> >> >The second weird thing is that sometimes "FOOBAR" is reached because t >> is >> >> >#. In particular, it generates the following >> >> code: >> >> > >> >> >safeSeqSet((var theArray : seq> := []; theArray), 1, [1]) >> >> ># roughly (vector-set (let ([theArray (vector) : (array (array >> >> >FOOBAR))]) theArray) 1 (vector 1)) >> >> > >> >> >where you can see that ?FOOBAR? here should really be int. >> >> > >> >> >In my experiment, replacing default-base-type with int seems to work >> well, >> >> >but it indicates something is obviously wrong. Here?s the call site of >> >> >type->string FWIW: >> >> > >> >> >[ImmutableArrayLiteral >> >> > (? (n) >> >> > (match (ast-children (ast-child 'expressions n)) >> >> > [(list) >> >> > ;; sorry for convoluted logic, but I used to use inner-type for >> >> >something and haven't refactored this. >> >> > (define inner-type (fresh-type-variable)) >> >> > (define the-array-type (immutable (array-type inner-type))) >> >> > (define the-type (concretize-type ($xsmith_type n))) >> >> > (unify! the-type the-array-type) >> >> > (text (format "(var theArray : ~a := []; theArray)" >> >> >(type->string the-type)))] >> >> > [xs (h-append lbracket >> >> > (comma-list (pretty-print-children xs)) >> >> > rbracket)]))] >> >> > >> >> >Any ideas? >> >> > >> >> > >> >> >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch >> >> wrote: >> >> > >> >> >> Awesome! We have a git repo where we are keeping track of bugs we >> >> >> find with Xsmith. If you want, we could add you to it and you can >> add >> >> >> any bugs you find to it. Otherwise I will probably put in anything >> >> >> you mention like this. >> >> >> >> >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase >> wrote: >> >> >> >By the way, here are the results so far: >> >> >> > >> >> >> >- https://github.com/dafny-lang/dafny/issues/1291 >> >> >> >- https://github.com/dafny-lang/dafny/issues/1297 >> >> >> > >> >> >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < >> >> >> >sorawee.pwase at gmail.com> wrote: >> >> >> > >> >> >> >> We are primarily focused on different compiler backends for Dafny >> >> (Dafny >> >> >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps more). >> >> >> >> >> >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch > > >> >> >> wrote: >> >> >> >> >> >> >> >>> >> Do you have a roadmap for implementing ... >> >> >> >>> >> >> >> >>> Not at the moment. Right now I'm focused on getting some of our >> >> >> >>> mostly-finished fuzzers *really* finished and ready for fuzzing, >> >> >> >>> running some fuzzing campaigns to find some bugs, and some >> features >> >> >> >>> that are immediately helpful for running fuzzing (eg. some stuff >> to >> >> >> >>> support feedback-directed fuzzing and choice weighting). >> >> >> >>> >> >> >> >>> >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you >> are >> >> >> >>> running. >> >> >> >>> >How many programs are you able to generate and test per minute? >> Do >> >> you >> >> >> >>> have >> >> >> >>> >any particular parallelization setup that works well for you? >> >> >> >>> >> >> >> >>> There is a harness repository that we use that was mostly >> written by >> >> >> >>> Guy Watson: >> >> >> >>> >> >> >> >>> https://gitlab.flux.utah.edu/xsmith/harness >> >> >> >>> >> >> >> >>> There is a `harness.py` script that takes a configuration file to >> >> run >> >> >> >>> an xsmith fuzzer, running its output through various >> implementations >> >> >> >>> of whatever language, comparing the output, and binning the >> result >> >> >> >>> (eg. everything matching, diff, crash, everything crashing (IE >> bad >> >> >> >>> program), xsmith itself crashing, ...). That harness script is >> >> >> >>> basically for running a single job, and there are 2 ways of >> doing it >> >> >> >>> in parallel. >> >> >> >>> >> >> >> >>> The first one is the `go-button`, which can run fuzzing on >> multiple >> >> >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab >> >> >> >>> (https://www.emulab.net/) to have things set up just right, and >> >> >> >>> requires at least 2 machines. It also generates a bunch of live >> >> >> >>> graphs about fuzzing. It uses ansible to run setup scripts to >> prep >> >> >> >>> each machine before starting the fuzz campaign. >> >> >> >>> >> >> >> >>> I mostly use `local-multi-harness` instead, which just runs on a >> >> >> >>> single machine, but runs multiple harness processes. It doesn't >> >> >> >>> automatically run the setup scripts like the `go-button` does, so >> >> >> >>> eg. if you want to use the `racket-kernel-fuzzer` configuration >> you >> >> >> >>> need to run the `racket-install.rkt` script first manually. >> >> >> >>> >> >> >> >>> As far as speed, xsmith is not particularly fast. There is a >> server >> >> >> >>> mode that at least saves on startup time, but I think the harness >> >> >> >>> might not run it particularly well (Guy correct me if I'm wrong >> -- >> >> at >> >> >> >>> any rate I haven't been using it with server mode, but that >> might be >> >> >> >>> because I never bothered changing my config after bugs were >> fixed). >> >> >> >>> Aside from Racket's slow startup speed (particularly bad with >> Xsmith >> >> >> >>> because it loads a lot of stuff), the runtime varies a lot based >> on >> >> >> >>> the max-depth configuration, the size of the grammar, etc. >> >> >> >>> >> >> >> >>> I haven't focused a lot on performance, necessarily. Basically >> >> >> >>> whenever I decide Xsmith is too slow for my taste I spend some >> time >> >> >> >>> trying to optimize it, and generally make a few gains until I >> think >> >> >> >>> ?that's enough optimization for now?. If you can find any ways >> to >> >> >> >>> make improvements, that would be great! I might have taken all >> of >> >> the >> >> >> >>> low-hanging fruit in my previous optimization passes, however. >> In >> >> >> >>> previous optimization passes I got the most out of optimizing in >> the >> >> >> >>> type system, but I made some major improvements to type checking >> >> speed >> >> >> >>> last time so it may not dominate anymore. >> >> >> >>> >> >> >> >>> >> >> >> >>> As an aside, a question for you: Are there multiple Dafny >> >> >> >>> implementations? Or different optimization levels? Or are you >> >> >> >>> running different versions of the compiler against each other? >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee Porncharoenwase >> >> >> wrote: >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you >> are >> >> >> >>> running. >> >> >> >>> >How many programs are you able to generate and test per minute? >> Do >> >> you >> >> >> >>> have >> >> >> >>> >any particular parallelization setup that works well for you? >> >> >> >>> > >> >> >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >> >> >> >>> >sorawee.pwase at gmail.com> wrote: >> >> >> >>> > >> >> >> >>> >> Do you have a roadmap for implementing "Provide mechanism for >> >> >> >>> precluding >> >> >> >>> >> types in `fresh-type-variable` invocations"? ( >> >> >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I >> also >> >> >> really >> >> >> >>> >> want this feature. >> >> >> >>> >> >> >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >> >> >> >> >> >>> wrote: >> >> >> >>> >> >> >> >> >>> >>> That would be great, thanks! >> >> >> >>> >>> >> >> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee >> >> Porncharoenwase >> >> >> >>> wrote: >> >> >> >>> >>> >Yes, this is very helpful. I will let you know if I >> encounter >> >> any >> >> >> >>> further >> >> >> >>> >>> >issues after updating XSmith. >> >> >> >>> >>> > >> >> >> >>> >>> >By the way, do you want any help with Scribble? I notice a >> lot >> >> of >> >> >> >>> typos >> >> >> >>> >>> and >> >> >> >>> >>> >link errors and can submit PRs to fix them. >> >> >> >>> >>> > >> >> >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >> >> > >> >> > >> >> >> >>> >>> wrote: >> >> >> >>> >>> > >> >> >> >>> >>> >> Sorry for the delay. I wanted to do the debugging before >> >> >> >>> responding, >> >> >> >>> >>> >> but then I didn't get to it for a while. >> >> >> >>> >>> >> >> >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed some >> >> bugs. >> >> >> >>> >>> >> >> >> >> >>> >>> >> First, that issue with the bud-node was a bug in the >> `edit` >> >> >> >>> property >> >> >> >>> >>> >> which the canned-components `ProcedureApplication` uses. >> >> It's >> >> >> >>> fixed >> >> >> >>> >>> >> now. >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be >> encoded >> >> as >> >> >> a >> >> >> >>> >>> >> >> function application node that only allows a reference >> as >> >> its >> >> >> >>> RHS >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that >> >> said, >> >> >> I'm >> >> >> >>> not >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >> >>> >>> >> >> >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it would >> be >> >> >> >>> function >> >> >> >>> >>> >> application that only allows a reference in the procedure >> >> >> >>> position. I >> >> >> >>> >>> >> don't know why I wrote RHS. >> >> >> >>> >>> >> >> >> >> >>> >>> >> Anyway, the strategy of having separate definition nodes >> for >> >> >> >>> functions >> >> >> >>> >>> >> and everything else is used in Cish. I don't recommend >> >> >> following >> >> >> >>> >>> >> anything Cish does. However, it accomplishes this by >> having >> >> the >> >> >> >>> >>> >> program node have non-function definitions first, and >> >> >> disallowing >> >> >> >>> use >> >> >> >>> >>> >> of functions on the RHS of those functions. >> >> >> >>> >>> >> >> >> >> >>> >>> >> The problem where it can't find a reference is a bug >> (that I >> >> >> >>> haven't >> >> >> >>> >>> >> fixed yet) caused by the lifting mechanics not really >> being >> >> >> >>> completely >> >> >> >>> >>> >> in sync with the reference mechanics with respect to nodes >> >> that >> >> >> >>> have >> >> >> >>> >>> >> multiple definition fields. When there are multiple >> >> definition >> >> >> >>> >>> >> fields, definitions in the lower field can reference the >> >> >> >>> definitions >> >> >> >>> >>> >> in the higher field, but not vice versa. The lifting >> >> machinery >> >> >> >>> >>> >> doesn't know about that ordering, it just stuffs the >> lifted >> >> >> >>> definition >> >> >> >>> >>> >> into the front of the definitions in whatever field it is >> >> >> >>> targetting. >> >> >> >>> >>> >> Thus it places a definition in a place that the reference >> >> >> machinery >> >> >> >>> >>> >> thinks is out of scope. >> >> >> >>> >>> >> >> >> >> >>> >>> >> So... if you want to have different binders that get >> >> different >> >> >> >>> fields >> >> >> >>> >>> >> in the program node you can only make it work if you >> disallow >> >> >> >>> stuff to >> >> >> >>> >>> >> be sure you never need to reference something that will be >> >> >> lifted >> >> >> >>> into >> >> >> >>> >>> >> the lower field. >> >> >> >>> >>> >> >> >> >> >>> >>> >> That said, I've just fixed some issues to allow definition >> >> nodes >> >> >> >>> to be >> >> >> >>> >>> >> subtypable, so now you can have a definition node that is >> a >> >> >> >>> subtype of >> >> >> >>> >>> >> Definition, but has some different rules. Importantly, >> >> though, >> >> >> it >> >> >> >>> can >> >> >> >>> >>> >> be lifted to the same field. >> >> >> >>> >>> >> >> >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your >> example >> >> to >> >> >> >>> look >> >> >> >>> >>> >> like this: >> >> >> >>> >>> >> >> >> >> >>> >>> >> ``` >> >> >> >>> >>> >> [MyDefinition Definition () >> >> >> >>> >>> >> #:prop fresh (hash 'Expression (make-hole >> >> >> >>> >>> >> 'ProcedureApplication))] >> >> >> >>> >>> >> ``` >> >> >> >>> >>> >> >> >> >> >>> >>> >> It can now be lifted among normal definitions, but always >> >> have >> >> >> >>> >>> >> ProcedureApplication as its RHS. >> >> >> >>> >>> >> >> >> >> >>> >>> >> However, I'm still not convinced that you actually want >> >> >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to >> restrict >> >> >> lambda >> >> >> >>> >>> >> nodes to only be generated on the RHS of a >> FunctionDefinition >> >> >> node >> >> >> >>> >>> >> (that's a subtype of Definition), and make >> FunctionDefinition >> >> >> >>> always >> >> >> >>> >>> >> have a Lambda node as its RHS. >> >> >> >>> >>> >> >> >> >> >>> >>> >> There is an example in the documentation that is like >> this, >> >> in >> >> >> the >> >> >> >>> >>> >> explanation of the `choice-filters-to-apply` property. >> But >> >> I'll >> >> >> >>> paste >> >> >> >>> >>> >> it here: >> >> >> >>> >>> >> >> >> >> >>> >>> >> ``` >> >> >> >>> >>> >> (add-choice-method >> >> >> >>> >>> >> my-component >> >> >> >>> >>> >> no-lambda-except-global-def >> >> >> >>> >>> >> [#f (? () #t)] >> >> >> >>> >>> >> [Lambda (? () >> >> >> >>> >>> >> (and (parent-node current-hole) >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> >> current-hole)) >> >> >> >>> >>> >> 'Definition) >> >> >> >>> >>> >> (parent-node (parent-node current-hole)) >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> >> (parent-node >> >> >> >>> >>> >> current-hole))) >> >> >> >>> >>> >> 'Program)))]) >> >> >> >>> >>> >> (add-property my-component >> >> >> >>> >>> >> choice-filters-to-apply >> >> >> >>> >>> >> [#f (no-lambda-except-global-def)]) >> >> >> >>> >>> >> ``` >> >> >> >>> >>> >> >> >> >> >>> >>> >> You can use something like that to allow Lambda nodes to >> >> only be >> >> >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and >> define >> >> >> >>> >>> >> FunctionDefinition like MyDefinition above to always >> generate >> >> >> >>> Lambda >> >> >> >>> >>> >> as its RHS, and I think it should work for a first-order >> >> >> language. >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type >> >> >> variables >> >> >> >>> >>> >> >with some concrete non-function type variables, but down >> the >> >> >> >>> road, I >> >> >> >>> >>> get >> >> >> >>> >>> >> an >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm >> not >> >> >> >>> supposed >> >> >> >>> >>> to >> >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >> >>> >>> >> >> >> >> >>> >>> >> Well, probably not. Generally the only place you want to >> >> >> `unify!` >> >> >> >>> is >> >> >> >>> >>> >> inside the type property in the function that determines >> >> child >> >> >> >>> types >> >> >> >>> >>> >> based on the parent type. Inside that function you can >> >> >> `unify!`, >> >> >> >>> but >> >> >> >>> >>> >> the unification needs to be consistent (IE not unified >> based >> >> on >> >> >> the >> >> >> >>> >>> >> result of `random`) OR the choice needs to be saved in a >> way >> >> >> that >> >> >> >>> it >> >> >> >>> >>> >> can be looked up when that function is run again. >> >> >> >>> >>> >> >> >> >> >>> >>> >> The problem is that the choice method gets its type >> >> information >> >> >> >>> based >> >> >> >>> >>> >> on a hole node that is later replaced, so the type is >> >> recomputed >> >> >> >>> >>> >> without necessarily consulting the type that the hole >> said it >> >> >> was. >> >> >> >>> >>> >> You should be able to constrain the type using >> unification if >> >> >> you >> >> >> >>> save >> >> >> >>> >>> >> that type to re-unify it consistently later. But without >> >> seeing >> >> >> >>> the >> >> >> >>> >>> >> exact error and code that generated it, I'm not 100% >> certain. >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> > In the last email, I asked if lift-type->ast-binder-type >> >> >> could be >> >> >> >>> >>> used >> >> >> >>> >>> >> > nondeterministically in this manner, and my experience >> >> with it >> >> >> >>> so far >> >> >> >>> >>> >> seems >> >> >> >>> >>> >> > to indicate no. >> >> >> >>> >>> >> >> >> >> >>> >>> >> I think this should work, barring the issues mentioned >> above >> >> >> about >> >> >> >>> the >> >> >> >>> >>> >> lifting vs referencing bug when you have the different >> >> >> definition >> >> >> >>> >>> >> types in different fields. Adjusting your example to have >> >> >> >>> >>> >> MyDefinition as a subtype, and then only having a >> Definition >> >> >> field >> >> >> >>> in >> >> >> >>> >>> >> MyProgram, random choice seems to work fine. But >> sometimes >> >> it >> >> >> >>> >>> >> generates gargantuan programs (that I kill before it >> >> finishes) >> >> >> >>> because >> >> >> >>> >>> >> it keeps choosing MyDefinition, requiring a function >> >> >> application, >> >> >> >>> >>> >> which likely lifts a reference to a function, which might >> >> choose >> >> >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates >> bigger >> >> and >> >> >> >>> >>> >> bigger types. >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> Anyway, does that all help? >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee >> >> >> Porncharoenwase >> >> >> >>> >>> wrote: >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be >> encoded >> >> as >> >> >> a >> >> >> >>> >>> >> >> function application node that only allows a reference >> as >> >> its >> >> >> >>> RHS >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But that >> >> said, >> >> >> I'm >> >> >> >>> not >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > 1. I have no problem with the definition site. That?s >> >> easy >> >> >> to >> >> >> >>> deal >> >> >> >>> >>> >> with. >> >> >> >>> >>> >> > 2. I also have no problem with restricting function >> >> >> >>> application. >> >> >> >>> >>> That?s >> >> >> >>> >>> >> > easy to deal with as well. >> >> >> >>> >>> >> > 3. The problem is primarily with a bare variable >> >> reference >> >> >> >>> bound >> >> >> >>> >>> to a >> >> >> >>> >>> >> > closure value, which must only occur in function >> >> >> application. >> >> >> >>> For >> >> >> >>> >>> >> example, >> >> >> >>> >>> >> > the following code is invalid. >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define (f) (void)) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define g f) ;; this is invalid >> >> >> >>> >>> >> >(define h (g)) >> >> >> >>> >>> >> >(define a (list f)) ;; this is also invalid >> >> >> >>> >>> >> >(define b (f)) ;; this is ok >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >I have been using choice-filters-to-apply to restrict >> >> variable >> >> >> >>> >>> references >> >> >> >>> >>> >> >whose type can-unify? with function-type that doesn't >> >> directly >> >> >> >>> appear >> >> >> >>> >>> >> under >> >> >> >>> >>> >> >function application. But this is obviously not ideal >> >> because >> >> >> the >> >> >> >>> >>> filtered >> >> >> >>> >>> >> >nodes are mostly those whose type is totally >> unrestricted. >> >> So >> >> >> >>> >>> filtering >> >> >> >>> >>> >> >them would prevent them from being concretized to other >> >> >> >>> non-function >> >> >> >>> >>> types >> >> >> >>> >>> >> >as well. (Also note that I tried unify! these >> unrestricted >> >> type >> >> >> >>> >>> variables >> >> >> >>> >>> >> >with some concrete non-function type variables, but down >> the >> >> >> >>> road, I >> >> >> >>> >>> get >> >> >> >>> >>> >> an >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps I'm >> not >> >> >> >>> supposed >> >> >> >>> >>> to >> >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called the >> >> >> >>> situation >> >> >> >>> >>> >> >> where a reference is generated that causes a new >> >> definition >> >> >> to >> >> >> >>> be >> >> >> >>> >>> >> >> added such that the variable is in scope at the use >> >> site. IE >> >> >> >>> the >> >> >> >>> >>> >> >> binding is lifted from the use site to some outer node >> >> that >> >> >> >>> contains >> >> >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to >> >> >> definitions >> >> >> >>> that >> >> >> >>> >>> >> >> are created automatically in this way (which is >> ultimately >> >> >> most >> >> >> >>> >>> >> >> definitions). The exception is caused because somehow >> the >> >> >> lift >> >> >> >>> >>> ended >> >> >> >>> >>> >> >> up in a place where it wasn't actually in scope. That >> >> >> shouldn't >> >> >> >>> >>> >> >> happen. Anyway, if you point me to a version of a >> fuzzer >> >> >> that >> >> >> >>> does >> >> >> >>> >>> >> >> this, I'll try to debug it. >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >#lang clotho >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(require >> >> >> >>> >>> >> > xsmith >> >> >> >>> >>> >> > racr >> >> >> >>> >>> >> > xsmith/racr-convenience >> >> >> >>> >>> >> > xsmith/canned-components >> >> >> >>> >>> >> > racket/string >> >> >> >>> >>> >> > racket/list >> >> >> >>> >>> >> > racket/pretty) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >> >> >>> expressions. >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >> >>> >>> >> > #:VoidExpression #t >> >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >> >>> >>> >> > #:VariableReference #t >> >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >> >>> >>> >> > #:IfExpression #t >> >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >> >>> >>> >> > #:LetSequential #t >> >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >> >>> >>> >> > #:Numbers #t >> >> >> >>> >>> >> > #:Booleans #t >> >> >> >>> >>> >> > #:Strings #t >> >> >> >>> >>> >> > #:ImmutableList #t) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-loop-over-container >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > #:name Map >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> >> (list-type >> >> >> >>> >>> inner))) >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >> >> >>> inner)))) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-to-grammar >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > [MyProgram #f >> >> >> >>> >>> >> > ([decls : Definition *] >> >> >> >>> >>> >> > [declsTwo : MyDefinition *] >> >> >> >>> >>> >> > [body : Expression]) >> >> >> >>> >>> >> > #:prop type-info >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >>> >>> >> > (? (n t) >> >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> >> >>> >>> >> > 'declsTwo (? (c) >> (fresh-type-variable)) >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >> >>> >>> >> > #:prop binder-info () >> >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? >> >> (n t) >> >> >> >>> (hash >> >> >> >>> >>> 'e >> >> >> >>> >>> >> t))]]) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-property >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >> >>> 'Definition))]) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete >> >> but >> >> >> >>> needs >> >> >> >>> >>> to be. >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >> >> >>> unconstrained >> >> >> >>> >>> >> >;; type variable. >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >> >>> >>> >> > (list >> >> >> >>> >>> >> > (?()int-type) >> >> >> >>> >>> >> > (?()bool-type) >> >> >> >>> >>> >> > (?()string-type) >> >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >> >>> >>> >> > (define out (open-output-string)) >> >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >> >>> >>> >> > (get-output-string out)) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >> >>> >>> >> > [somelisp] >> >> >> >>> >>> >> > #:program-node MyProgram >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >> >>> >>> >> > (string-join >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) >> lines) >> >> >> >>> >>> >> > "\n")) >> >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >> Probably something is going wrong in the `fresh` >> property. >> >> >> >>> Have you >> >> >> >>> >>> >> >> changed that? >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >I don?t think so. See the code at the end of the email. >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >> I should change that error message. Anyway, it seems >> to >> >> >> think >> >> >> >>> there >> >> >> >>> >>> >> >> is no outer node that has a `Definition *` child that >> can >> >> be >> >> >> the >> >> >> >>> >>> >> >> target for lifting a definition. Generally this >> happens >> >> when >> >> >> >>> you >> >> >> >>> >>> >> >> forget to have an outer Program node or something that >> can >> >> >> hold >> >> >> >>> >>> >> >> definitions. >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >#lang clotho >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(require >> >> >> >>> >>> >> > xsmith >> >> >> >>> >>> >> > racr >> >> >> >>> >>> >> > xsmith/racr-convenience >> >> >> >>> >>> >> > xsmith/canned-components >> >> >> >>> >>> >> > racket/string >> >> >> >>> >>> >> > racket/list >> >> >> >>> >>> >> > racket/pretty) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch of >> >> >> >>> expressions. >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >> >>> >>> >> > #:VoidExpression #t >> >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >> >>> >>> >> > #:VariableReference #t >> >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >> >>> >>> >> > #:IfExpression #t >> >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >> >>> >>> >> > #:LetSequential #t >> >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >> >>> >>> >> > #:Numbers #t >> >> >> >>> >>> >> > #:Booleans #t >> >> >> >>> >>> >> > #:Strings #t >> >> >> >>> >>> >> > #:ImmutableList #t) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-loop-over-container >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > #:name Map >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> >> (list-type >> >> >> >>> >>> inner))) >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable (list-type >> >> >> >>> inner)))) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-to-grammar >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > [MyProgram #f >> >> >> >>> >>> >> > ([decls : Definition *] >> >> >> >>> >>> >> > [body : SubProgram]) >> >> >> >>> >>> >> > #:prop type-info >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >>> >>> >> > (? (n t) >> >> >> >>> >>> >> > (hash 'decls (? (c) (fresh-type-variable)) >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >>> >>> >> > [SubProgram #f >> >> >> >>> >>> >> > ([decls : MyDefinition *] >> >> >> >>> >>> >> > [body : Expression]) >> >> >> >>> >>> >> > #:prop type-info >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >>> >>> >> > (? (n t) >> >> >> >>> >>> >> > (hash 'decls (? (c) >> (fresh-type-variable)) >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >> >>> >>> >> > #:prop binder-info () >> >> >> >>> >>> >> > #:prop type-info [(fresh-type-variable) (? >> >> (n t) >> >> >> >>> (hash >> >> >> >>> >>> 'e >> >> >> >>> >>> >> t))]]) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(add-property >> >> >> >>> >>> >> > somelisp >> >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >> >>> 'Definition))]) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not concrete >> >> but >> >> >> >>> needs >> >> >> >>> >>> to be. >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for an >> >> >> >>> unconstrained >> >> >> >>> >>> >> >;; type variable. >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >> >>> >>> >> > (list >> >> >> >>> >>> >> > (?()int-type) >> >> >> >>> >>> >> > (?()bool-type) >> >> >> >>> >>> >> > (?()string-type) >> >> >> >>> >>> >> > (?()(immutable (list-type (fresh-type-variable)))))) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >> >>> >>> >> > (define out (open-output-string)) >> >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >> >>> >>> >> > (get-output-string out)) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >> >>> >>> >> > [somelisp] >> >> >> >>> >>> >> > #:program-node MyProgram >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >> >>> >>> >> > (string-join >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) >> lines) >> >> >> >>> >>> >> > "\n")) >> >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> From sorawee.pwase at gmail.com Sun Jul 25 00:08:29 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sat, 24 Jul 2021 23:08:29 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: Message-ID: Here's another one: https://github.com/dafny-lang/dafny/issues/1325 On Tue, Jul 20, 2021 at 11:02 AM Eric Eide wrote: > Sorawee Porncharoenwase writes: > > > By the way, here are the results so far: > > - https://github.com/dafny-lang/dafny/issues/1291 > > - https://github.com/dafny-lang/dafny/issues/1297 > > Thank you! We are eager to keep track of these. > > -- > > ------------------------------------------------------------------------------- > Eric Eide . University of Utah School of > Computing > https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, > Utah, USA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sun Jul 25 09:32:28 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 25 Jul 2021 08:32:28 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: Message-ID: Here's another one: https://github.com/dafny-lang/dafny/issues/1326 On Sat, Jul 24, 2021 at 11:08 PM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Here's another one: https://github.com/dafny-lang/dafny/issues/1325 > > On Tue, Jul 20, 2021 at 11:02 AM Eric Eide wrote: > >> Sorawee Porncharoenwase writes: >> >> > By the way, here are the results so far: >> > - https://github.com/dafny-lang/dafny/issues/1291 >> > - https://github.com/dafny-lang/dafny/issues/1297 >> >> Thank you! We are eager to keep track of these. >> >> -- >> >> ------------------------------------------------------------------------------- >> Eric Eide . University of Utah School of >> Computing >> https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, >> Utah, USA >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Sun Jul 25 10:08:18 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Sun, 25 Jul 2021 09:08:18 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: The equal operator works on all datatypes that can be compared. That means pretty much everything except function, and a container that contains function. How should I express this constraint? Use choice-filters-to-apply to inspect the type of the current hole, and reject it if it can-unify? with function or (recursively) container of a function? My main problem with choice-filters-to-apply is that the type of the current-hole is mostly a type variable that looks unconstrained, so it will unify with everything, making rejection look wrong. On Thu, Jul 22, 2021 at 9:33 AM William G Hatch wrote: > >If this is not the right way to do things, I would appreciate it if you > can > >point to me the right way. > > No, it is the right way. > > >Are you suggesting that this call is not necessary? > > > > > https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 > > It is necessary for two things: > > (1) In case of unconstrained types like you found. Your list example > is a good point. I think the SML fuzzer was actually where I was > seeing this bug myself, probably for similar reasons. But I haven't > been looking at the Xsmith crashes with the SML fuzzer lately, only > the wrong code generation issues. > > (2) Even if the types were fully constrained everywhere, they still > might have type-variable wrappers that make it so you can't directly > use the accessor functions. Of course, I should write functions like > `function-type-argument!` that unify and unwrap themselves. But as I > haven't, the result of `concretize-type` has no type variable > wrappers. > > > On Wed, Jul 21, 2021 at 06:47:12PM -0700, Sorawee Porncharoenwase wrote: > >Are you suggesting that this call is not necessary? > > > > > https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 > > > >One instance where concretize-type is used for my fuzzer is when the AST > is: > > > >(PrintStmt (Array (list (Array (list)) (Array (list))))) > > > > > > - PrintStmt doesn?t constrain the type of the child node. > > - The inner type of (Array (list)) is unconstrained, and indeed, by > > itself it can have any type. E.g., seq, seq, seq> > > - There?s a constraint that the type of the left (Array (list)) must be > > the same as the type of the right (Array (list)). > > > >Using concretize-type + unify!, I can communicate across both empty array > >printing that they will have the same type like seq, making the outer > >list have type seq>. > > > >If this is not the right way to do things, I would appreciate it if you > can > >point to me the right way. > > > >On Wed, Jul 21, 2021 at 6:26 PM William G Hatch > wrote: > > > >> Thanks! Note that if concretization needs to call up these thunks > >> during printing it is because you still have unconstrained type > >> variables at printing time. That's not necessarily a problem, > >> exactly, but it might mean that thing is not really used. > >> > >> On Wed, Jul 21, 2021 at 03:45:38PM -0700, Sorawee Porncharoenwase wrote: > >> >I submitted a PR that fixes the problem. > >> >current-xsmith-type-constructor-thunks needs to be parameterized for > >> >printing. Note that this is not only required by my fuzzer, but also by > >> >several example fuzzers that use type->string. I.e., standard-ml and > cish. > >> > > >> >On Wed, Jul 21, 2021 at 3:08 PM William G Hatch > >> wrote: > >> > > >> >> The default-base-type should never show up in a fuzzer that has > >> >> defined the type property for any node. So this is definitely a bug. > >> >> I had the default-base-type show up once before, but some other > change > >> >> made me stop seeing it, and I never got to the bottom of it. It's > >> >> probably because of the #:type-thunks argument that you need for > >> >> concretizing types of lifted definitions. The default was a list > with > >> >> the default base type. This was always a bad idea. I've just pushed > >> >> a change that instead errors if the concretization list is used > >> >> without being parameterized with something new first. > >> >> > >> >> At any rate, I think this is an xsmith bug, not a bug in your code. > >> >> Somewhere the `current-xsmith-type-constructor-thunks` parameter is > >> >> being called where it hasn't been parameterized. If the new > exception > >> >> that will be raised gives any useful information about where this is > >> >> happening, let me know. > >> >> > >> >> On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase > wrote: > >> >> >Sure, I can do that. > >> >> > > >> >> >I have another question about type->string. I have been using > >> type->string > >> >> >from standard-ml / cish examples. > >> >> > > >> >> >(define (type->string t) > >> >> > (let loop ([t* t]) > >> >> > (define t (concretize-type t*)) > >> >> > (unify! t t*) > >> >> > (cond > >> >> > [(can-unify? t (immutable (array-type (fresh-type-variable)))) > >> >> > (define inner (fresh-type-variable)) > >> >> > (unify! t (immutable (array-type inner))) > >> >> > (format "seq<~a>" (loop inner))] > >> >> > [(can-unify? t (product-type #f)) > >> >> > (match (product-type-inner-type-list t) > >> >> > [(list) "()"] > >> >> > [(list a) (loop a)] > >> >> > [inners (format "(~a)" (string-join (map loop inners) ", > >> "))])] > >> >> > [(can-unify? t (function-type (fresh-type-variable) > >> >> >(fresh-type-variable))) > >> >> > (define ret (fresh-type-variable)) > >> >> > (define arg (fresh-type-variable)) > >> >> > (unify! t (function-type arg ret)) > >> >> > (format "(~a -> ~a)" (loop arg) (loop ret))] > >> >> > [(can-unify? t number-type) "int"] > >> >> > [(can-unify? t int-type) "int"] > >> >> > [(can-unify? t bool-type) "bool"] > >> >> > [(can-unify? t string-type) (if (zero? (random 2)) "string" > >> >> "seq")] > >> >> > [else #;(error 'type->string "Type not implemented yet: ~v" t) > >> >> > "FOOBAR"]))) > >> >> > > >> >> >The first weird thing is that the number-type branch is necessary, > >> despite > >> >> >the fact that I don?t have any float in the program. > >> >> > > >> >> >The second weird thing is that sometimes "FOOBAR" is reached > because t > >> is > >> >> >#. In particular, it generates the > following > >> >> code: > >> >> > > >> >> >safeSeqSet((var theArray : seq> := []; theArray), 1, > [1]) > >> >> ># roughly (vector-set (let ([theArray (vector) : (array (array > >> >> >FOOBAR))]) theArray) 1 (vector 1)) > >> >> > > >> >> >where you can see that ?FOOBAR? here should really be int. > >> >> > > >> >> >In my experiment, replacing default-base-type with int seems to work > >> well, > >> >> >but it indicates something is obviously wrong. Here?s the call site > of > >> >> >type->string FWIW: > >> >> > > >> >> >[ImmutableArrayLiteral > >> >> > (? (n) > >> >> > (match (ast-children (ast-child 'expressions n)) > >> >> > [(list) > >> >> > ;; sorry for convoluted logic, but I used to use inner-type > for > >> >> >something and haven't refactored this. > >> >> > (define inner-type (fresh-type-variable)) > >> >> > (define the-array-type (immutable (array-type inner-type))) > >> >> > (define the-type (concretize-type ($xsmith_type n))) > >> >> > (unify! the-type the-array-type) > >> >> > (text (format "(var theArray : ~a := []; theArray)" > >> >> >(type->string the-type)))] > >> >> > [xs (h-append lbracket > >> >> > (comma-list (pretty-print-children xs)) > >> >> > rbracket)]))] > >> >> > > >> >> >Any ideas? > >> >> > > >> >> > > >> >> >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch > > >> >> wrote: > >> >> > > >> >> >> Awesome! We have a git repo where we are keeping track of bugs we > >> >> >> find with Xsmith. If you want, we could add you to it and you can > >> add > >> >> >> any bugs you find to it. Otherwise I will probably put in > anything > >> >> >> you mention like this. > >> >> >> > >> >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase > >> wrote: > >> >> >> >By the way, here are the results so far: > >> >> >> > > >> >> >> >- https://github.com/dafny-lang/dafny/issues/1291 > >> >> >> >- https://github.com/dafny-lang/dafny/issues/1297 > >> >> >> > > >> >> >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < > >> >> >> >sorawee.pwase at gmail.com> wrote: > >> >> >> > > >> >> >> >> We are primarily focused on different compiler backends for > Dafny > >> >> (Dafny > >> >> >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps > more). > >> >> >> >> > >> >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch > >> > > >> >> >> wrote: > >> >> >> >> > >> >> >> >>> >> Do you have a roadmap for implementing ... > >> >> >> >>> > >> >> >> >>> Not at the moment. Right now I'm focused on getting some of > our > >> >> >> >>> mostly-finished fuzzers *really* finished and ready for > fuzzing, > >> >> >> >>> running some fuzzing campaigns to find some bugs, and some > >> features > >> >> >> >>> that are immediately helpful for running fuzzing (eg. some > stuff > >> to > >> >> >> >>> support feedback-directed fuzzing and choice weighting). > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you > >> are > >> >> >> >>> running. > >> >> >> >>> >How many programs are you able to generate and test per > minute? > >> Do > >> >> you > >> >> >> >>> have > >> >> >> >>> >any particular parallelization setup that works well for you? > >> >> >> >>> > >> >> >> >>> There is a harness repository that we use that was mostly > >> written by > >> >> >> >>> Guy Watson: > >> >> >> >>> > >> >> >> >>> https://gitlab.flux.utah.edu/xsmith/harness > >> >> >> >>> > >> >> >> >>> There is a `harness.py` script that takes a configuration > file to > >> >> run > >> >> >> >>> an xsmith fuzzer, running its output through various > >> implementations > >> >> >> >>> of whatever language, comparing the output, and binning the > >> result > >> >> >> >>> (eg. everything matching, diff, crash, everything crashing (IE > >> bad > >> >> >> >>> program), xsmith itself crashing, ...). That harness script > is > >> >> >> >>> basically for running a single job, and there are 2 ways of > >> doing it > >> >> >> >>> in parallel. > >> >> >> >>> > >> >> >> >>> The first one is the `go-button`, which can run fuzzing on > >> multiple > >> >> >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab > >> >> >> >>> (https://www.emulab.net/) to have things set up just right, > and > >> >> >> >>> requires at least 2 machines. It also generates a bunch of > live > >> >> >> >>> graphs about fuzzing. It uses ansible to run setup scripts to > >> prep > >> >> >> >>> each machine before starting the fuzz campaign. > >> >> >> >>> > >> >> >> >>> I mostly use `local-multi-harness` instead, which just runs > on a > >> >> >> >>> single machine, but runs multiple harness processes. It > doesn't > >> >> >> >>> automatically run the setup scripts like the `go-button` > does, so > >> >> >> >>> eg. if you want to use the `racket-kernel-fuzzer` > configuration > >> you > >> >> >> >>> need to run the `racket-install.rkt` script first manually. > >> >> >> >>> > >> >> >> >>> As far as speed, xsmith is not particularly fast. There is a > >> server > >> >> >> >>> mode that at least saves on startup time, but I think the > harness > >> >> >> >>> might not run it particularly well (Guy correct me if I'm > wrong > >> -- > >> >> at > >> >> >> >>> any rate I haven't been using it with server mode, but that > >> might be > >> >> >> >>> because I never bothered changing my config after bugs were > >> fixed). > >> >> >> >>> Aside from Racket's slow startup speed (particularly bad with > >> Xsmith > >> >> >> >>> because it loads a lot of stuff), the runtime varies a lot > based > >> on > >> >> >> >>> the max-depth configuration, the size of the grammar, etc. > >> >> >> >>> > >> >> >> >>> I haven't focused a lot on performance, necessarily. > Basically > >> >> >> >>> whenever I decide Xsmith is too slow for my taste I spend some > >> time > >> >> >> >>> trying to optimize it, and generally make a few gains until I > >> think > >> >> >> >>> ?that's enough optimization for now?. If you can find any > ways > >> to > >> >> >> >>> make improvements, that would be great! I might have taken > all > >> of > >> >> the > >> >> >> >>> low-hanging fruit in my previous optimization passes, however. > >> In > >> >> >> >>> previous optimization passes I got the most out of optimizing > in > >> the > >> >> >> >>> type system, but I made some major improvements to type > checking > >> >> speed > >> >> >> >>> last time so it may not dominate anymore. > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> As an aside, a question for you: Are there multiple Dafny > >> >> >> >>> implementations? Or different optimization levels? Or are > you > >> >> >> >>> running different versions of the compiler against each other? > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee > Porncharoenwase > >> >> >> wrote: > >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you > >> are > >> >> >> >>> running. > >> >> >> >>> >How many programs are you able to generate and test per > minute? > >> Do > >> >> you > >> >> >> >>> have > >> >> >> >>> >any particular parallelization setup that works well for you? > >> >> >> >>> > > >> >> >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < > >> >> >> >>> >sorawee.pwase at gmail.com> wrote: > >> >> >> >>> > > >> >> >> >>> >> Do you have a roadmap for implementing "Provide mechanism > for > >> >> >> >>> precluding > >> >> >> >>> >> types in `fresh-type-variable` invocations"? ( > >> >> >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I > >> also > >> >> >> really > >> >> >> >>> >> want this feature. > >> >> >> >>> >> > >> >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch > >> >> > >> >> >> >>> wrote: > >> >> >> >>> >> > >> >> >> >>> >>> That would be great, thanks! > >> >> >> >>> >>> > >> >> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee > >> >> Porncharoenwase > >> >> >> >>> wrote: > >> >> >> >>> >>> >Yes, this is very helpful. I will let you know if I > >> encounter > >> >> any > >> >> >> >>> further > >> >> >> >>> >>> >issues after updating XSmith. > >> >> >> >>> >>> > > >> >> >> >>> >>> >By the way, do you want any help with Scribble? I notice > a > >> lot > >> >> of > >> >> >> >>> typos > >> >> >> >>> >>> and > >> >> >> >>> >>> >link errors and can submit PRs to fix them. > >> >> >> >>> >>> > > >> >> >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch > >> >> >> >> >> > > >> >> >> >>> >>> wrote: > >> >> >> >>> >>> > > >> >> >> >>> >>> >> Sorry for the delay. I wanted to do the debugging > before > >> >> >> >>> responding, > >> >> >> >>> >>> >> but then I didn't get to it for a while. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed > some > >> >> bugs. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> First, that issue with the bud-node was a bug in the > >> `edit` > >> >> >> >>> property > >> >> >> >>> >>> >> which the canned-components `ProcedureApplication` > uses. > >> >> It's > >> >> >> >>> fixed > >> >> >> >>> >>> >> now. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be > >> encoded > >> >> as > >> >> >> a > >> >> >> >>> >>> >> >> function application node that only allows a > reference > >> as > >> >> its > >> >> >> >>> RHS > >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But > that > >> >> said, > >> >> >> I'm > >> >> >> >>> not > >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it > would > >> be > >> >> >> >>> function > >> >> >> >>> >>> >> application that only allows a reference in the > procedure > >> >> >> >>> position. I > >> >> >> >>> >>> >> don't know why I wrote RHS. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Anyway, the strategy of having separate definition > nodes > >> for > >> >> >> >>> functions > >> >> >> >>> >>> >> and everything else is used in Cish. I don't recommend > >> >> >> following > >> >> >> >>> >>> >> anything Cish does. However, it accomplishes this by > >> having > >> >> the > >> >> >> >>> >>> >> program node have non-function definitions first, and > >> >> >> disallowing > >> >> >> >>> use > >> >> >> >>> >>> >> of functions on the RHS of those functions. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> The problem where it can't find a reference is a bug > >> (that I > >> >> >> >>> haven't > >> >> >> >>> >>> >> fixed yet) caused by the lifting mechanics not really > >> being > >> >> >> >>> completely > >> >> >> >>> >>> >> in sync with the reference mechanics with respect to > nodes > >> >> that > >> >> >> >>> have > >> >> >> >>> >>> >> multiple definition fields. When there are multiple > >> >> definition > >> >> >> >>> >>> >> fields, definitions in the lower field can reference > the > >> >> >> >>> definitions > >> >> >> >>> >>> >> in the higher field, but not vice versa. The lifting > >> >> machinery > >> >> >> >>> >>> >> doesn't know about that ordering, it just stuffs the > >> lifted > >> >> >> >>> definition > >> >> >> >>> >>> >> into the front of the definitions in whatever field it > is > >> >> >> >>> targetting. > >> >> >> >>> >>> >> Thus it places a definition in a place that the > reference > >> >> >> machinery > >> >> >> >>> >>> >> thinks is out of scope. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> So... if you want to have different binders that get > >> >> different > >> >> >> >>> fields > >> >> >> >>> >>> >> in the program node you can only make it work if you > >> disallow > >> >> >> >>> stuff to > >> >> >> >>> >>> >> be sure you never need to reference something that > will be > >> >> >> lifted > >> >> >> >>> into > >> >> >> >>> >>> >> the lower field. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> That said, I've just fixed some issues to allow > definition > >> >> nodes > >> >> >> >>> to be > >> >> >> >>> >>> >> subtypable, so now you can have a definition node that > is > >> a > >> >> >> >>> subtype of > >> >> >> >>> >>> >> Definition, but has some different rules. Importantly, > >> >> though, > >> >> >> it > >> >> >> >>> can > >> >> >> >>> >>> >> be lifted to the same field. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your > >> example > >> >> to > >> >> >> >>> look > >> >> >> >>> >>> >> like this: > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> ``` > >> >> >> >>> >>> >> [MyDefinition Definition () > >> >> >> >>> >>> >> #:prop fresh (hash 'Expression > (make-hole > >> >> >> >>> >>> >> 'ProcedureApplication))] > >> >> >> >>> >>> >> ``` > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> It can now be lifted among normal definitions, but > always > >> >> have > >> >> >> >>> >>> >> ProcedureApplication as its RHS. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> However, I'm still not convinced that you actually want > >> >> >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to > >> restrict > >> >> >> lambda > >> >> >> >>> >>> >> nodes to only be generated on the RHS of a > >> FunctionDefinition > >> >> >> node > >> >> >> >>> >>> >> (that's a subtype of Definition), and make > >> FunctionDefinition > >> >> >> >>> always > >> >> >> >>> >>> >> have a Lambda node as its RHS. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> There is an example in the documentation that is like > >> this, > >> >> in > >> >> >> the > >> >> >> >>> >>> >> explanation of the `choice-filters-to-apply` property. > >> But > >> >> I'll > >> >> >> >>> paste > >> >> >> >>> >>> >> it here: > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> ``` > >> >> >> >>> >>> >> (add-choice-method > >> >> >> >>> >>> >> my-component > >> >> >> >>> >>> >> no-lambda-except-global-def > >> >> >> >>> >>> >> [#f (? () #t)] > >> >> >> >>> >>> >> [Lambda (? () > >> >> >> >>> >>> >> (and (parent-node current-hole) > >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node > >> >> >> current-hole)) > >> >> >> >>> >>> >> 'Definition) > >> >> >> >>> >>> >> (parent-node (parent-node > current-hole)) > >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node > >> >> >> (parent-node > >> >> >> >>> >>> >> current-hole))) > >> >> >> >>> >>> >> 'Program)))]) > >> >> >> >>> >>> >> (add-property my-component > >> >> >> >>> >>> >> choice-filters-to-apply > >> >> >> >>> >>> >> [#f (no-lambda-except-global-def)]) > >> >> >> >>> >>> >> ``` > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> You can use something like that to allow Lambda nodes > to > >> >> only be > >> >> >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and > >> define > >> >> >> >>> >>> >> FunctionDefinition like MyDefinition above to always > >> generate > >> >> >> >>> Lambda > >> >> >> >>> >>> >> as its RHS, and I think it should work for a > first-order > >> >> >> language. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type > >> >> >> variables > >> >> >> >>> >>> >> >with some concrete non-function type variables, but > down > >> the > >> >> >> >>> road, I > >> >> >> >>> >>> get > >> >> >> >>> >>> >> an > >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps > I'm > >> not > >> >> >> >>> supposed > >> >> >> >>> >>> to > >> >> >> >>> >>> >> >unify! inside add-choice-method? > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Well, probably not. Generally the only place you want > to > >> >> >> `unify!` > >> >> >> >>> is > >> >> >> >>> >>> >> inside the type property in the function that > determines > >> >> child > >> >> >> >>> types > >> >> >> >>> >>> >> based on the parent type. Inside that function you can > >> >> >> `unify!`, > >> >> >> >>> but > >> >> >> >>> >>> >> the unification needs to be consistent (IE not unified > >> based > >> >> on > >> >> >> the > >> >> >> >>> >>> >> result of `random`) OR the choice needs to be saved in > a > >> way > >> >> >> that > >> >> >> >>> it > >> >> >> >>> >>> >> can be looked up when that function is run again. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> The problem is that the choice method gets its type > >> >> information > >> >> >> >>> based > >> >> >> >>> >>> >> on a hole node that is later replaced, so the type is > >> >> recomputed > >> >> >> >>> >>> >> without necessarily consulting the type that the hole > >> said it > >> >> >> was. > >> >> >> >>> >>> >> You should be able to constrain the type using > >> unification if > >> >> >> you > >> >> >> >>> save > >> >> >> >>> >>> >> that type to re-unify it consistently later. But > without > >> >> seeing > >> >> >> >>> the > >> >> >> >>> >>> >> exact error and code that generated it, I'm not 100% > >> certain. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > In the last email, I asked if > lift-type->ast-binder-type > >> >> >> could be > >> >> >> >>> >>> used > >> >> >> >>> >>> >> > nondeterministically in this manner, and my > experience > >> >> with it > >> >> >> >>> so far > >> >> >> >>> >>> >> seems > >> >> >> >>> >>> >> > to indicate no. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> I think this should work, barring the issues mentioned > >> above > >> >> >> about > >> >> >> >>> the > >> >> >> >>> >>> >> lifting vs referencing bug when you have the different > >> >> >> definition > >> >> >> >>> >>> >> types in different fields. Adjusting your example to > have > >> >> >> >>> >>> >> MyDefinition as a subtype, and then only having a > >> Definition > >> >> >> field > >> >> >> >>> in > >> >> >> >>> >>> >> MyProgram, random choice seems to work fine. But > >> sometimes > >> >> it > >> >> >> >>> >>> >> generates gargantuan programs (that I kill before it > >> >> finishes) > >> >> >> >>> because > >> >> >> >>> >>> >> it keeps choosing MyDefinition, requiring a function > >> >> >> application, > >> >> >> >>> >>> >> which likely lifts a reference to a function, which > might > >> >> choose > >> >> >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates > >> bigger > >> >> and > >> >> >> >>> >>> >> bigger types. > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> Anyway, does that all help? > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> > >> >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee > >> >> >> Porncharoenwase > >> >> >> >>> >>> wrote: > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be > >> encoded > >> >> as > >> >> >> a > >> >> >> >>> >>> >> >> function application node that only allows a > reference > >> as > >> >> its > >> >> >> >>> RHS > >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But > that > >> >> said, > >> >> >> I'm > >> >> >> >>> not > >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> > 1. I have no problem with the definition site. > That?s > >> >> easy > >> >> >> to > >> >> >> >>> deal > >> >> >> >>> >>> >> with. > >> >> >> >>> >>> >> > 2. I also have no problem with restricting function > >> >> >> >>> application. > >> >> >> >>> >>> That?s > >> >> >> >>> >>> >> > easy to deal with as well. > >> >> >> >>> >>> >> > 3. The problem is primarily with a bare variable > >> >> reference > >> >> >> >>> bound > >> >> >> >>> >>> to a > >> >> >> >>> >>> >> > closure value, which must only occur in function > >> >> >> application. > >> >> >> >>> For > >> >> >> >>> >>> >> example, > >> >> >> >>> >>> >> > the following code is invalid. > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define (f) (void)) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define g f) ;; this is invalid > >> >> >> >>> >>> >> >(define h (g)) > >> >> >> >>> >>> >> >(define a (list f)) ;; this is also invalid > >> >> >> >>> >>> >> >(define b (f)) ;; this is ok > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >I have been using choice-filters-to-apply to restrict > >> >> variable > >> >> >> >>> >>> references > >> >> >> >>> >>> >> >whose type can-unify? with function-type that doesn't > >> >> directly > >> >> >> >>> appear > >> >> >> >>> >>> >> under > >> >> >> >>> >>> >> >function application. But this is obviously not ideal > >> >> because > >> >> >> the > >> >> >> >>> >>> filtered > >> >> >> >>> >>> >> >nodes are mostly those whose type is totally > >> unrestricted. > >> >> So > >> >> >> >>> >>> filtering > >> >> >> >>> >>> >> >them would prevent them from being concretized to > other > >> >> >> >>> non-function > >> >> >> >>> >>> types > >> >> >> >>> >>> >> >as well. (Also note that I tried unify! these > >> unrestricted > >> >> type > >> >> >> >>> >>> variables > >> >> >> >>> >>> >> >with some concrete non-function type variables, but > down > >> the > >> >> >> >>> road, I > >> >> >> >>> >>> get > >> >> >> >>> >>> >> an > >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps > I'm > >> not > >> >> >> >>> supposed > >> >> >> >>> >>> to > >> >> >> >>> >>> >> >unify! inside add-choice-method? > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called > the > >> >> >> >>> situation > >> >> >> >>> >>> >> >> where a reference is generated that causes a new > >> >> definition > >> >> >> to > >> >> >> >>> be > >> >> >> >>> >>> >> >> added such that the variable is in scope at the use > >> >> site. IE > >> >> >> >>> the > >> >> >> >>> >>> >> >> binding is lifted from the use site to some outer > node > >> >> that > >> >> >> >>> contains > >> >> >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to > >> >> >> definitions > >> >> >> >>> that > >> >> >> >>> >>> >> >> are created automatically in this way (which is > >> ultimately > >> >> >> most > >> >> >> >>> >>> >> >> definitions). The exception is caused because > somehow > >> the > >> >> >> lift > >> >> >> >>> >>> ended > >> >> >> >>> >>> >> >> up in a place where it wasn't actually in scope. > That > >> >> >> shouldn't > >> >> >> >>> >>> >> >> happen. Anyway, if you point me to a version of a > >> fuzzer > >> >> >> that > >> >> >> >>> does > >> >> >> >>> >>> >> >> this, I'll try to debug it. > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >#lang clotho > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(require > >> >> >> >>> >>> >> > xsmith > >> >> >> >>> >>> >> > racr > >> >> >> >>> >>> >> > xsmith/racr-convenience > >> >> >> >>> >>> >> > xsmith/canned-components > >> >> >> >>> >>> >> > racket/string > >> >> >> >>> >>> >> > racket/list > >> >> >> >>> >>> >> > racket/pretty) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch > of > >> >> >> >>> expressions. > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-basic-expressions somelisp > >> >> >> >>> >>> >> > #:ProgramWithSequence #t > >> >> >> >>> >>> >> > #:VoidExpression #t > >> >> >> >>> >>> >> > #:AssignmentExpression #t > >> >> >> >>> >>> >> > #:VariableReference #t > >> >> >> >>> >>> >> > #:ProcedureApplication #t > >> >> >> >>> >>> >> > #:IfExpression #t > >> >> >> >>> >>> >> > #:ExpressionSequence #t > >> >> >> >>> >>> >> > #:LetSequential #t > >> >> >> >>> >>> >> > #:LambdaWithExpression #t > >> >> >> >>> >>> >> > #:Numbers #t > >> >> >> >>> >>> >> > #:Booleans #t > >> >> >> >>> >>> >> > #:Strings #t > >> >> >> >>> >>> >> > #:ImmutableList #t) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-loop-over-container > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > #:name Map > >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > >> >> (list-type > >> >> >> >>> >>> inner))) > >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable > (list-type > >> >> >> >>> inner)))) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-to-grammar > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > [MyProgram #f > >> >> >> >>> >>> >> > ([decls : Definition *] > >> >> >> >>> >>> >> > [declsTwo : MyDefinition *] > >> >> >> >>> >>> >> > [body : Expression]) > >> >> >> >>> >>> >> > #:prop type-info > >> >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >> >>> >>> >> > (? (n t) > >> >> >> >>> >>> >> > (hash 'decls (? (c) > (fresh-type-variable)) > >> >> >> >>> >>> >> > 'declsTwo (? (c) > >> (fresh-type-variable)) > >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >> >>> >>> >> > [MyDefinition #f ([type] > >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >> >> >>> >>> >> > [e : ProcedureApplication]) > >> >> >> >>> >>> >> > #:prop binder-info () > >> >> >> >>> >>> >> > #:prop type-info > [(fresh-type-variable) (? > >> >> (n t) > >> >> >> >>> (hash > >> >> >> >>> >>> 'e > >> >> >> >>> >>> >> t))]]) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-property > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > lift-type->ast-binder-type > >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >> >> >>> 'Definition))]) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not > concrete > >> >> but > >> >> >> >>> needs > >> >> >> >>> >>> to be. > >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for > an > >> >> >> >>> unconstrained > >> >> >> >>> >>> >> >;; type variable. > >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >> >> >>> >>> >> > (list > >> >> >> >>> >>> >> > (?()int-type) > >> >> >> >>> >>> >> > (?()bool-type) > >> >> >> >>> >>> >> > (?()string-type) > >> >> >> >>> >>> >> > (?()(immutable (list-type > (fresh-type-variable)))))) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >> >> >>> >>> >> > (define out (open-output-string)) > >> >> >> >>> >>> >> > (for ([symex s-exps]) > >> >> >> >>> >>> >> > (pretty-print symex out 1)) > >> >> >> >>> >>> >> > (get-output-string out)) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define-xsmith-interface-functions > >> >> >> >>> >>> >> > [somelisp] > >> >> >> >>> >>> >> > #:program-node MyProgram > >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >> >> >>> >>> >> > #:comment-wrap (? (lines) > >> >> >> >>> >>> >> > (string-join > >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) > >> lines) > >> >> >> >>> >>> >> > "\n")) > >> >> >> >>> >>> >> > #:format-render somelisp-format-render) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >> Probably something is going wrong in the `fresh` > >> property. > >> >> >> >>> Have you > >> >> >> >>> >>> >> >> changed that? > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >I don?t think so. See the code at the end of the > email. > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >> I should change that error message. Anyway, it > seems > >> to > >> >> >> think > >> >> >> >>> there > >> >> >> >>> >>> >> >> is no outer node that has a `Definition *` child > that > >> can > >> >> be > >> >> >> the > >> >> >> >>> >>> >> >> target for lifting a definition. Generally this > >> happens > >> >> when > >> >> >> >>> you > >> >> >> >>> >>> >> >> forget to have an outer Program node or something > that > >> can > >> >> >> hold > >> >> >> >>> >>> >> >> definitions. > >> >> >> >>> >>> >> >> > >> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >#lang clotho > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(require > >> >> >> >>> >>> >> > xsmith > >> >> >> >>> >>> >> > racr > >> >> >> >>> >>> >> > xsmith/racr-convenience > >> >> >> >>> >>> >> > xsmith/canned-components > >> >> >> >>> >>> >> > racket/string > >> >> >> >>> >>> >> > racket/list > >> >> >> >>> >>> >> > racket/pretty) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch > of > >> >> >> >>> expressions. > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-basic-expressions somelisp > >> >> >> >>> >>> >> > #:ProgramWithSequence #t > >> >> >> >>> >>> >> > #:VoidExpression #t > >> >> >> >>> >>> >> > #:AssignmentExpression #t > >> >> >> >>> >>> >> > #:VariableReference #t > >> >> >> >>> >>> >> > #:ProcedureApplication #t > >> >> >> >>> >>> >> > #:IfExpression #t > >> >> >> >>> >>> >> > #:ExpressionSequence #t > >> >> >> >>> >>> >> > #:LetSequential #t > >> >> >> >>> >>> >> > #:LambdaWithExpression #t > >> >> >> >>> >>> >> > #:Numbers #t > >> >> >> >>> >>> >> > #:Booleans #t > >> >> >> >>> >>> >> > #:Strings #t > >> >> >> >>> >>> >> > #:ImmutableList #t) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-loop-over-container > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > #:name Map > >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable > >> >> (list-type > >> >> >> >>> >>> inner))) > >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable > (list-type > >> >> >> >>> inner)))) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-to-grammar > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > [MyProgram #f > >> >> >> >>> >>> >> > ([decls : Definition *] > >> >> >> >>> >>> >> > [body : SubProgram]) > >> >> >> >>> >>> >> > #:prop type-info > >> >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >> >>> >>> >> > (? (n t) > >> >> >> >>> >>> >> > (hash 'decls (? (c) > (fresh-type-variable)) > >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >> >>> >>> >> > [SubProgram #f > >> >> >> >>> >>> >> > ([decls : MyDefinition *] > >> >> >> >>> >>> >> > [body : Expression]) > >> >> >> >>> >>> >> > #:prop type-info > >> >> >> >>> >>> >> > [(fresh-type-variable) > >> >> >> >>> >>> >> > (? (n t) > >> >> >> >>> >>> >> > (hash 'decls (? (c) > >> (fresh-type-variable)) > >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] > >> >> >> >>> >>> >> > [MyDefinition #f ([type] > >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] > >> >> >> >>> >>> >> > [e : ProcedureApplication]) > >> >> >> >>> >>> >> > #:prop binder-info () > >> >> >> >>> >>> >> > #:prop type-info > [(fresh-type-variable) (? > >> >> (n t) > >> >> >> >>> (hash > >> >> >> >>> >>> 'e > >> >> >> >>> >>> >> t))]]) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(add-property > >> >> >> >>> >>> >> > somelisp > >> >> >> >>> >>> >> > lift-type->ast-binder-type > >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition > >> >> >> >>> 'Definition))]) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not > concrete > >> >> but > >> >> >> >>> needs > >> >> >> >>> >>> to be. > >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for > an > >> >> >> >>> unconstrained > >> >> >> >>> >>> >> >;; type variable. > >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) > >> >> >> >>> >>> >> > (list > >> >> >> >>> >>> >> > (?()int-type) > >> >> >> >>> >>> >> > (?()bool-type) > >> >> >> >>> >>> >> > (?()string-type) > >> >> >> >>> >>> >> > (?()(immutable (list-type > (fresh-type-variable)))))) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) > >> >> >> >>> >>> >> > (define out (open-output-string)) > >> >> >> >>> >>> >> > (for ([symex s-exps]) > >> >> >> >>> >>> >> > (pretty-print symex out 1)) > >> >> >> >>> >>> >> > (get-output-string out)) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(define-xsmith-interface-functions > >> >> >> >>> >>> >> > [somelisp] > >> >> >> >>> >>> >> > #:program-node MyProgram > >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization > >> >> >> >>> >>> >> > #:comment-wrap (? (lines) > >> >> >> >>> >>> >> > (string-join > >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) > >> lines) > >> >> >> >>> >>> >> > "\n")) > >> >> >> >>> >>> >> > #:format-render somelisp-format-render) > >> >> >> >>> >>> >> > > >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) > >> >> >> >>> >>> >> > >> >> >> >>> >>> > >> >> >> >>> >> > >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From regehr at cs.utah.edu Sun Jul 25 12:20:31 2021 From: regehr at cs.utah.edu (John Regehr) Date: Sun, 25 Jul 2021 12:20:31 -0600 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: Message-ID: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> This is cool. I hope someone is tweeting about it :) John On 7/25/21 12:08 AM, Sorawee Porncharoenwase wrote: > Here's another one: https://github.com/dafny-lang/dafny/issues/1325 > > > On Tue, Jul 20, 2021 at 11:02 AM Eric Eide > wrote: > > Sorawee Porncharoenwase > writes: > > > By the way, here are the results so far: > > - https://github.com/dafny-lang/dafny/issues/1291 > > > - https://github.com/dafny-lang/dafny/issues/1297 > > > Thank you!? We are eager to keep track of these. > > -- > ------------------------------------------------------------------------------- > Eric Eide >? ?. > University of Utah School of Computing > https://www.cs.utah.edu/~eeide/ . > +1 801-585-5512 .? ?Salt Lake City, Utah, USA > From sorawee.pwase at gmail.com Tue Jul 27 10:33:40 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 27 Jul 2021 09:33:40 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Here's another one: https://github.com/dafny-lang/dafny/issues/1331 On Sun, Jul 25, 2021 at 11:20 AM John Regehr wrote: > This is cool. I hope someone is tweeting about it :) > > John > > > On 7/25/21 12:08 AM, Sorawee Porncharoenwase wrote: > > Here's another one: https://github.com/dafny-lang/dafny/issues/1325 > > > > > > On Tue, Jul 20, 2021 at 11:02 AM Eric Eide > > wrote: > > > > Sorawee Porncharoenwase > > writes: > > > > > By the way, here are the results so far: > > > - https://github.com/dafny-lang/dafny/issues/1291 > > > > > - https://github.com/dafny-lang/dafny/issues/1297 > > > > > > Thank you! We are eager to keep track of these. > > > > -- > > > ------------------------------------------------------------------------------- > > Eric Eide > . > > University of Utah School of Computing > > https://www.cs.utah.edu/~eeide/ . > > +1 801-585-5512 . Salt Lake City, Utah, USA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sorawee.pwase at gmail.com Tue Jul 27 10:41:11 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Tue, 27 Jul 2021 09:41:11 -0700 Subject: [xsmith-dev] Early Dafny Results (Re: XSmith questions) In-Reply-To: References: <095e7eca-7276-cb50-21b8-1581bb9624bb@cs.utah.edu> Message-ID: Hmm, that was not a hard bug. Apparently, Dafny intentionally limits the depth of an AST structure to be at most 20, causing the crash. Perhaps it should not crash like that, but it seems to "work" the way Dafny developers intend. On Tue, Jul 27, 2021 at 9:33 AM Sorawee Porncharoenwase < sorawee.pwase at gmail.com> wrote: > Here's another one: https://github.com/dafny-lang/dafny/issues/1331 > > On Sun, Jul 25, 2021 at 11:20 AM John Regehr wrote: > >> This is cool. I hope someone is tweeting about it :) >> >> John >> >> >> On 7/25/21 12:08 AM, Sorawee Porncharoenwase wrote: >> > Here's another one: https://github.com/dafny-lang/dafny/issues/1325 >> > >> > >> > On Tue, Jul 20, 2021 at 11:02 AM Eric Eide > > > wrote: >> > >> > Sorawee Porncharoenwase > > > writes: >> > >> > > By the way, here are the results so far: >> > > - https://github.com/dafny-lang/dafny/issues/1291 >> > >> > > - https://github.com/dafny-lang/dafny/issues/1297 >> > >> > >> > Thank you! We are eager to keep track of these. >> > >> > -- >> > >> ------------------------------------------------------------------------------- >> > Eric Eide > . >> > University of Utah School of Computing >> > https://www.cs.utah.edu/~eeide/ . >> > +1 801-585-5512 . Salt Lake City, Utah, USA >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Jul 30 15:19:57 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 30 Jul 2021 15:19:57 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: Sorry, I don't think I've responded to this yet. >The equal operator works on all datatypes that can be compared. That means >pretty much everything except function, and a container that contains >function. How should I express this constraint? Use choice-filters-to-apply to >inspect the type of the current hole, and reject it if it can-unify? with >function or (recursively) container of a function? > >My main problem with choice-filters-to-apply is that the type of the >current-hole is mostly a type variable that looks unconstrained, so it will >unify with everything, making rejection look wrong. You're definitely right that testing if something (that is likely unrestrained) `can-unify?` with a problematic type is not the right way to go about it. I really should add the feature to disallow types in type variables, which is of course the right fix for this issue. However, that feature just hasn't made it to the top of my priorities yet... Sorry :( When I've had this situation, as a workaround, I've just made a function that returns a fresh-type-variable with all of the supported base types and maybe some constructed types. It's... not great. I might add this feature in the coming days. But... I'm not going to promise it right now. On Sun, Jul 25, 2021 at 09:08:18AM -0700, Sorawee Porncharoenwase wrote: >The equal operator works on all datatypes that can be compared. That means >pretty much everything except function, and a container that contains >function. How should I express this constraint? Use choice-filters-to-apply to >inspect the type of the current hole, and reject it if it can-unify? with >function or (recursively) container of a function? > >My main problem with choice-filters-to-apply is that the type of the >current-hole is mostly a type variable that looks unconstrained, so it will >unify with everything, making rejection look wrong. > >On Thu, Jul 22, 2021 at 9:33 AM William G Hatch wrote: > >> >If this is not the right way to do things, I would appreciate it if you >> can >> >point to me the right way. >> >> No, it is the right way. >> >> >Are you suggesting that this call is not necessary? >> > >> > >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 >> >> It is necessary for two things: >> >> (1) In case of unconstrained types like you found. Your list example >> is a good point. I think the SML fuzzer was actually where I was >> seeing this bug myself, probably for similar reasons. But I haven't >> been looking at the Xsmith crashes with the SML fuzzer lately, only >> the wrong code generation issues. >> >> (2) Even if the types were fully constrained everywhere, they still >> might have type-variable wrappers that make it so you can't directly >> use the accessor functions. Of course, I should write functions like >> `function-type-argument!` that unify and unwrap themselves. But as I >> haven't, the result of `concretize-type` has no type variable >> wrappers. >> >> >> On Wed, Jul 21, 2021 at 06:47:12PM -0700, Sorawee Porncharoenwase wrote: >> >Are you suggesting that this call is not necessary? >> > >> > >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/standard-ml/standard-ml.rkt#L881-882 >> > >> >One instance where concretize-type is used for my fuzzer is when the AST >> is: >> > >> >(PrintStmt (Array (list (Array (list)) (Array (list))))) >> > >> > >> > - PrintStmt doesn?t constrain the type of the child node. >> > - The inner type of (Array (list)) is unconstrained, and indeed, by >> > itself it can have any type. E.g., seq, seq, seq> >> > - There?s a constraint that the type of the left (Array (list)) must be >> > the same as the type of the right (Array (list)). >> > >> >Using concretize-type + unify!, I can communicate across both empty array >> >printing that they will have the same type like seq, making the outer >> >list have type seq>. >> > >> >If this is not the right way to do things, I would appreciate it if you >> can >> >point to me the right way. >> > >> >On Wed, Jul 21, 2021 at 6:26 PM William G Hatch >> wrote: >> > >> >> Thanks! Note that if concretization needs to call up these thunks >> >> during printing it is because you still have unconstrained type >> >> variables at printing time. That's not necessarily a problem, >> >> exactly, but it might mean that thing is not really used. >> >> >> >> On Wed, Jul 21, 2021 at 03:45:38PM -0700, Sorawee Porncharoenwase wrote: >> >> >I submitted a PR that fixes the problem. >> >> >current-xsmith-type-constructor-thunks needs to be parameterized for >> >> >printing. Note that this is not only required by my fuzzer, but also by >> >> >several example fuzzers that use type->string. I.e., standard-ml and >> cish. >> >> > >> >> >On Wed, Jul 21, 2021 at 3:08 PM William G Hatch >> >> wrote: >> >> > >> >> >> The default-base-type should never show up in a fuzzer that has >> >> >> defined the type property for any node. So this is definitely a bug. >> >> >> I had the default-base-type show up once before, but some other >> change >> >> >> made me stop seeing it, and I never got to the bottom of it. It's >> >> >> probably because of the #:type-thunks argument that you need for >> >> >> concretizing types of lifted definitions. The default was a list >> with >> >> >> the default base type. This was always a bad idea. I've just pushed >> >> >> a change that instead errors if the concretization list is used >> >> >> without being parameterized with something new first. >> >> >> >> >> >> At any rate, I think this is an xsmith bug, not a bug in your code. >> >> >> Somewhere the `current-xsmith-type-constructor-thunks` parameter is >> >> >> being called where it hasn't been parameterized. If the new >> exception >> >> >> that will be raised gives any useful information about where this is >> >> >> happening, let me know. >> >> >> >> >> >> On Wed, Jul 21, 2021 at 11:22:07AM -0700, Sorawee Porncharoenwase >> wrote: >> >> >> >Sure, I can do that. >> >> >> > >> >> >> >I have another question about type->string. I have been using >> >> type->string >> >> >> >from standard-ml / cish examples. >> >> >> > >> >> >> >(define (type->string t) >> >> >> > (let loop ([t* t]) >> >> >> > (define t (concretize-type t*)) >> >> >> > (unify! t t*) >> >> >> > (cond >> >> >> > [(can-unify? t (immutable (array-type (fresh-type-variable)))) >> >> >> > (define inner (fresh-type-variable)) >> >> >> > (unify! t (immutable (array-type inner))) >> >> >> > (format "seq<~a>" (loop inner))] >> >> >> > [(can-unify? t (product-type #f)) >> >> >> > (match (product-type-inner-type-list t) >> >> >> > [(list) "()"] >> >> >> > [(list a) (loop a)] >> >> >> > [inners (format "(~a)" (string-join (map loop inners) ", >> >> "))])] >> >> >> > [(can-unify? t (function-type (fresh-type-variable) >> >> >> >(fresh-type-variable))) >> >> >> > (define ret (fresh-type-variable)) >> >> >> > (define arg (fresh-type-variable)) >> >> >> > (unify! t (function-type arg ret)) >> >> >> > (format "(~a -> ~a)" (loop arg) (loop ret))] >> >> >> > [(can-unify? t number-type) "int"] >> >> >> > [(can-unify? t int-type) "int"] >> >> >> > [(can-unify? t bool-type) "bool"] >> >> >> > [(can-unify? t string-type) (if (zero? (random 2)) "string" >> >> >> "seq")] >> >> >> > [else #;(error 'type->string "Type not implemented yet: ~v" t) >> >> >> > "FOOBAR"]))) >> >> >> > >> >> >> >The first weird thing is that the number-type branch is necessary, >> >> despite >> >> >> >the fact that I don?t have any float in the program. >> >> >> > >> >> >> >The second weird thing is that sometimes "FOOBAR" is reached >> because t >> >> is >> >> >> >#. In particular, it generates the >> following >> >> >> code: >> >> >> > >> >> >> >safeSeqSet((var theArray : seq> := []; theArray), 1, >> [1]) >> >> >> ># roughly (vector-set (let ([theArray (vector) : (array (array >> >> >> >FOOBAR))]) theArray) 1 (vector 1)) >> >> >> > >> >> >> >where you can see that ?FOOBAR? here should really be int. >> >> >> > >> >> >> >In my experiment, replacing default-base-type with int seems to work >> >> well, >> >> >> >but it indicates something is obviously wrong. Here?s the call site >> of >> >> >> >type->string FWIW: >> >> >> > >> >> >> >[ImmutableArrayLiteral >> >> >> > (? (n) >> >> >> > (match (ast-children (ast-child 'expressions n)) >> >> >> > [(list) >> >> >> > ;; sorry for convoluted logic, but I used to use inner-type >> for >> >> >> >something and haven't refactored this. >> >> >> > (define inner-type (fresh-type-variable)) >> >> >> > (define the-array-type (immutable (array-type inner-type))) >> >> >> > (define the-type (concretize-type ($xsmith_type n))) >> >> >> > (unify! the-type the-array-type) >> >> >> > (text (format "(var theArray : ~a := []; theArray)" >> >> >> >(type->string the-type)))] >> >> >> > [xs (h-append lbracket >> >> >> > (comma-list (pretty-print-children xs)) >> >> >> > rbracket)]))] >> >> >> > >> >> >> >Any ideas? >> >> >> > >> >> >> > >> >> >> >On Tue, Jul 20, 2021 at 10:45 AM William G Hatch > > >> >> >> wrote: >> >> >> > >> >> >> >> Awesome! We have a git repo where we are keeping track of bugs we >> >> >> >> find with Xsmith. If you want, we could add you to it and you can >> >> add >> >> >> >> any bugs you find to it. Otherwise I will probably put in >> anything >> >> >> >> you mention like this. >> >> >> >> >> >> >> >> On Tue, Jul 20, 2021 at 09:58:55AM -0700, Sorawee Porncharoenwase >> >> wrote: >> >> >> >> >By the way, here are the results so far: >> >> >> >> > >> >> >> >> >- https://github.com/dafny-lang/dafny/issues/1291 >> >> >> >> >- https://github.com/dafny-lang/dafny/issues/1297 >> >> >> >> > >> >> >> >> >On Tue, Jul 20, 2021 at 9:57 AM Sorawee Porncharoenwase < >> >> >> >> >sorawee.pwase at gmail.com> wrote: >> >> >> >> > >> >> >> >> >> We are primarily focused on different compiler backends for >> Dafny >> >> >> (Dafny >> >> >> >> >> can be compiled to Java, C#, Rust, JS, Go, C++, and perhaps >> more). >> >> >> >> >> >> >> >> >> >> On Tue, Jul 20, 2021 at 9:34 AM William G Hatch >> > >> > >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >>> >> Do you have a roadmap for implementing ... >> >> >> >> >>> >> >> >> >> >>> Not at the moment. Right now I'm focused on getting some of >> our >> >> >> >> >>> mostly-finished fuzzers *really* finished and ready for >> fuzzing, >> >> >> >> >>> running some fuzzing campaigns to find some bugs, and some >> >> features >> >> >> >> >>> that are immediately helpful for running fuzzing (eg. some >> stuff >> >> to >> >> >> >> >>> support feedback-directed fuzzing and choice weighting). >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you >> >> are >> >> >> >> >>> running. >> >> >> >> >>> >How many programs are you able to generate and test per >> minute? >> >> Do >> >> >> you >> >> >> >> >>> have >> >> >> >> >>> >any particular parallelization setup that works well for you? >> >> >> >> >>> >> >> >> >> >>> There is a harness repository that we use that was mostly >> >> written by >> >> >> >> >>> Guy Watson: >> >> >> >> >>> >> >> >> >> >>> https://gitlab.flux.utah.edu/xsmith/harness >> >> >> >> >>> >> >> >> >> >>> There is a `harness.py` script that takes a configuration >> file to >> >> >> run >> >> >> >> >>> an xsmith fuzzer, running its output through various >> >> implementations >> >> >> >> >>> of whatever language, comparing the output, and binning the >> >> result >> >> >> >> >>> (eg. everything matching, diff, crash, everything crashing (IE >> >> bad >> >> >> >> >>> program), xsmith itself crashing, ...). That harness script >> is >> >> >> >> >>> basically for running a single job, and there are 2 ways of >> >> doing it >> >> >> >> >>> in parallel. >> >> >> >> >>> >> >> >> >> >>> The first one is the `go-button`, which can run fuzzing on >> >> multiple >> >> >> >> >>> machines at once. It is fairly dependent on Emulab/Cloudlab >> >> >> >> >>> (https://www.emulab.net/) to have things set up just right, >> and >> >> >> >> >>> requires at least 2 machines. It also generates a bunch of >> live >> >> >> >> >>> graphs about fuzzing. It uses ansible to run setup scripts to >> >> prep >> >> >> >> >>> each machine before starting the fuzz campaign. >> >> >> >> >>> >> >> >> >> >>> I mostly use `local-multi-harness` instead, which just runs >> on a >> >> >> >> >>> single machine, but runs multiple harness processes. It >> doesn't >> >> >> >> >>> automatically run the setup scripts like the `go-button` >> does, so >> >> >> >> >>> eg. if you want to use the `racket-kernel-fuzzer` >> configuration >> >> you >> >> >> >> >>> need to run the `racket-install.rkt` script first manually. >> >> >> >> >>> >> >> >> >> >>> As far as speed, xsmith is not particularly fast. There is a >> >> server >> >> >> >> >>> mode that at least saves on startup time, but I think the >> harness >> >> >> >> >>> might not run it particularly well (Guy correct me if I'm >> wrong >> >> -- >> >> >> at >> >> >> >> >>> any rate I haven't been using it with server mode, but that >> >> might be >> >> >> >> >>> because I never bothered changing my config after bugs were >> >> fixed). >> >> >> >> >>> Aside from Racket's slow startup speed (particularly bad with >> >> Xsmith >> >> >> >> >>> because it loads a lot of stuff), the runtime varies a lot >> based >> >> on >> >> >> >> >>> the max-depth configuration, the size of the grammar, etc. >> >> >> >> >>> >> >> >> >> >>> I haven't focused a lot on performance, necessarily. >> Basically >> >> >> >> >>> whenever I decide Xsmith is too slow for my taste I spend some >> >> time >> >> >> >> >>> trying to optimize it, and generally make a few gains until I >> >> think >> >> >> >> >>> ?that's enough optimization for now?. If you can find any >> ways >> >> to >> >> >> >> >>> make improvements, that would be great! I might have taken >> all >> >> of >> >> >> the >> >> >> >> >>> low-hanging fruit in my previous optimization passes, however. >> >> In >> >> >> >> >>> previous optimization passes I got the most out of optimizing >> in >> >> the >> >> >> >> >>> type system, but I made some major improvements to type >> checking >> >> >> speed >> >> >> >> >>> last time so it may not dominate anymore. >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> As an aside, a question for you: Are there multiple Dafny >> >> >> >> >>> implementations? Or different optimization levels? Or are >> you >> >> >> >> >>> running different versions of the compiler against each other? >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> On Mon, Jul 19, 2021 at 04:15:04PM -0700, Sorawee >> Porncharoenwase >> >> >> >> wrote: >> >> >> >> >>> >Also, I'm curious about the setup of various fuzzers that you >> >> are >> >> >> >> >>> running. >> >> >> >> >>> >How many programs are you able to generate and test per >> minute? >> >> Do >> >> >> you >> >> >> >> >>> have >> >> >> >> >>> >any particular parallelization setup that works well for you? >> >> >> >> >>> > >> >> >> >> >>> >On Fri, Jul 16, 2021 at 8:32 PM Sorawee Porncharoenwase < >> >> >> >> >>> >sorawee.pwase at gmail.com> wrote: >> >> >> >> >>> > >> >> >> >> >>> >> Do you have a roadmap for implementing "Provide mechanism >> for >> >> >> >> >>> precluding >> >> >> >> >>> >> types in `fresh-type-variable` invocations"? ( >> >> >> >> >>> >> https://gitlab.flux.utah.edu/xsmith/xsmith/-/issues/56). I >> >> also >> >> >> >> really >> >> >> >> >>> >> want this feature. >> >> >> >> >>> >> >> >> >> >> >>> >> On Thu, Jul 15, 2021 at 9:22 PM William G Hatch >> >> >> >> >> >> >> >>> wrote: >> >> >> >> >>> >> >> >> >> >> >>> >>> That would be great, thanks! >> >> >> >> >>> >>> >> >> >> >> >>> >>> On Thu, Jul 15, 2021 at 08:09:13PM -0700, Sorawee >> >> >> Porncharoenwase >> >> >> >> >>> wrote: >> >> >> >> >>> >>> >Yes, this is very helpful. I will let you know if I >> >> encounter >> >> >> any >> >> >> >> >>> further >> >> >> >> >>> >>> >issues after updating XSmith. >> >> >> >> >>> >>> > >> >> >> >> >>> >>> >By the way, do you want any help with Scribble? I notice >> a >> >> lot >> >> >> of >> >> >> >> >>> typos >> >> >> >> >>> >>> and >> >> >> >> >>> >>> >link errors and can submit PRs to fix them. >> >> >> >> >>> >>> > >> >> >> >> >>> >>> >On Thu, Jul 15, 2021 at 8:00 PM William G Hatch >> >> >> > >> >> >> > >> >> >> >> >>> >>> wrote: >> >> >> >> >>> >>> > >> >> >> >> >>> >>> >> Sorry for the delay. I wanted to do the debugging >> before >> >> >> >> >>> responding, >> >> >> >> >>> >>> >> but then I didn't get to it for a while. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Anyway, I now see what's going wrong, and I've fixed >> some >> >> >> bugs. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> First, that issue with the bud-node was a bug in the >> >> `edit` >> >> >> >> >>> property >> >> >> >> >>> >>> >> which the canned-components `ProcedureApplication` >> uses. >> >> >> It's >> >> >> >> >>> fixed >> >> >> >> >>> >>> >> now. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be >> >> encoded >> >> >> as >> >> >> >> a >> >> >> >> >>> >>> >> >> function application node that only allows a >> reference >> >> as >> >> >> its >> >> >> >> >>> RHS >> >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But >> that >> >> >> said, >> >> >> >> I'm >> >> >> >> >>> not >> >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Erm, I misspoke (mistyped) a bit. I meant that it >> would >> >> be >> >> >> >> >>> function >> >> >> >> >>> >>> >> application that only allows a reference in the >> procedure >> >> >> >> >>> position. I >> >> >> >> >>> >>> >> don't know why I wrote RHS. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Anyway, the strategy of having separate definition >> nodes >> >> for >> >> >> >> >>> functions >> >> >> >> >>> >>> >> and everything else is used in Cish. I don't recommend >> >> >> >> following >> >> >> >> >>> >>> >> anything Cish does. However, it accomplishes this by >> >> having >> >> >> the >> >> >> >> >>> >>> >> program node have non-function definitions first, and >> >> >> >> disallowing >> >> >> >> >>> use >> >> >> >> >>> >>> >> of functions on the RHS of those functions. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> The problem where it can't find a reference is a bug >> >> (that I >> >> >> >> >>> haven't >> >> >> >> >>> >>> >> fixed yet) caused by the lifting mechanics not really >> >> being >> >> >> >> >>> completely >> >> >> >> >>> >>> >> in sync with the reference mechanics with respect to >> nodes >> >> >> that >> >> >> >> >>> have >> >> >> >> >>> >>> >> multiple definition fields. When there are multiple >> >> >> definition >> >> >> >> >>> >>> >> fields, definitions in the lower field can reference >> the >> >> >> >> >>> definitions >> >> >> >> >>> >>> >> in the higher field, but not vice versa. The lifting >> >> >> machinery >> >> >> >> >>> >>> >> doesn't know about that ordering, it just stuffs the >> >> lifted >> >> >> >> >>> definition >> >> >> >> >>> >>> >> into the front of the definitions in whatever field it >> is >> >> >> >> >>> targetting. >> >> >> >> >>> >>> >> Thus it places a definition in a place that the >> reference >> >> >> >> machinery >> >> >> >> >>> >>> >> thinks is out of scope. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> So... if you want to have different binders that get >> >> >> different >> >> >> >> >>> fields >> >> >> >> >>> >>> >> in the program node you can only make it work if you >> >> disallow >> >> >> >> >>> stuff to >> >> >> >> >>> >>> >> be sure you never need to reference something that >> will be >> >> >> >> lifted >> >> >> >> >>> into >> >> >> >> >>> >>> >> the lower field. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> That said, I've just fixed some issues to allow >> definition >> >> >> nodes >> >> >> >> >>> to be >> >> >> >> >>> >>> >> subtypable, so now you can have a definition node that >> is >> >> a >> >> >> >> >>> subtype of >> >> >> >> >>> >>> >> Definition, but has some different rules. Importantly, >> >> >> though, >> >> >> >> it >> >> >> >> >>> can >> >> >> >> >>> >>> >> be lifted to the same field. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Eg. if I change the MyDefinition declaration in your >> >> example >> >> >> to >> >> >> >> >>> look >> >> >> >> >>> >>> >> like this: >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> ``` >> >> >> >> >>> >>> >> [MyDefinition Definition () >> >> >> >> >>> >>> >> #:prop fresh (hash 'Expression >> (make-hole >> >> >> >> >>> >>> >> 'ProcedureApplication))] >> >> >> >> >>> >>> >> ``` >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> It can now be lifted among normal definitions, but >> always >> >> >> have >> >> >> >> >>> >>> >> ProcedureApplication as its RHS. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> However, I'm still not convinced that you actually want >> >> >> >> >>> >>> >> ProcedureApplication as the RHS. Maybe you want to >> >> restrict >> >> >> >> lambda >> >> >> >> >>> >>> >> nodes to only be generated on the RHS of a >> >> FunctionDefinition >> >> >> >> node >> >> >> >> >>> >>> >> (that's a subtype of Definition), and make >> >> FunctionDefinition >> >> >> >> >>> always >> >> >> >> >>> >>> >> have a Lambda node as its RHS. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> There is an example in the documentation that is like >> >> this, >> >> >> in >> >> >> >> the >> >> >> >> >>> >>> >> explanation of the `choice-filters-to-apply` property. >> >> But >> >> >> I'll >> >> >> >> >>> paste >> >> >> >> >>> >>> >> it here: >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> ``` >> >> >> >> >>> >>> >> (add-choice-method >> >> >> >> >>> >>> >> my-component >> >> >> >> >>> >>> >> no-lambda-except-global-def >> >> >> >> >>> >>> >> [#f (? () #t)] >> >> >> >> >>> >>> >> [Lambda (? () >> >> >> >> >>> >>> >> (and (parent-node current-hole) >> >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> >> >> current-hole)) >> >> >> >> >>> >>> >> 'Definition) >> >> >> >> >>> >>> >> (parent-node (parent-node >> current-hole)) >> >> >> >> >>> >>> >> (equal? (ast-node-type (parent-node >> >> >> >> (parent-node >> >> >> >> >>> >>> >> current-hole))) >> >> >> >> >>> >>> >> 'Program)))]) >> >> >> >> >>> >>> >> (add-property my-component >> >> >> >> >>> >>> >> choice-filters-to-apply >> >> >> >> >>> >>> >> [#f (no-lambda-except-global-def)]) >> >> >> >> >>> >>> >> ``` >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> You can use something like that to allow Lambda nodes >> to >> >> >> only be >> >> >> >> >>> >>> >> generated on the RHS of FunctionDefinition nodes, and >> >> define >> >> >> >> >>> >>> >> FunctionDefinition like MyDefinition above to always >> >> generate >> >> >> >> >>> Lambda >> >> >> >> >>> >>> >> as its RHS, and I think it should work for a >> first-order >> >> >> >> language. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >(Also note that I tried unify! these unrestricted type >> >> >> >> variables >> >> >> >> >>> >>> >> >with some concrete non-function type variables, but >> down >> >> the >> >> >> >> >>> road, I >> >> >> >> >>> >>> get >> >> >> >> >>> >>> >> an >> >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps >> I'm >> >> not >> >> >> >> >>> supposed >> >> >> >> >>> >>> to >> >> >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Well, probably not. Generally the only place you want >> to >> >> >> >> `unify!` >> >> >> >> >>> is >> >> >> >> >>> >>> >> inside the type property in the function that >> determines >> >> >> child >> >> >> >> >>> types >> >> >> >> >>> >>> >> based on the parent type. Inside that function you can >> >> >> >> `unify!`, >> >> >> >> >>> but >> >> >> >> >>> >>> >> the unification needs to be consistent (IE not unified >> >> based >> >> >> on >> >> >> >> the >> >> >> >> >>> >>> >> result of `random`) OR the choice needs to be saved in >> a >> >> way >> >> >> >> that >> >> >> >> >>> it >> >> >> >> >>> >>> >> can be looked up when that function is run again. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> The problem is that the choice method gets its type >> >> >> information >> >> >> >> >>> based >> >> >> >> >>> >>> >> on a hole node that is later replaced, so the type is >> >> >> recomputed >> >> >> >> >>> >>> >> without necessarily consulting the type that the hole >> >> said it >> >> >> >> was. >> >> >> >> >>> >>> >> You should be able to constrain the type using >> >> unification if >> >> >> >> you >> >> >> >> >>> save >> >> >> >> >>> >>> >> that type to re-unify it consistently later. But >> without >> >> >> seeing >> >> >> >> >>> the >> >> >> >> >>> >>> >> exact error and code that generated it, I'm not 100% >> >> certain. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> > In the last email, I asked if >> lift-type->ast-binder-type >> >> >> >> could be >> >> >> >> >>> >>> used >> >> >> >> >>> >>> >> > nondeterministically in this manner, and my >> experience >> >> >> with it >> >> >> >> >>> so far >> >> >> >> >>> >>> >> seems >> >> >> >> >>> >>> >> > to indicate no. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> I think this should work, barring the issues mentioned >> >> above >> >> >> >> about >> >> >> >> >>> the >> >> >> >> >>> >>> >> lifting vs referencing bug when you have the different >> >> >> >> definition >> >> >> >> >>> >>> >> types in different fields. Adjusting your example to >> have >> >> >> >> >>> >>> >> MyDefinition as a subtype, and then only having a >> >> Definition >> >> >> >> field >> >> >> >> >>> in >> >> >> >> >>> >>> >> MyProgram, random choice seems to work fine. But >> >> sometimes >> >> >> it >> >> >> >> >>> >>> >> generates gargantuan programs (that I kill before it >> >> >> finishes) >> >> >> >> >>> because >> >> >> >> >>> >>> >> it keeps choosing MyDefinition, requiring a function >> >> >> >> application, >> >> >> >> >>> >>> >> which likely lifts a reference to a function, which >> might >> >> >> choose >> >> >> >> >>> >>> >> MyDefinition, etc, in a bad lift chain that generates >> >> bigger >> >> >> and >> >> >> >> >>> >>> >> bigger types. >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> Anyway, does that all help? >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> On Tue, Jul 13, 2021 at 02:55:10PM -0700, Sorawee >> >> >> >> Porncharoenwase >> >> >> >> >>> >>> wrote: >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >> Hmm, it seems like first-order functions would be >> >> encoded >> >> >> as >> >> >> >> a >> >> >> >> >>> >>> >> >> function application node that only allows a >> reference >> >> as >> >> >> its >> >> >> >> >>> RHS >> >> >> >> >>> >>> >> >> rather than allowing arbitrary expressions. But >> that >> >> >> said, >> >> >> >> I'm >> >> >> >> >>> not >> >> >> >> >>> >>> >> >> sure I fully understand what you're trying to do. >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >I?m not sure what you mean by that. But: >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> > 1. I have no problem with the definition site. >> That?s >> >> >> easy >> >> >> >> to >> >> >> >> >>> deal >> >> >> >> >>> >>> >> with. >> >> >> >> >>> >>> >> > 2. I also have no problem with restricting function >> >> >> >> >>> application. >> >> >> >> >>> >>> That?s >> >> >> >> >>> >>> >> > easy to deal with as well. >> >> >> >> >>> >>> >> > 3. The problem is primarily with a bare variable >> >> >> reference >> >> >> >> >>> bound >> >> >> >> >>> >>> to a >> >> >> >> >>> >>> >> > closure value, which must only occur in function >> >> >> >> application. >> >> >> >> >>> For >> >> >> >> >>> >>> >> example, >> >> >> >> >>> >>> >> > the following code is invalid. >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define (f) (void)) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define g f) ;; this is invalid >> >> >> >> >>> >>> >> >(define h (g)) >> >> >> >> >>> >>> >> >(define a (list f)) ;; this is also invalid >> >> >> >> >>> >>> >> >(define b (f)) ;; this is ok >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >I have been using choice-filters-to-apply to restrict >> >> >> variable >> >> >> >> >>> >>> references >> >> >> >> >>> >>> >> >whose type can-unify? with function-type that doesn't >> >> >> directly >> >> >> >> >>> appear >> >> >> >> >>> >>> >> under >> >> >> >> >>> >>> >> >function application. But this is obviously not ideal >> >> >> because >> >> >> >> the >> >> >> >> >>> >>> filtered >> >> >> >> >>> >>> >> >nodes are mostly those whose type is totally >> >> unrestricted. >> >> >> So >> >> >> >> >>> >>> filtering >> >> >> >> >>> >>> >> >them would prevent them from being concretized to >> other >> >> >> >> >>> non-function >> >> >> >> >>> >>> types >> >> >> >> >>> >>> >> >as well. (Also note that I tried unify! these >> >> unrestricted >> >> >> type >> >> >> >> >>> >>> variables >> >> >> >> >>> >>> >> >with some concrete non-function type variables, but >> down >> >> the >> >> >> >> >>> road, I >> >> >> >> >>> >>> get >> >> >> >> >>> >>> >> an >> >> >> >> >>> >>> >> >internal error about unification failure.). Perhaps >> I'm >> >> not >> >> >> >> >>> supposed >> >> >> >> >>> >>> to >> >> >> >> >>> >>> >> >unify! inside add-choice-method? >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >That's probably a bug. ?Lifting? is what I've called >> the >> >> >> >> >>> situation >> >> >> >> >>> >>> >> >> where a reference is generated that causes a new >> >> >> definition >> >> >> >> to >> >> >> >> >>> be >> >> >> >> >>> >>> >> >> added such that the variable is in scope at the use >> >> >> site. IE >> >> >> >> >>> the >> >> >> >> >>> >>> >> >> binding is lifted from the use site to some outer >> node >> >> >> that >> >> >> >> >>> contains >> >> >> >> >>> >>> >> >> definitions. `lift_X` is the name Xsmith gives to >> >> >> >> definitions >> >> >> >> >>> that >> >> >> >> >>> >>> >> >> are created automatically in this way (which is >> >> ultimately >> >> >> >> most >> >> >> >> >>> >>> >> >> definitions). The exception is caused because >> somehow >> >> the >> >> >> >> lift >> >> >> >> >>> >>> ended >> >> >> >> >>> >>> >> >> up in a place where it wasn't actually in scope. >> That >> >> >> >> shouldn't >> >> >> >> >>> >>> >> >> happen. Anyway, if you point me to a version of a >> >> fuzzer >> >> >> >> that >> >> >> >> >>> does >> >> >> >> >>> >>> >> >> this, I'll try to debug it. >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >#lang clotho >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(require >> >> >> >> >>> >>> >> > xsmith >> >> >> >> >>> >>> >> > racr >> >> >> >> >>> >>> >> > xsmith/racr-convenience >> >> >> >> >>> >>> >> > xsmith/canned-components >> >> >> >> >>> >>> >> > racket/string >> >> >> >> >>> >>> >> > racket/list >> >> >> >> >>> >>> >> > racket/pretty) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch >> of >> >> >> >> >>> expressions. >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >> >> >>> >>> >> > #:VoidExpression #t >> >> >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >> >> >>> >>> >> > #:VariableReference #t >> >> >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >> >> >>> >>> >> > #:IfExpression #t >> >> >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >> >> >>> >>> >> > #:LetSequential #t >> >> >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >> >> >>> >>> >> > #:Numbers #t >> >> >> >> >>> >>> >> > #:Booleans #t >> >> >> >> >>> >>> >> > #:Strings #t >> >> >> >> >>> >>> >> > #:ImmutableList #t) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-loop-over-container >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > #:name Map >> >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> >> >> (list-type >> >> >> >> >>> >>> inner))) >> >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable >> (list-type >> >> >> >> >>> inner)))) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-to-grammar >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > [MyProgram #f >> >> >> >> >>> >>> >> > ([decls : Definition *] >> >> >> >> >>> >>> >> > [declsTwo : MyDefinition *] >> >> >> >> >>> >>> >> > [body : Expression]) >> >> >> >> >>> >>> >> > #:prop type-info >> >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >> >>> >>> >> > (? (n t) >> >> >> >> >>> >>> >> > (hash 'decls (? (c) >> (fresh-type-variable)) >> >> >> >> >>> >>> >> > 'declsTwo (? (c) >> >> (fresh-type-variable)) >> >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >> >> >>> >>> >> > #:prop binder-info () >> >> >> >> >>> >>> >> > #:prop type-info >> [(fresh-type-variable) (? >> >> >> (n t) >> >> >> >> >>> (hash >> >> >> >> >>> >>> 'e >> >> >> >> >>> >>> >> t))]]) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-property >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >> >> >>> 'Definition))]) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not >> concrete >> >> >> but >> >> >> >> >>> needs >> >> >> >> >>> >>> to be. >> >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for >> an >> >> >> >> >>> unconstrained >> >> >> >> >>> >>> >> >;; type variable. >> >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >> >> >>> >>> >> > (list >> >> >> >> >>> >>> >> > (?()int-type) >> >> >> >> >>> >>> >> > (?()bool-type) >> >> >> >> >>> >>> >> > (?()string-type) >> >> >> >> >>> >>> >> > (?()(immutable (list-type >> (fresh-type-variable)))))) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >> >> >>> >>> >> > (define out (open-output-string)) >> >> >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >> >> >>> >>> >> > (get-output-string out)) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >> >> >>> >>> >> > [somelisp] >> >> >> >> >>> >>> >> > #:program-node MyProgram >> >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >> >> >>> >>> >> > (string-join >> >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) >> >> lines) >> >> >> >> >>> >>> >> > "\n")) >> >> >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >> Probably something is going wrong in the `fresh` >> >> property. >> >> >> >> >>> Have you >> >> >> >> >>> >>> >> >> changed that? >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >I don?t think so. See the code at the end of the >> email. >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >> I should change that error message. Anyway, it >> seems >> >> to >> >> >> >> think >> >> >> >> >>> there >> >> >> >> >>> >>> >> >> is no outer node that has a `Definition *` child >> that >> >> can >> >> >> be >> >> >> >> the >> >> >> >> >>> >>> >> >> target for lifting a definition. Generally this >> >> happens >> >> >> when >> >> >> >> >>> you >> >> >> >> >>> >>> >> >> forget to have an outer Program node or something >> that >> >> can >> >> >> >> hold >> >> >> >> >>> >>> >> >> definitions. >> >> >> >> >>> >>> >> >> >> >> >> >> >>> >>> >> >I don?t think that?s the case, but I could be wrong. >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >#lang clotho >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(require >> >> >> >> >>> >>> >> > xsmith >> >> >> >> >>> >>> >> > racr >> >> >> >> >>> >>> >> > xsmith/racr-convenience >> >> >> >> >>> >>> >> > xsmith/canned-components >> >> >> >> >>> >>> >> > racket/string >> >> >> >> >>> >>> >> > racket/list >> >> >> >> >>> >>> >> > racket/pretty) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >;; We first define a basic component and add a bunch >> of >> >> >> >> >>> expressions. >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define-basic-spec-component somelisp) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-basic-expressions somelisp >> >> >> >> >>> >>> >> > #:ProgramWithSequence #t >> >> >> >> >>> >>> >> > #:VoidExpression #t >> >> >> >> >>> >>> >> > #:AssignmentExpression #t >> >> >> >> >>> >>> >> > #:VariableReference #t >> >> >> >> >>> >>> >> > #:ProcedureApplication #t >> >> >> >> >>> >>> >> > #:IfExpression #t >> >> >> >> >>> >>> >> > #:ExpressionSequence #t >> >> >> >> >>> >>> >> > #:LetSequential #t >> >> >> >> >>> >>> >> > #:LambdaWithExpression #t >> >> >> >> >>> >>> >> > #:Numbers #t >> >> >> >> >>> >>> >> > #:Booleans #t >> >> >> >> >>> >>> >> > #:Strings #t >> >> >> >> >>> >>> >> > #:ImmutableList #t) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-loop-over-container >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > #:name Map >> >> >> >> >>> >>> >> > #:collection-type-constructor (? (inner) (immutable >> >> >> (list-type >> >> >> >> >>> >>> inner))) >> >> >> >> >>> >>> >> > #:loop-type-constructor (? (inner) (immutable >> (list-type >> >> >> >> >>> inner)))) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-to-grammar >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > [MyProgram #f >> >> >> >> >>> >>> >> > ([decls : Definition *] >> >> >> >> >>> >>> >> > [body : SubProgram]) >> >> >> >> >>> >>> >> > #:prop type-info >> >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >> >>> >>> >> > (? (n t) >> >> >> >> >>> >>> >> > (hash 'decls (? (c) >> (fresh-type-variable)) >> >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >> >>> >>> >> > [SubProgram #f >> >> >> >> >>> >>> >> > ([decls : MyDefinition *] >> >> >> >> >>> >>> >> > [body : Expression]) >> >> >> >> >>> >>> >> > #:prop type-info >> >> >> >> >>> >>> >> > [(fresh-type-variable) >> >> >> >> >>> >>> >> > (? (n t) >> >> >> >> >>> >>> >> > (hash 'decls (? (c) >> >> (fresh-type-variable)) >> >> >> >> >>> >>> >> > 'body (fresh-type-variable)))]] >> >> >> >> >>> >>> >> > [MyDefinition #f ([type] >> >> >> >> >>> >>> >> > [name = (fresh-var-name "b_")] >> >> >> >> >>> >>> >> > [e : ProcedureApplication]) >> >> >> >> >>> >>> >> > #:prop binder-info () >> >> >> >> >>> >>> >> > #:prop type-info >> [(fresh-type-variable) (? >> >> >> (n t) >> >> >> >> >>> (hash >> >> >> >> >>> >>> 'e >> >> >> >> >>> >>> >> t))]]) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(add-property >> >> >> >> >>> >>> >> > somelisp >> >> >> >> >>> >>> >> > lift-type->ast-binder-type >> >> >> >> >>> >>> >> > [#f (? (type) (if (zero? (random 2)) 'MyDefinition >> >> >> >> >>> 'Definition))]) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >;; Sometimes we have a type variable that is not >> concrete >> >> >> but >> >> >> >> >>> needs >> >> >> >> >>> >>> to be. >> >> >> >> >>> >>> >> >;; Here we provide a list of options we can pick for >> an >> >> >> >> >>> unconstrained >> >> >> >> >>> >>> >> >;; type variable. >> >> >> >> >>> >>> >> >(define (type-thunks-for-concretization) >> >> >> >> >>> >>> >> > (list >> >> >> >> >>> >>> >> > (?()int-type) >> >> >> >> >>> >>> >> > (?()bool-type) >> >> >> >> >>> >>> >> > (?()string-type) >> >> >> >> >>> >>> >> > (?()(immutable (list-type >> (fresh-type-variable)))))) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define (somelisp-format-render s-exps) >> >> >> >> >>> >>> >> > (define out (open-output-string)) >> >> >> >> >>> >>> >> > (for ([symex s-exps]) >> >> >> >> >>> >>> >> > (pretty-print symex out 1)) >> >> >> >> >>> >>> >> > (get-output-string out)) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(define-xsmith-interface-functions >> >> >> >> >>> >>> >> > [somelisp] >> >> >> >> >>> >>> >> > #:program-node MyProgram >> >> >> >> >>> >>> >> > #:type-thunks type-thunks-for-concretization >> >> >> >> >>> >>> >> > #:comment-wrap (? (lines) >> >> >> >> >>> >>> >> > (string-join >> >> >> >> >>> >>> >> > (map (? (l) (format ";; ~a" l)) >> >> lines) >> >> >> >> >>> >>> >> > "\n")) >> >> >> >> >>> >>> >> > #:format-render somelisp-format-render) >> >> >> >> >>> >>> >> > >> >> >> >> >>> >>> >> >(module+ main (somelisp-command-line)) >> >> >> >> >>> >>> >> >> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From sorawee.pwase at gmail.com Fri Jul 30 16:32:54 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 30 Jul 2021 15:32:54 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: > > However, that feature just hasn't made it to the top of my priorities > yet... Sorry :( > No worries at all! If there's anything that I can help regarding this feature, let me know. It looks like the representation of a type right now consists of a lower bound and an upper bound, and probably this feature will require changing the representation, correct? > When I've had this situation, as a workaround, I've just made a > function that returns a fresh-type-variable with all of the supported > base types and maybe some constructed types. It's... not great. > My question is, say, I want to disallow a function, a list of functions, a list of list of functions, a list of list of list of functions, and so on. Of course, I can express things positively: it's an int or bool or listof... what? A list of fresh-type-variable runs a risk of choosing function type. So I could do a listof (int or bool or listof ...) for another level, and keep increasing levels. Is this what you have in mind? -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Jul 30 18:28:49 2021 From: william at hatch.uno (William G Hatch) Date: Fri, 30 Jul 2021 18:28:49 -0600 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase wrote: >> When I've had this situation, as a workaround, I've just made a >> function that returns a fresh-type-variable with all of the supported >> base types and maybe some constructed types. It's... not great. >> > >My question is, say, I want to disallow a function, a list of functions, a >list of list of functions, a list of list of list of functions, and so on. >Of course, I can express things positively: it's an int or bool or >listof... what? A list of fresh-type-variable runs a risk of choosing >function type. So I could do a listof (int or bool or listof ...) for >another level, and keep increasing levels. Is this what you have in mind? Yeah, that's the problem with it. You can realistically only specify a small set of the infinite set of constructible types (granted in practice the set of types usable by xsmith is bounded as well since it bounds the tree depth of constructed types). But specifying that list is a chore and you probably get poor performance if you actually try to generate many combinations with constructed types. So in practice when I have this situation so far I just... don't generate as many things as I otherwise could. >> However, that feature just hasn't made it to the top of my priorities >> yet... Sorry :( >> > >No worries at all! If there's anything that I can help regarding this >feature, let me know. It looks like the representation of a type right now >consists of a lower bound and an upper bound, and probably this feature >will require changing the representation, correct? The upper and lower bounds are about subtyping. If a type changes through unification it can affect the types connected as upper/lower bounds. This feature shouldn't change that, other than being more information that will be conveyed to the related type variables. When a variable is constrained by unification to remove a type from its allowed list, all related types in its upper/lower bounds lists also remove that type. Eg. if variable A is (or string int (-> int int)) and variable B is (or (subtype-of string) (subtype-of int) (subtype-of (-> int int))) and variable A is constrained to definitely not be a function, variable B could no longer be a function and still be a subtype of A. If two variables are unified with `unify!` instead of `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both directions) they are more or less merged into a single variable. (IE the object that you get from `fresh-type-variable` can be merged with others and they are always canonicalized before doing anything to them.) Right now a type variable has a list of possible types (or #f for unconstrained). The list can have any number of base types and every compound type, but it's limited in that it can only have one of each compound type (though each compound type may have type variables inside and thus represent potentially many types). I think the needed change would be to add a list of disallowed types. I think this list should probably be applied recursively -- IE it should be pushed to child types. So eg. if you disallow function types in a type variable it also disallows lists of functions, etc. I'm not certain if there are exceptions, though. Would it make sense to have a recursive and non-recursive version of this? At any rate, when checking if things `can-unify?`, you would use the disallowed list as well as the allowed list of both variables to determine the outcome. When variables are unified, they need to propagate their disallowed list to related type variables (IE the ones in the upper/lower bounds lists). Another thing that would need to be worked out is what exactly to store in the disallowed list. Constructors for compound types? Or maybe predicates? Probably base types and predicates is my guess. At any rate, since we keep emailing about the issue I've thought through more of it than before, so now I'm much more likely to actually sit down and do it soon. :P From sorawee.pwase at gmail.com Fri Jul 30 18:35:09 2021 From: sorawee.pwase at gmail.com (Sorawee Porncharoenwase) Date: Fri, 30 Jul 2021 17:35:09 -0700 Subject: [xsmith-dev] XSmith questions In-Reply-To: References: Message-ID: As always, thanks! :) On Fri, Jul 30, 2021 at 5:29 PM William G Hatch wrote: > On Fri, Jul 30, 2021 at 03:32:54PM -0700, Sorawee Porncharoenwase wrote: > >> When I've had this situation, as a workaround, I've just made a > >> function that returns a fresh-type-variable with all of the supported > >> base types and maybe some constructed types. It's... not great. > >> > > > >My question is, say, I want to disallow a function, a list of functions, a > >list of list of functions, a list of list of list of functions, and so on. > >Of course, I can express things positively: it's an int or bool or > >listof... what? A list of fresh-type-variable runs a risk of choosing > >function type. So I could do a listof (int or bool or listof ...) for > >another level, and keep increasing levels. Is this what you have in mind? > > Yeah, that's the problem with it. You can realistically only specify > a small set of the infinite set of constructible types (granted in > practice the set of types usable by xsmith is bounded as well since it > bounds the tree depth of constructed types). But specifying that list > is a chore and you probably get poor performance if you actually try > to generate many combinations with constructed types. So in practice > when I have this situation so far I just... don't generate as many > things as I otherwise could. > > >> However, that feature just hasn't made it to the top of my priorities > >> yet... Sorry :( > >> > > > >No worries at all! If there's anything that I can help regarding this > >feature, let me know. It looks like the representation of a type right now > >consists of a lower bound and an upper bound, and probably this feature > >will require changing the representation, correct? > > The upper and lower bounds are about subtyping. If a type changes > through unification it can affect the types connected as upper/lower > bounds. This feature shouldn't change that, other than being more > information that will be conveyed to the related type variables. When > a variable is constrained by unification to remove a type from its > allowed list, all related types in its upper/lower bounds lists also > remove that type. Eg. if variable A is (or string int (-> int int)) > and variable B is (or (subtype-of string) (subtype-of int) (subtype-of > (-> int int))) and variable A is constrained to definitely not be a > function, variable B could no longer be a function and still be a > subtype of A. If two variables are unified with `unify!` instead of > `subtype-unify!` (or equivalently by being `subtype-unify!`ed in both > directions) they are more or less merged into a single variable. (IE > the object that you get from `fresh-type-variable` can be merged with > others and they are always canonicalized before doing anything to > them.) > > Right now a type variable has a list of possible types (or #f for > unconstrained). The list can have any number of base types and every > compound type, but it's limited in that it can only have one of each > compound type (though each compound type may have type variables > inside and thus represent potentially many types). > > I think the needed change would be to add a list of disallowed types. > I think this list should probably be applied recursively -- IE it > should be pushed to child types. So eg. if you disallow function > types in a type variable it also disallows lists of functions, etc. > I'm not certain if there are exceptions, though. Would it make sense > to have a recursive and non-recursive version of this? > > At any rate, when checking if things `can-unify?`, you would use the > disallowed list as well as the allowed list of both variables to > determine the outcome. When variables are unified, they need to > propagate their disallowed list to related type variables (IE the ones > in the upper/lower bounds lists). > > Another thing that would need to be worked out is what exactly to > store in the disallowed list. Constructors for compound types? Or > maybe predicates? Probably base types and predicates is my guess. > > At any rate, since we keep emailing about the issue I've thought > through more of it than before, so now I'm much more likely to > actually sit down and do it soon. :P > -------------- next part -------------- An HTML attachment was scrubbed... URL: