From o.f.sayilir at student.utwente.nl Fri Dec 22 03:51:41 2023 From: o.f.sayilir at student.utwente.nl (Sayilir, O.F. (Omer, Student M-CS)) Date: Fri, 22 Dec 2023 10:51:41 +0000 Subject: [xsmith-dev] Adding "else if" branches to if-statements Message-ID: Hello everyone, I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. I am using the ?simple/javascript.rkt? file as my main source of inspiration. I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: -------------------- (add-to-grammar javascript-comp [ElseIf #f ([test : Expression] [then : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t))]] [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf *] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) -------------------- I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. -------------------- !!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Error encountered while printing program! Options: Fuzzer: simple-javascript Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) Options: --max-depth 2 Seed: 106358447 Debug Log: Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 Exception: RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." Program output captured: -------------------- My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. Kind regards, ?mer Sayilir -------------- next part -------------- An HTML attachment was scrubbed... URL: From guy.watson at utah.edu Fri Dec 22 09:07:36 2023 From: guy.watson at utah.edu (GUY ALEXANDRE WATSON) Date: Fri, 22 Dec 2023 16:07:36 +0000 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: References: Message-ID: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> Hi ?mer, I think everything worked well on the grammar side of things, actually. There are a few clues in the error output as to what exactly went wrong: Error encountered while printing program! and "Unexpected" 'xsmith_render-node "cycle.? This tells me that the AST was successfully constructed, and the error happened as Xsmith was rendering the program. In particular, a node was visited twice making a loop when printing. Could you send us your `render-node-info`? My best guess is that the `render-node-info` for the `elseif` child of the `IfElseStatement` nodes is either trying to print the first of the `elseif` list multiple times, or the recursion is set up from child to child (multiple paths from the `elseif to the `else`) instead of parent to each child. We should be able to help you more once we see your `render-node-info`. Please keep on asking questions if things are confusing! RACR error messages are sometimes obtuse, and it?s hard to know what exactly to look out for. As far as debug tooling goes, Xsmith has a wonderful option to print the S-expression form of the program. You can try it out by passing the flag `--s-exp-on-error true` to print it only when errors occur, or even `--s-exp-print-override true` to print the S-expression form in place of normal program output. This lets you examine the components of your program in AST form if you?re trying to narrow down exactly where something went wrong. I?m fairly certain this flag has saved me dozens of hours of debugging. -Guy On Dec 22, 2023, at 3:51 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: Hello everyone, I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. I am using the ?simple/javascript.rkt? file as my main source of inspiration. I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: -------------------- (add-to-grammar javascript-comp [ElseIf #f ([test : Expression] [then : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t))]] [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf *] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) -------------------- I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. -------------------- !!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Error encountered while printing program! Options: Fuzzer: simple-javascript Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) Options: --max-depth 2 Seed: 106358447 Debug Log: Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 Exception: RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." Program output captured: -------------------- My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. Kind regards, ?mer Sayilir -------------- next part -------------- An HTML attachment was scrubbed... URL: From o.f.sayilir at student.utwente.nl Fri Dec 22 10:21:49 2023 From: o.f.sayilir at student.utwente.nl (Sayilir, O.F. (Omer, Student M-CS)) Date: Fri, 22 Dec 2023 17:21:49 +0000 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> References: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> Message-ID: Hello Guy, Thank you for the reply! I currently have this as the render-node-info: -------------------- [ElseIf (? (n) (h-append (h-append (text "else if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) ))] [IfElseStatement (? (n) (h-append (h-append (text "if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) (att-value 'xsmith_render-node (ast-child 'elseif n)) (text "else") (att-value 'xsmith_render-node (ast-child 'else n))))] -------------------- The main modifications to the original JavaScript example are the addition of an ?elseif? branch in the IfElseStatement render info, and the inclusion of ElseIf that is pretty much the same as the IfElseStatement with the text changed to ?else if? and the else branch removed. Thank you for sharing the debug flags, those will certainly come in handy! I have taken a quick look at the differences between the S-expression outputs with and without the Kleene star and I have noticed that when the star is included the elseif branches seem to be empty (as in ?(elseif ())?). Kind regards, ?mer Sayilir From: GUY ALEXANDRE WATSON Sent: Friday, 22 December 2023 17:08 To: Sayilir, ?.F. (?mer, Student M-CS) Cc: xsmith-dev at flux.utah.edu Subject: Re: [xsmith-dev] Adding "else if" branches to if-statements Hi ?mer, I think everything worked well on the grammar side of things, actually. There are a few clues in the error output as to what exactly went wrong: Error encountered while printing program! and "Unexpected" 'xsmith_render-node "cycle.? This tells me that the AST was successfully constructed, and the error happened as Xsmith was rendering the program. In particular, a node was visited twice making a loop when printing. Could you send us your `render-node-info`? My best guess is that the `render-node-info` for the `elseif` child of the `IfElseStatement` nodes is either trying to print the first of the `elseif` list multiple times, or the recursion is set up from child to child (multiple paths from the `elseif to the `else`) instead of parent to each child. We should be able to help you more once we see your `render-node-info`. Please keep on asking questions if things are confusing! RACR error messages are sometimes obtuse, and it?s hard to know what exactly to look out for. As far as debug tooling goes, Xsmith has a wonderful option to print the S-expression form of the program. You can try it out by passing the flag `--s-exp-on-error true` to print it only when errors occur, or even `--s-exp-print-override true` to print the S-expression form in place of normal program output. This lets you examine the components of your program in AST form if you?re trying to narrow down exactly where something went wrong. I?m fairly certain this flag has saved me dozens of hours of debugging. -Guy On Dec 22, 2023, at 3:51 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: Hello everyone, I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. I am using the ?simple/javascript.rkt? file as my main source of inspiration. I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: -------------------- (add-to-grammar javascript-comp [ElseIf #f ([test : Expression] [then : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t))]] [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf *] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) -------------------- I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. -------------------- !!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Error encountered while printing program! Options: Fuzzer: simple-javascript Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) Options: --max-depth 2 Seed: 106358447 Debug Log: Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 Exception: RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." Program output captured: -------------------- My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. Kind regards, ?mer Sayilir -------------- next part -------------- An HTML attachment was scrubbed... URL: From guy.watson at utah.edu Fri Dec 22 12:15:09 2023 From: guy.watson at utah.edu (GUY ALEXANDRE WATSON) Date: Fri, 22 Dec 2023 19:15:09 +0000 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: References: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> Message-ID: <07DDE0C2-A19D-4379-B5CE-4029258E7D6D@utah.edu> Hi ?mer, Good catch on the else if branches being empty. This is probably what the problem is. I was way off on my guess, but that?s a good thing, because this is much easier to fix. The reason you are seeing an error is because as you render the `IfElseStatement`, it renders `?test` and `?then` just fine, but then it tries to render the `?elseif` child. Remember that the `?elseif` child is a *list* of `ElseIf` nodes. From the reference: add-to-grammar. > If the optional kleene star is supplied, the field will be a list field. > The init-expr for each field specifies a default value for the field. When a node is generated, each init-expr is evaluated unless a non-default value is supplied to the generating function. If no init-expr is supplied, the following defaults are used: [?] For fields with a kleene star, an empty list. So we need two things to happen. 1) We want the generator to properly handle the case where there is no `ElseIf`. 2) We need to randomly generate `ElseIf` nodes to make interesting programs. 1) This is pretty easy. Before you render the list of else ifs, check the length of the `?elseif` list (and maybe rename the child to `?elseifs` for clarity) and if the list is empty, simply don?t render anything, then move on to the else. 2) This should also be straightforward. The `ElseIf` node has `Expression` and `Block` children, which should already be setup for random generation, so we just need to fill the `?elseif` list. Xsmith makes this pretty easy! > For nodes with a kleene star, init-expr may return a list or a number. If a number is provided, the default value is a list of that many of the non-kleene-star default value. By this, the reference means that if you give it a number, it will fill the list with that many of the type without the Kleene star (`ElseIf *` -> list of `ElseIf`). To make interesting programs, you might want to vary the amount of else ifs between 0 and let?s say 10. To do that, you would change the `add-to-grammar` declaration to: ``` [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf * = (random 0 11)] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) ``` William would be able to tell you more about exactly why you got the error you did and why it said there was a cycle. I?m not 100% sure myself. Let me know if anything else pops up! -Guy On Dec 22, 2023, at 10:21 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: Hello Guy, Thank you for the reply! I currently have this as the render-node-info: -------------------- [ElseIf (? (n) (h-append (h-append (text "else if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) ))] [IfElseStatement (? (n) (h-append (h-append (text "if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) (att-value 'xsmith_render-node (ast-child 'elseif n)) (text "else") (att-value 'xsmith_render-node (ast-child 'else n))))] -------------------- The main modifications to the original JavaScript example are the addition of an ?elseif? branch in the IfElseStatement render info, and the inclusion of ElseIf that is pretty much the same as the IfElseStatement with the text changed to ?else if? and the else branch removed. Thank you for sharing the debug flags, those will certainly come in handy! I have taken a quick look at the differences between the S-expression outputs with and without the Kleene star and I have noticed that when the star is included the elseif branches seem to be empty (as in ?(elseif ())?). Kind regards, ?mer Sayilir From: GUY ALEXANDRE WATSON > Sent: Friday, 22 December 2023 17:08 To: Sayilir, ?.F. (?mer, Student M-CS) > Cc: xsmith-dev at flux.utah.edu Subject: Re: [xsmith-dev] Adding "else if" branches to if-statements Hi ?mer, I think everything worked well on the grammar side of things, actually. There are a few clues in the error output as to what exactly went wrong: Error encountered while printing program! and "Unexpected" 'xsmith_render-node "cycle.? This tells me that the AST was successfully constructed, and the error happened as Xsmith was rendering the program. In particular, a node was visited twice making a loop when printing. Could you send us your `render-node-info`? My best guess is that the `render-node-info` for the `elseif` child of the `IfElseStatement` nodes is either trying to print the first of the `elseif` list multiple times, or the recursion is set up from child to child (multiple paths from the `elseif to the `else`) instead of parent to each child. We should be able to help you more once we see your `render-node-info`. Please keep on asking questions if things are confusing! RACR error messages are sometimes obtuse, and it?s hard to know what exactly to look out for. As far as debug tooling goes, Xsmith has a wonderful option to print the S-expression form of the program. You can try it out by passing the flag `--s-exp-on-error true` to print it only when errors occur, or even `--s-exp-print-override true` to print the S-expression form in place of normal program output. This lets you examine the components of your program in AST form if you?re trying to narrow down exactly where something went wrong. I?m fairly certain this flag has saved me dozens of hours of debugging. -Guy On Dec 22, 2023, at 3:51 AM, Sayilir, O.F. (Omer, Student M-CS) wrote: Hello everyone, I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. I am using the ?simple/javascript.rkt? file as my main source of inspiration. I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: -------------------- (add-to-grammar javascript-comp [ElseIf #f ([test : Expression] [then : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t))]] [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf *] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) -------------------- I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. -------------------- !!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Error encountered while printing program! Options: Fuzzer: simple-javascript Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) Options: --max-depth 2 Seed: 106358447 Debug Log: Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 Exception: RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." Program output captured: -------------------- My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. Kind regards, ?mer Sayilir -------------- next part -------------- An HTML attachment was scrubbed... URL: From guy.watson at utah.edu Fri Dec 22 12:24:35 2023 From: guy.watson at utah.edu (GUY ALEXANDRE WATSON) Date: Fri, 22 Dec 2023 19:24:35 +0000 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: References: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> Message-ID: Hi ?mer, It?s always right as I send it that I remember I forgot to include something. 2) This should also be straightforward. The `ElseIf` node has `Expression` and `Block` children, which should already be setup for random generation, so we just need to fill the `?elseif` list. Xsmith makes this pretty easy! Additionally, remember that you need to render each `ElseIf` inside the list. Maybe something like: ``` (h-append (map (? (e) ) (ast-child ?elseif n))) ``` On Dec 22, 2023, at 10:21 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: Hello Guy, Thank you for the reply! I currently have this as the render-node-info: -------------------- [ElseIf (? (n) (h-append (h-append (text "else if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) ))] [IfElseStatement (? (n) (h-append (h-append (text "if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) (att-value 'xsmith_render-node (ast-child 'then n)) (att-value 'xsmith_render-node (ast-child 'elseif n)) (text "else") (att-value 'xsmith_render-node (ast-child 'else n))))] -------------------- The main modifications to the original JavaScript example are the addition of an ?elseif? branch in the IfElseStatement render info, and the inclusion of ElseIf that is pretty much the same as the IfElseStatement with the text changed to ?else if? and the else branch removed. Thank you for sharing the debug flags, those will certainly come in handy! I have taken a quick look at the differences between the S-expression outputs with and without the Kleene star and I have noticed that when the star is included the elseif branches seem to be empty (as in ?(elseif ())?). Kind regards, ?mer Sayilir From: GUY ALEXANDRE WATSON > Sent: Friday, 22 December 2023 17:08 To: Sayilir, ?.F. (?mer, Student M-CS) > Cc: xsmith-dev at flux.utah.edu Subject: Re: [xsmith-dev] Adding "else if" branches to if-statements Hi ?mer, I think everything worked well on the grammar side of things, actually. There are a few clues in the error output as to what exactly went wrong: Error encountered while printing program! and "Unexpected" 'xsmith_render-node "cycle.? This tells me that the AST was successfully constructed, and the error happened as Xsmith was rendering the program. In particular, a node was visited twice making a loop when printing. Could you send us your `render-node-info`? My best guess is that the `render-node-info` for the `elseif` child of the `IfElseStatement` nodes is either trying to print the first of the `elseif` list multiple times, or the recursion is set up from child to child (multiple paths from the `elseif to the `else`) instead of parent to each child. We should be able to help you more once we see your `render-node-info`. Please keep on asking questions if things are confusing! RACR error messages are sometimes obtuse, and it?s hard to know what exactly to look out for. As far as debug tooling goes, Xsmith has a wonderful option to print the S-expression form of the program. You can try it out by passing the flag `--s-exp-on-error true` to print it only when errors occur, or even `--s-exp-print-override true` to print the S-expression form in place of normal program output. This lets you examine the components of your program in AST form if you?re trying to narrow down exactly where something went wrong. I?m fairly certain this flag has saved me dozens of hours of debugging. -Guy On Dec 22, 2023, at 3:51 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: Hello everyone, I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. I am using the ?simple/javascript.rkt? file as my main source of inspiration. I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: -------------------- (add-to-grammar javascript-comp [ElseIf #f ([test : Expression] [then : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t))]] [IfElseStatement Statement ([test : Expression] [then : Block] [elseif : ElseIf *] [else : Block]) #:prop block-user? #t #:prop strict-child-order? #t #:prop type-info [(fresh-maybe-return-type) (? (n t) (hash 'test bool-type 'then t 'elseif t 'else t))]]) -------------------- I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. -------------------- !!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Error encountered while printing program! Options: Fuzzer: simple-javascript Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) Options: --max-depth 2 Seed: 106358447 Debug Log: Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 Exception: RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." Program output captured: -------------------- My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. Kind regards, ?mer Sayilir -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at hatch.uno Fri Dec 22 22:32:21 2023 From: william at hatch.uno (William G Hatch) Date: Fri, 22 Dec 2023 22:32:21 -0700 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: References: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> Message-ID: <2023c9BxfdZYZw5XKVewRdu1Bq@localhost> Guy's advice is great! Let me add one more point. Note that when accessing list children with `ast-child` you get a list node back, then you need to use `ast-children` to turn that into a normal Racket list. You wrote: >[IfElseStatement > (? (n) > (h-append > (h-append (text "if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) > (att-value 'xsmith_render-node (ast-child 'then n)) > (att-value 'xsmith_render-node (ast-child 'elseif n)) > (text "else") > (att-value 'xsmith_render-node (ast-child 'else n))))] I think the line where you access the else-if child should probably be something like: ``` (map (? (x) (att-value 'xsmith_render-node x)) (ast-children (ast-child 'elseif n))) ``` On Fri, Dec 22, 2023 at 07:24:35PM +0000, GUY ALEXANDRE WATSON wrote: >Hi ?mer, > >It?s always right as I send it that I remember I forgot to include something. > >2) This should also be straightforward. The `ElseIf` node has `Expression` and `Block` children, which should already be setup for random generation, so we just need to fill the `?elseif` list. Xsmith makes this pretty easy! >Additionally, remember that you need to render each `ElseIf` inside the list. Maybe something like: >``` >(h-append (map (? (e) ) > (ast-child ?elseif n))) >``` > > >On Dec 22, 2023, at 10:21 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: > >Hello Guy, > >Thank you for the reply! >I currently have this as the render-node-info: >-------------------- > [ElseIf > (? (n) > (h-append > (h-append (text "else if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) > (att-value 'xsmith_render-node (ast-child 'then n)) > ))] > >[IfElseStatement > (? (n) > (h-append > (h-append (text "if") space lparen (att-value 'xsmith_render-node (ast-child 'test n)) rparen) > (att-value 'xsmith_render-node (ast-child 'then n)) > (att-value 'xsmith_render-node (ast-child 'elseif n)) > (text "else") > (att-value 'xsmith_render-node (ast-child 'else n))))] > >-------------------- >The main modifications to the original JavaScript example are the addition of an ?elseif? branch in the IfElseStatement render info, and the inclusion of ElseIf that is pretty much the same as the IfElseStatement with the text changed to ?else if? and the else branch removed. > >Thank you for sharing the debug flags, those will certainly come in handy! >I have taken a quick look at the differences between the S-expression outputs with and without the Kleene star and I have noticed that when the star is included the elseif branches seem to be empty (as in ?(elseif ())?). > > >Kind regards, >?mer Sayilir > > >From: GUY ALEXANDRE WATSON > >Sent: Friday, 22 December 2023 17:08 >To: Sayilir, ?.F. (?mer, Student M-CS) > >Cc: xsmith-dev at flux.utah.edu >Subject: Re: [xsmith-dev] Adding "else if" branches to if-statements > >Hi ?mer, > >I think everything worked well on the grammar side of things, actually. There are a few clues in the error output as to what exactly went wrong: > >Error encountered while printing program! >and >"Unexpected" 'xsmith_render-node "cycle.? > >This tells me that the AST was successfully constructed, and the error happened as Xsmith was rendering the program. In particular, a node was visited twice making a loop when printing. Could you send us your `render-node-info`? My best guess is that the `render-node-info` for the `elseif` child of the `IfElseStatement` nodes is either trying to print the first of the `elseif` list multiple times, or the recursion is set up from child to child (multiple paths from the `elseif to the `else`) instead of parent to each child. We should be able to help you more once we see your `render-node-info`. > >Please keep on asking questions if things are confusing! RACR error messages are sometimes obtuse, and it?s hard to know what exactly to look out for. > >As far as debug tooling goes, Xsmith has a wonderful option to print the S-expression form of the program. You can try it out by passing the flag `--s-exp-on-error true` to print it only when errors occur, or even `--s-exp-print-override true` to print the S-expression form in place of normal program output. This lets you examine the components of your program in AST form if you?re trying to narrow down exactly where something went wrong. I?m fairly certain this flag has saved me dozens of hours of debugging. > >-Guy > > > >On Dec 22, 2023, at 3:51 AM, Sayilir, O.F. (Omer, Student M-CS) > wrote: > >Hello everyone, > >I am currently working on a fuzzer for the language Oberon-0 using Xsmith to create synthetic language samples that are to be used as input for a grammatical inference algorithm. >I am using the ?simple/javascript.rkt? file as my main source of inspiration. >I noticed that this javascript fuzzer does not have an ?else if? in its definition for if-statements. As an experiment, I tried adding it myself by copying the grammar definition for if-else statements from the canned components and modifying it to generate zero-or-more ?else if? branches in the if-statements. The modified definition is displayed below: > >-------------------- >(add-to-grammar > javascript-comp > [ElseIf #f ([test : Expression] > [then : Block]) > #:prop block-user? #t > #:prop strict-child-order? #t > #:prop type-info [(fresh-maybe-return-type) > (? (n t) (hash 'test bool-type > 'then t))]] > > [IfElseStatement Statement > ([test : Expression] > [then : Block] > [elseif : ElseIf *] > [else : Block]) > #:prop block-user? #t > #:prop strict-child-order? #t > #:prop type-info [(fresh-maybe-return-type) > (? (n t) (hash 'test bool-type > 'then t > 'elseif t > 'else t))]]) >-------------------- > >I am wondering if this is correct way to define this since, without the Kleene star, this definition seems to be generating code where every if-statement has an ?else if? branch as it should. Once the Kleene star is added however, the fuzzer no longer seems to work as expected and outputs error messages similar to the one below. > >-------------------- >!!! Xsmith Error !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! >Error encountered while printing program! > >Options: >Fuzzer: simple-javascript >Version: simple-javascript 1.0, xsmith 2.0.6 (de6ace1), in Racket 8.2 (vm-type chez-scheme) >Options: --max-depth 2 >Seed: 106358447 >Debug Log: >Concretizing binding b_3. Type: #)>)>)>, concretized to: #)>)> >Lifting binding "lift_4", of type #)>)>, with SN 14, caused by node with SN 13 >Lifting binding "lift_6", of type #, with SN 19, caused by node with SN 18 >Lifting binding "lift_7", of type #, with SN 25, caused by node with SN 22 >Lifting binding "lift_8", of type #, with SN 29, caused by node with SN 27 >Lifting binding "lift_9", of type #, with SN 32, caused by node with SN 30 >Lifting binding "lift_10", of type #, with SN 34, caused by node with SN 33 >Lifting binding "lift_11", of type #, with SN 36, caused by node with SN 35 >Lifting binding "lift_12", of type #)>, with SN 41, caused by node with SN 39 > > >Exception: >RACR exception: "AG evaluator exception;" "Unexpected" 'xsmith_render-node "cycle." > >Program output captured: >-------------------- > >My apologies if this is an obvious question, I am still new to the world of Racket and RACR an Xsmith and I could not find an example of something similar in the examples folder in the git repository. > >Kind regards, >?mer Sayilir > From eeide at cs.utah.edu Sun Dec 24 10:22:16 2023 From: eeide at cs.utah.edu (Eric Eide) Date: Sun, 24 Dec 2023 10:22:16 -0700 Subject: [xsmith-dev] Adding "else if" branches to if-statements In-Reply-To: <2023c9BxfdZYZw5XKVewRdu1Bq@localhost> (William G. Hatch's message of "Fri, 22 Dec 2023 22:32:21 -0700") References: <2933BA2A-D087-4C9D-B3FD-560CF97740F3@utah.edu> <2023c9BxfdZYZw5XKVewRdu1Bq@localhost> Message-ID: Thank you, Guy and William, for answering this question! -- ------------------------------------------------------------------------------- Eric Eide . University of Utah Kahlert School of Computing https://www.cs.utah.edu/~eeide/ . +1 801-585-5512 . Salt Lake City, Utah, USA