The Sequel to a Dream of Ruby Parser's Grammar

Transcript

1. The Sequel to a Dream of Ruby Parser's Grammar RubyKaigi

   2024 follow up 31 August 2024 Yudai Takada (@ydah)

2. Yudai Takada https://ydah.net X: @ydah_ GitHub: @ydah Mastodon: @[email protected] photo

   by: @hachi

3. Osaka, Japan

4. Live Tour 2024 •2024.8.31 (ONLINE) RubyKaigi 2024 followup •2024.9.13 (Sarajevo)

   EuRuKo 2024 •2024.10.25-26 (Tokyo) Kaigi on Rails 2024

5. RuboCop Headquarters RuboCop RSpec team

6. interagent/committee Committee maintainers team

7. ruby/lrama Committer

8. kyobashi.rb Co-Founder

9. Thank you for coming!

10. None

11. These Slides

12. Today’s talk ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

13. The Sequel to a Dream ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

14. 夢の続き ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

15. ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar Previous Episode

16. Ruby Parser's Grammar ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

17. パーサーの文法? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

18. expr : NUMBER { $$ = $1; } | expr

    '+' expr { $$ = $1 + $3; } | expr '-' expr { $$ = $1 - $3; } | expr '*' expr { $$ = $1 * $3; } | expr '/' expr { $$ = $1 / $3; } | '(' expr ')' { $$ = $2; } ; BNF ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

19. parse.y ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar •File de fi nes the Ruby

    grammar and processing. •Extremely di ff i cult to understand. •Be likened to a Demon Castle, Hell, Monstrous

20. What's di ffi culty? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar •Grammar provided by

    Bison is primitive. •Parser and Lexer are tightly coupled. •Di ff i cult to resolve S/R or R/R con fl icts.

21. Bison provides primitive grammar ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

22. What if there are duplicate codes? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

23. Ruby ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

24. Extract to method ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

25. Bison ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

26. Nothing! ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

27. えっ??? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

28. opt_args_tail : ',' args_tail { $$ = $2; / *

    % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

29. opt_block_args_tail : ',' block_args_tail { $$ = $2; / *

    % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

30. 何がちがうの... ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

31. opt_args_tail : ',' args_tail { $$ = $2; / *

    % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ???????

32. opt_block_args_tail : ',' block_args_tail { $$ = $2; / *

    % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ????????????

33. ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar Seek the Savior…

34. Parameterizing Rules ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

35. %rules args_tail(tail) : ',' tail { $$ = $2; ɹɹ

    ɹ / * % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; % % opt_args_tail: args_tail(arg_tail) ; opt_block_args_tail : args_tail(blk_tail) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

36. %rules args_tail(tail) : ',' tail { $$ = $2; ɹɹ

    ɹ / * % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; % % opt_args_tail: args_tail(arg_tail) ; opt_block_args_tail : args_tail(blk_tail) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar 共通的な構造の定義

37. %rules args_tail(tail) : ',' tail { $$ = $2; ɹɹ

    ɹ / * % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; % % opt_args_tail: args_tail(arg_tail) ; opt_block_args_tail : args_tail(blk_tail) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar 定義されたルールを使う

38. %rules args_tail(tail) : ',' tail { $$ = $2; ɹɹ

    ɹ / * % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; % % opt_args_tail: args_tail(arg_tail) ; opt_block_args_tail : args_tail(blk_tail) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar 引数 これが引数として渡る

39. %rules args_tail(tail) : ',' tail { $$ = $2; ɹɹ

    ɹ / * % ripper: get_value($ : 2); % * / } | / * none * / { $$ = new_args_tail(p, 0, 0, 0, &@0); / * % ripper: rb_ary_new_from_args(3, Qnil, Qnil, Qnil); % * / } ; % % opt_args_tail: args_tail(arg_tail) ; opt_block_args_tail : args_tail(blk_tail) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar 展開される 展開される

40. べんり!! ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

41. Bison provides primitive grammar ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

42. Conditionals ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

43. Ruby ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

44. if down? if right - lower? if right? if P?

    puts 'Hadouken' end end end end ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

45. Bison ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

46. Nothing! ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

47. はい。 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

48. Is it necessary? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar •Yes!!! •Cases where the

    syntax is valid for certain conditions but invalid for others.

49. irb(main) : 001> - > (a) {} = > #<Proc:0

    x 000000012316e698 (irb) : 1 (lambda)> irb(main) : 002> - > (a=1) {} = > #<Proc:0 x 0000000123119918 (irb) : 2 (lambda)> irb(main) : 003> - > ( . . . ) {} <internal:kernel > : 187:in `loop': (irb) : 3 : syntax error, unexpected . . . , expecting ')' (SyntaxError) - > ( . . . ) {} ^ ~ ~ Example ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

50. https://github.com/ruby/lrama/pull/418 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

51. %rule def i ned_rule(X, condition) : %if(condition) X %endif X

    { $$ = $1; } ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 1 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

52. %rule def i ned_rule(X, condition) : %if(condition) X %endif X

    { $$ = $1; } ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar Idea 1 trueを渡す %if(true)

53. %rule def i ned_rule(X, condition) : %if(condition) X %endif X

    { $$ = $1; } ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar =>X X { $$ = $1; } Idea 1 ????????????????????????????????

54. %rule def i ned_rule(X, condition) : %if(condition) X %endif X

    { $$ = $1; } ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar Idea 1 falseを渡す %if(false)

55. %rule def i ned_rule(X, condition) : %if(condition) X %endif X

    { $$ = $1; } ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar => X { $$ = $1; } Idea 1 ????????????

56. Rejected ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

57. Why? ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar •Di ff i cult to understand

    that `%if` and `%endif` are mixed in the terminal, non-terminal symbols.

58. %rule def i ned_rule(X, condition) : X { $$ =

    $1; } %if(condition) ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

59. %rule def i ned_rule(X, condition) : X { $$ =

    $1; } %if(condition) ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar %if(true) trueを渡す

60. %rule def i ned_rule(X, condition) : X { $$ =

    $1; } %if(condition) ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ????????????? => X { $$ = $1; }

61. %rule def i ned_rule(X, condition) : X { $$ =

    $1; } %if(condition) ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar falseを渡す %if(false)

62. %rule def i ned_rule(X, condition) : X { $$ =

    $1; } %if(condition) ; % % r_true : def i ned_rule(number, %true) ; r_false : def i ned_rule(number, %false) ; Idea 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ??????????????????????????? => /* empty */

63. Approved! ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

64. だけど、 改善点はある ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < 構文をもう少し改善したい

65. %rule if_example(foo, cond) : foo { $$ = $1; }

    %if(cond) ; % % example : if_example(value, %true) ; Point 1 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

66. %rule if_example(foo, cond) : foo { $$ = $1; }

    %if(cond) ; % % example : if_example(value, %true) ; Point 1 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ここ < foo と cond は構文を分けたい

67. %rule if_example(foo, cond) : foo { $$ = $1; }

    %if(cond) ; % % example : if_example(value, %true) ; Point 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

68. %rule if_example(foo, cond) : foo { $$ = $1; }

    %if(cond) ; % % example : if_example(value, %true) ; Point 2 ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar ここ < 条件引数って “式” の一種なのでは?

69. A Few Moments Later… ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

70. %rule if_example(foo, cond: Conditional) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < OsakaRubyKaigi04を準備しつつ考えてました

71. %rule if_example(foo, cond: Conditional) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

72. %rule if_example(foo, cond: Conditional) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < 式なので、 { foo==bar }の様に書けると良い?

73. %rule if_example(foo, cond: Conditional) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < まだ少しなやんでいます。。。

74. %rule if_example(foo, cond: Boolean) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Other Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < これだと型に見える??

75. %rule if_example(foo, @cond) : foo { $$ = $1; }

    %if(cond) ; % % example : if_example(value, { %true }) ; Other Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < 接頭語をつける案。いい記号があればいいケド

76. %rule if_example(foo, cond: Conditional) : foo { $$ = $1;

    } %if(cond) ; % % example : if_example(value, { %true }) ; Proposal ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar < という訳でこれが最有力候補です。WDYT?

77. Further Sequel to a Dream ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar

78. Wish List ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar •Release v0.6.10 of Lrama. •Adding

    location information to the node structure. < がんばるか、 超がんばるかしかない!!!

79. end ydah | https://speakerdeck.com/ydah/the-sequel-to-a-dream-of-ruby-parsers-grammar