Improve ux for rust users #909

kdy1 · 2020-07-28T10:15:24Z

Comments refinement

Comments is now trait and swc_common only contains single-threaded Comments impl.
This allow dropping dependency on dashmap.

Multi-threaded implementation of Comments will live in the swc crate.

Span testing

It's bit messy, but span is rendered with code.

Closes #583.

No duplicated comments

Lexer now insert comments only if start of it is after last comment. This variable is shared, so even when the parser decides to backtrack, comments are not duplicated.

Closes #856.

Span, `GLOBALS` issue

This is about making swc_common::GLOBALS not required whiling parsing.

Benachmark:

Parser benchmark with compressed span

test lexer::tests::lex_colors_js ... bench: 29,862 ns/iter (+/- 677) = 38 MB/s
test lexer::tests::lex_colors_ts ... bench: 30,328 ns/iter (+/- 4,249) = 38 MB/s
test lexer::tests::lex_dec_lit ... bench: 39,349 ns/iter (+/- 3,514) = 13 MB/s
test lexer::tests::lex_escaped_char ... bench: 9,008 ns/iter (+/- 2,104) = 25 MB/s
test lexer::tests::lex_ident ... bench: 34,146 ns/iter (+/- 905) = 16 MB/s
test lexer::tests::lex_large_number ... bench: 13,607 ns/iter (+/- 2,067) = 16 MB/s
test lexer::tests::lex_legact_octal_lit ... bench: 65,269 ns/iter (+/- 6,505) = 9 MB/s
test lexer::tests::lex_long_ident ... bench: 6,393 ns/iter (+/- 427) = 74 MB/s
test lexer::tests::lex_regex ... bench: 11,146 ns/iter (+/- 837) = 13 MB/s
test lexer::tests::lex_semicolons ... bench: 164,109 ns/iter (+/- 11,798) = 11 MB/s
test parser::expr::tests::bench_member_expr_es ... bench: 4,944 ns/iter (+/- 415) = 2 MB/s
test parser::expr::tests::bench_member_expr_ts ... bench: 5,199 ns/iter (+/- 545) = 2 MB/s
test parser::expr::tests::bench_new_expr_es ... bench: 2,665 ns/iter (+/- 583) = 3 MB/s
test parser::expr::tests::bench_new_expr_ts ... bench: 2,876 ns/iter (+/- 391) = 3 MB/s

test result: ok. 0 passed; 0 failed; 137 ignored; 14 measured; 0 filtered out

 Running /Users/kdy1/projects/master-swc/target/release/deps/lexer-a67dc5a3727da1b3

running 8 tests
test angular ... bench: 12,073,907 ns/iter (+/- 223,565) = 59 MB/s
test backbone ... bench: 1,516,604 ns/iter (+/- 73,519) = 39 MB/s
test colors ... bench: 25,818 ns/iter (+/- 2,599) = 44 MB/s
test jquery ... bench: 8,338,700 ns/iter (+/- 186,822) = 32 MB/s
test jquery_mobile ... bench: 13,596,096 ns/iter (+/- 311,483) = 33 MB/s
test mootools ... bench: 6,370,238 ns/iter (+/- 204,655) = 25 MB/s
test underscore ... bench: 1,248,574 ns/iter (+/- 22,867) = 34 MB/s
test yui ... bench: 6,964,947 ns/iter (+/- 111,208) = 48 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 8 measured; 0 filtered out

 Running /Users/kdy1/projects/master-swc/target/release/deps/parser-a03b4a2703c51f27

running 17 tests
test angular ... bench: 31,571,375 ns/iter (+/- 929,317) = 22 MB/s
test angular_ts ... bench: 33,139,153 ns/iter (+/- 1,329,377) = 21 MB/s
test backbone ... bench: 4,143,316 ns/iter (+/- 121,542) = 14 MB/s
test backbone_ts ... bench: 4,321,886 ns/iter (+/- 191,845) = 13 MB/s
test colors ... bench: 56,804 ns/iter (+/- 63,156) = 20 MB/s
test colors_ts ... bench: 58,379 ns/iter (+/- 36,899) = 19 MB/s
test jquery ... bench: 24,677,749 ns/iter (+/- 931,515) = 10 MB/s
test jquery_mobile ... bench: 40,658,794 ns/iter (+/- 1,473,480) = 11 MB/s
test jquery_mobile_ts ... bench: 41,106,544 ns/iter (+/- 1,307,308) = 11 MB/s
test jquery_ts ... bench: 25,117,499 ns/iter (+/- 508,530) = 10 MB/s
test large ... bench: 733,485 ns/iter (+/- 30,160) = 6 MB/s
test mootools ... bench: 19,319,509 ns/iter (+/- 473,669) = 8 MB/s
test mootools_ts ... bench: 19,977,002 ns/iter (+/- 627,177) = 8 MB/s
test underscore ... bench: 3,497,250 ns/iter (+/- 110,568) = 12 MB/s
test underscore_ts ... bench: 3,704,073 ns/iter (+/- 126,591) = 11 MB/s
test yui ... bench: 18,210,932 ns/iter (+/- 667,750) = 18 MB/s
test yui_ts ... bench: 19,198,935 ns/iter (+/- 403,819) = 17 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 17 measured; 0 filtered out

Parser benchmark without span compression

test lexer::tests::lex_colors_js ... bench: 29,885 ns/iter (+/- 2,840) = 38 MB/s
test lexer::tests::lex_colors_ts ... bench: 30,181 ns/iter (+/- 3,857) = 38 MB/s
test lexer::tests::lex_dec_lit ... bench: 38,751 ns/iter (+/- 1,476) = 13 MB/s
test lexer::tests::lex_escaped_char ... bench: 9,036 ns/iter (+/- 1,625) = 25 MB/s
test lexer::tests::lex_ident ... bench: 34,185 ns/iter (+/- 4,779) = 16 MB/s
test lexer::tests::lex_large_number ... bench: 13,465 ns/iter (+/- 991) = 16 MB/s
test lexer::tests::lex_legact_octal_lit ... bench: 64,013 ns/iter (+/- 10,854) = 9 MB/s
test lexer::tests::lex_long_ident ... bench: 6,381 ns/iter (+/- 1,075) = 74 MB/s
test lexer::tests::lex_regex ... bench: 11,151 ns/iter (+/- 2,097) = 13 MB/s
test lexer::tests::lex_semicolons ... bench: 162,000 ns/iter (+/- 14,307) = 11 MB/s
test parser::expr::tests::bench_member_expr_es ... bench: 4,942 ns/iter (+/- 223) = 2 MB/s
test parser::expr::tests::bench_member_expr_ts ... bench: 5,115 ns/iter (+/- 663) = 2 MB/s
test parser::expr::tests::bench_new_expr_es ... bench: 2,562 ns/iter (+/- 263) = 3 MB/s
test parser::expr::tests::bench_new_expr_ts ... bench: 2,822 ns/iter (+/- 378) = 3 MB/s

test result: ok. 0 passed; 0 failed; 137 ignored; 14 measured; 0 filtered out

 Running /Users/kdy1/projects/swc/target/release/deps/lexer-91bd039baebdb422

running 8 tests
test angular ... bench: 11,844,198 ns/iter (+/- 277,584) = 60 MB/s
test backbone ... bench: 1,495,697 ns/iter (+/- 97,404) = 40 MB/s
test colors ... bench: 25,441 ns/iter (+/- 2,405) = 45 MB/s
test jquery ... bench: 8,276,801 ns/iter (+/- 197,501) = 32 MB/s
test jquery_mobile ... bench: 13,395,942 ns/iter (+/- 320,989) = 33 MB/s
test mootools ... bench: 6,323,569 ns/iter (+/- 184,925) = 25 MB/s
test underscore ... bench: 1,236,126 ns/iter (+/- 83,913) = 35 MB/s
test yui ... bench: 6,843,345 ns/iter (+/- 134,186) = 49 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 8 measured; 0 filtered out

 Running /Users/kdy1/projects/swc/target/release/deps/parser-3205c5afd12a24fc

running 17 tests
test angular ... bench: 31,115,052 ns/iter (+/- 1,265,163) = 23 MB/s
test angular_ts ... bench: 32,343,882 ns/iter (+/- 862,398) = 22 MB/s
test backbone ... bench: 4,069,784 ns/iter (+/- 322,482) = 14 MB/s
test backbone_ts ... bench: 4,242,501 ns/iter (+/- 592,016) = 14 MB/s
test colors ... bench: 51,999 ns/iter (+/- 12,837) = 22 MB/s
test colors_ts ... bench: 53,359 ns/iter (+/- 5,282) = 21 MB/s
test jquery ... bench: 24,954,176 ns/iter (+/- 1,207,941) = 10 MB/s
test jquery_mobile ... bench: 41,242,172 ns/iter (+/- 833,360) = 10 MB/s
test jquery_mobile_ts ... bench: 41,527,312 ns/iter (+/- 935,822) = 10 MB/s
test jquery_ts ... bench: 24,565,976 ns/iter (+/- 664,778) = 10 MB/s
test large ... bench: 709,609 ns/iter (+/- 207,575) = 7 MB/s
test mootools ... bench: 18,688,197 ns/iter (+/- 3,951,869) = 8 MB/s
test mootools_ts ... bench: 19,808,024 ns/iter (+/- 873,979) = 8 MB/s
test underscore ... bench: 3,398,891 ns/iter (+/- 255,715) = 12 MB/s
test underscore_ts ... bench: 3,632,732 ns/iter (+/- 183,886) = 11 MB/s
test yui ... bench: 17,540,363 ns/iter (+/- 683,704) = 19 MB/s
test yui_ts ... bench: 18,607,831 ns/iter (+/- 1,143,203) = 18 MB/s

test result: ok. 0 passed; 0 failed; 0 ignored; 17 measured; 0 filtered out

Less dependencies

swc_common

Cargo feature tty-emiiter (To remove tty related stuffs )
Cargo feature sourcemap (To remove sourcemap for web assets)
Removed dependency on dashmap

swc_ecma_parser:

Removed dependency on once_cell and regex

swc_ecma_utils:

Removed dependency on parser

testing:

Remove dependency on relative_path

swc:

Remove dependency on derive_more and path-clean

swc_ecmascript

It contains ast, codegen, parser, utils, visit.

bartlomieju · 2020-07-30T09:16:11Z

common/src/comments.rs

+    fn take_leading(&self, pos: BytePos) -> Option<Vec<Comment>> {
+        (**self).take_leading(pos)
+    }


Would it be possible to have get_leading(&self, pos: BytePos) -> Option<&[&Comment]> and respective method for trailing ones?

I expect it to require another complex generic traits.

But api like

fn with_leading<F, Ret>(&self, op: F) -> Ret where F: FnOnce(&[Comment]) -> Ret;

is possible, regardless of threadedness

Great, that would work for my use case 👍

kdy1 · 2020-07-31T09:06:15Z

cc @ry @bartlomieju as I'm almost done.

I'll add more span tests via other prs.

This will help preventing stack overflow

Closes swc-project#583

95th · 2020-08-16T03:15:11Z

common/src/comments.rs

+/// [&Comments] to the parser. Thirdly, `(&self)` allows multi-threaded
+/// use-cases such as swc itself.
+///
+/// We use [Option] instead of no-op Comments implementation to avoid allocation


AFAIK Empty vec doesn't allocate.

You are correct, but to avoid allocation in parser, you need to determine if comment parser is enabled.

It can be done by passing comment map whilest passing a flag to parser, but Option<&dyn Comments> can be used as a comment map combined with a flag.

kdy1 marked this pull request as draft July 28, 2020 10:35

kdy1 force-pushed the crate-ux branch 3 times, most recently from 9ce0162 to b629cda Compare July 29, 2020 05:59

kdy1 self-assigned this Jul 29, 2020

kdy1 added this to the v1.2.10 milestone Jul 29, 2020

kdy1 changed the title ~~UX for rust users~~ Improve ux for rust users Jul 29, 2020

kdy1 modified the milestones: v1.2.10, v1.2.11 Jul 29, 2020

bartlomieju reviewed Jul 30, 2020

View reviewed changes

kdy1 force-pushed the crate-ux branch from 127c89f to 985791d Compare July 31, 2020 09:14

kdy1 added 16 commits July 31, 2020 18:20

Add some cargo features to swc_common

8c5cf65

Removing parking_lot is complex

3dd6478

stable

2440c7f

Fix

b067749

Stable rustc

bd0803f

Fix testing utilities

75800d6

parser: Drop dependency on once_cell and regex

cd26799

codegen: Cargo feature

cf9b8d8

swc_ecma_utils: Remove dependency on parser

53be8e2

preset_env: Remove dependency on walkdir

e2c54ed

testing: Remove dependency on relative_path

c0aefda

swc: Remove dependency on derive_more and path-clean

778176c

typo

facdc2a

pluralize

4666dee

walkdir is used

d350003

SourceFileInput -> StringInput (with documentation)

29647e8

kdy1 added 24 commits July 31, 2020 18:20

Fix for type modification of Lexer.leading_comments_buf

4039372

I found better way than trakcing backtracing state

17a946a

parser: Fix lints

0b6ff72

Make parser error even smaller (8 byte)

308ea6a

This will help preventing stack overflow

Fix input

afa526d

Fix duplicated comments

d0d9279

Fix compilation

bcd4c60

Fix

4ea4ca5

Add span tests.

1f6426b

Closes swc-project#583

Bump version

304b420

Do not test .spans file

c56858f

More span tests

39f00ae

Fix spans test

42da76e

More span test

7b49033

More span test

c6fc0a2

Make Comments trait

96a66ba

Restore swc

ec4581f

Add some methods to SingleThreadComments

452daff

Comments impl for the swc itself

e4b5082

WIP: Fixing lifetime issue

be29a5f

Fix compilation

61186b2

swc_ecmascript as a crate

9931521

cargo features for swc_ecmascript

9c7e404

Bump versions

05358c7

kdy1 force-pushed the crate-ux branch from 985791d to 05358c7 Compare July 31, 2020 09:20

kdy1 marked this pull request as ready for review July 31, 2020 09:48

kdy1 merged commit 0ac55ae into swc-project:master Jul 31, 2020

kdy1 deleted the crate-ux branch July 31, 2020 09:49

95th reviewed Aug 16, 2020

View reviewed changes

swc-project locked as resolved and limited conversation to collaborators Nov 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Improve ux for rust users #909

Improve ux for rust users #909

Uh oh!

kdy1 commented Jul 28, 2020 •

edited

Loading

Uh oh!

bartlomieju Jul 30, 2020

Uh oh!

kdy1 Jul 30, 2020

Uh oh!

bartlomieju Jul 30, 2020

Uh oh!

kdy1 commented Jul 31, 2020 •

edited

Loading

Uh oh!

95th Aug 16, 2020

Uh oh!

kdy1 Aug 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Uh oh!

Improve ux for rust users #909

Improve ux for rust users #909

Uh oh!

Conversation

kdy1 commented Jul 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments refinement

Span testing

No duplicated comments

Span, GLOBALS issue

Less dependencies

swc_ecmascript

Uh oh!

bartlomieju Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

kdy1 Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

bartlomieju Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

kdy1 commented Jul 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

95th Aug 16, 2020

Choose a reason for hiding this comment

Uh oh!

kdy1 Aug 16, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

kdy1 commented Jul 28, 2020 •

edited

Loading

Span, `GLOBALS` issue

kdy1 commented Jul 31, 2020 •

edited

Loading