Add option and scripts to display and compare performance timings by lukastaegert · Pull Request #2045 · rollup/rollup

lukastaegert · 2018-03-11T11:52:37Z

With the growing need to get reliable performance metrics, this PR aims to provide a comprehensive solution.

The JavaScript API receives a new perf: true/false option. When this option is supplied, performance information is gathered and the bundle receives a new getTimings function that can be used to query all accumulated timings up to a certain point
If perf: true is used, in addition to timing rollup's own code, timing information will also gathered for most plugin hooks individually by plugin.
If the option is not supplied, all timing functions present in the code will be replaced with empty functions.
If perf: true is used via the config file interface (or --perf via the CLI), gathered timings will be displayed after the run. Timings are identified via labels that use a markdown-like syntax to receive some formatting.
A new perf:init script has been added to the package.json file. This script receives a <Github username>/<repo name> or <Github username>/<repo name>#<branch> string as parameter to identify a repo that will be checked out into the /perf folder. The repo needs to contain a rollup.config.js file at the top level (you can use e.g. rollup/rollup) and can be used as a base for performance measurements.
A new perf script has been added to the package.json file. This script will use the current version of rollup in /dist and use it via the JavaScript API to bundle the code in the /perf folder. If there are several configurations present, only the first input and output configuration is used.
If the script is run for the first time, a new file /perf/rollup.perf.json is created to store the current measurements. If this file is already present, it will not be overwritten. Instead, deviations from the previous run will be displayed.
As measurements can be very volatile, "npm run perf" by default bundles the repo 6 times and discards for each timing label the 3 largest results. The rest will then be averaged. You can control this behaviour by e.g. running "npm run perf 8 2" to do 8 runs with 2 discarded results instead.
To get even better code-level performance information, "npm run perf:debug" will instead run rollup in e.g. Chrome devtools.

Otherwise I think the best way to get a feeling for this, try it our yourself! Run

npm run perf:init rollup/rollup
npm run perf
npm run perf

and see what happens.

guybedford

Amazing work - this will be an awesome base to work on performance further!

guybedford · 2018-03-12T09:48:48Z

+if (process.argv.length !== 3 || !VALID_REPO.test(repo)) {
+	console.error(
+		'You need to provide a GitHub repo in the form <username>/<repo>, e.g. ' +
+			'"npm run perf:init rollup/rollup"'


Just thought I'd mention - it could be useful to support npm run perf:init rollup/rollup#branch-name where the branch becomes -b ${branchName} in the git clone when provided.

Good idea! Even though the intention is to test is your current installation of rollup and not what you install in the perf folder, this feature could also be used to easily work on and debug rollup problems with branches in repos.

guybedford · 2018-03-12T09:49:41Z

+	.catch(error => {
+		console.error(error);
+		process.exit(1);
+	});


This context could be a good excuse to use async/await...

guybedford · 2018-03-12T09:51:11Z


+const TIME_RENDER_MODULES = '- render modules';
+const TIME_RENDER_FORMAT = '- render format';
+const TIME_SOURCEMAP = '- sourcemap';


Perhaps have the names without the - and add this in the timers utility code rather?

guybedford · 2018-03-12T09:54:37Z

-
-		if (optionError) inputOptions.onwarn({ message: optionError, code: 'UNKNOWN_OPTION' });
+const TIME_BUILD = '# BUILD';
+const TIME_GENERATE = '# GENERATE';


Ah I see you included the markers to distinguish hierarchy at this level.

Perhaps timeStart can take a second argument representing the heading number?

guybedford · 2018-03-12T10:01:53Z

+	await buildAndGetTimings(config);
+	console.info('Completed initial run (Discarded).');
+	const timings = await buildAndGetTimings(config);
+	console.info('Completed run 1.');


It could be useful to have some basic timing information as these show for quick perf testing when one doesn't want to wait for all the runs to complete (think eg ping).

guybedford · 2018-03-12T10:04:56Z

I've been testing this out and there still appears to be quite a bit of variation in the timings.

Perhaps run a manual GC call between runs to help avoid GC randomness?

This can be done with node --expose-gc wrapper for the main execution thread and a gc() call between runs.

lukastaegert · 2018-03-13T07:06:52Z

@guybedford I have implemented all suggestions and did some further changes:

The npm script will now garbage collect before each run
Instead of discarding outliers, we are discarding the largest results each
While gathering timings, current top level results are displayed after each run (already doing some of the discarding/averaging)

Maybe you want to have another look so that we can merge this soon.

guybedford · 2018-03-13T12:30:02Z

Looks great to me, please do go ahead with the merge.

* Extract some constants * Improve scripts

* Display current (averaged) results after each run

mourner · 2018-03-14T11:40:31Z

@lukastaegert trying this out on mapbox/mapbox-gl-js, I'm getting:

(node:70891) UnhandledPromiseRejectionWarning: Error: Could not resolve entry (src/index.js)
    at error (/Users/mourner/projects/rollup/dist/rollup.js:168:15)
    at /Users/mourner/projects/rollup/dist/rollup.js:21174:17
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
    at Function.Module.runMain (module.js:695:11)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
(node:70891) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:70891) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Any clues?

mourner · 2018-03-14T11:42:50Z

Nevermind, forgot to npm install after pulling in the changes.

lukastaegert mentioned this pull request Mar 12, 2018

perf: cache file stats/reads rollup/rollup-plugin-node-resolve#126

Merged

lukastaegert force-pushed the enable-perf-display branch 2 times, most recently from b62a209 to a60a21c Compare March 12, 2018 08:00

guybedford approved these changes Mar 12, 2018

View reviewed changes

guybedford reviewed Mar 12, 2018

View reviewed changes

lukastaegert mentioned this pull request Mar 13, 2018

Hidden classes and Rollup performance #2050

Closed

lukastaegert added this to the 0.57.0 milestone Mar 14, 2018

lukastaegert added 10 commits March 14, 2018 06:10

Sort options alphabetically (also sorts them in the output)

58c13d2

Implement new --perf option

046113f

Add local perf scripts.

626cc71

Add timing wrappers to plugin hooks

9f6a741

* Automatically discard outliers

b2d6470

* Extract some constants * Improve scripts

Add tests and code-splitting support

09031b3

Run garbage collector between performance runs

3ce12cf

* Always discard the largest results

eae635b

* Display current (averaged) results after each run

Use abstract levels for timer labels

53dbe5d

Round numbers when using CLI interface

bbd8289

lukastaegert force-pushed the enable-perf-display branch from 17fcc07 to bbd8289 Compare March 14, 2018 05:13

lukastaegert merged commit bbd8289 into master Mar 14, 2018

lukastaegert deleted the enable-perf-display branch March 14, 2018 05:16

lukastaegert added the Fixed pending release label Mar 14, 2018

lukastaegert removed the Fixed pending release label Mar 15, 2018

This was referenced Mar 15, 2018

Update rollup requirement to ^0.57.0 stephenreynolds/screeps#74

Closed

Bump rollup from 0.56.5 to 0.57.0 verkholantsev/superoverload#32

Merged

Bump rollup from 0.56.5 to 0.57.0 EHDFE/scms-modules#25

Closed

Navigation Menu

Uh oh!

Uh oh!

Add option and scripts to display and compare performance timings - #2045

Add option and scripts to display and compare performance timings#2045
lukastaegert merged 10 commits into
masterfrom
enable-perf-display

lukastaegert commented Mar 11, 2018 •

edited

Loading

Uh oh!

guybedford left a comment

Uh oh!

guybedford Mar 12, 2018

Uh oh!

lukastaegert Mar 12, 2018

Uh oh!

guybedford Mar 12, 2018

Uh oh!

guybedford Mar 12, 2018

Uh oh!

guybedford Mar 12, 2018

Uh oh!

guybedford Mar 12, 2018 •

edited

Loading

Uh oh!

guybedford commented Mar 12, 2018

Uh oh!

lukastaegert commented Mar 13, 2018

Uh oh!

guybedford commented Mar 13, 2018

Uh oh!

mourner commented Mar 14, 2018

Uh oh!

mourner commented Mar 14, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

lukastaegert commented Mar 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guybedford left a comment

Choose a reason for hiding this comment

Uh oh!

guybedford Mar 12, 2018

Choose a reason for hiding this comment

Uh oh!

lukastaegert Mar 12, 2018

Choose a reason for hiding this comment

Uh oh!

guybedford Mar 12, 2018

Choose a reason for hiding this comment

Uh oh!

guybedford Mar 12, 2018

Choose a reason for hiding this comment

Uh oh!

guybedford Mar 12, 2018

Choose a reason for hiding this comment

Uh oh!

guybedford Mar 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guybedford commented Mar 12, 2018

Uh oh!

lukastaegert commented Mar 13, 2018

Uh oh!

guybedford commented Mar 13, 2018

Uh oh!

mourner commented Mar 14, 2018

Uh oh!

mourner commented Mar 14, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukastaegert commented Mar 11, 2018 •

edited

Loading

guybedford Mar 12, 2018 •

edited

Loading