Skip to content

Manual chunks support - #2084

Merged
lukastaegert merged 4 commits into
masterfrom
manual-chunks
Mar 28, 2018
Merged

Manual chunks support#2084
lukastaegert merged 4 commits into
masterfrom
manual-chunks

Conversation

@guybedford

@guybedford guybedford commented Mar 22, 2018

Copy link
Copy Markdown
Contributor

This provides support for a new manualChunks input option, with the work based off of the chunk-naming branch (so that should land first, and I will ensure this is rebased to follow changes there prior to landing).

The usage is:

rollup.rollup({
  input: ['entryA', 'entryB'],
  manualChunks: {
    'common': ['babel'],
    'page1-common': ['./page1-dep.js'],
  }
})
  • Manual chunks alter execution in that any module listed in manual chunks is treated as an entry point from an execution perspective, that it can execute in isolation of any parent dependencies.
  • Manual chunks include all dependencies of the modules listed in manual chunks, unless that dependency is found in a previously-defined manual chunk. So the babel above will include all dependencies of Babel.
  • Manual chunks are ordered - the first chunk defined should be considered the base-level chunk. Each successive chunk will delegate any shared modules with previous chunks to those chunks. Eg if page1-dep.js happened to use a lodash module that is shared with Babel, then page1-dep would delegate to the common chunk.
  • Manual chunks will export any used exports just like any other chunk. They are not considered entry point chunks in that their exported names can be considered internal names.
  • If a manual chunk happens to match up to an entry module export, then it will be promoted to being treated as an entry point facade just like any other chunk. Similarly if its exports are tainted for that entry point it contains, then any entry point module will get a facade over the chunk.
  • The name of the manual chunk is treated as the chunk's alias, so that by default the above configuration will output a common-d8f87s9d.js and page1-common-s98ff7s9.js files. The filenames of manual chunks can be customized just like other chunks through the chunkNames option as in the chunk naming PR.

I think this might actually be a pretty good default... but let's experiment with some real world usage and see how it goes.

Since this is all under --experimentalCodeSplitting we can always change the approach if we find limitations in the wild.

Quite excited for this one :)

@guybedford
guybedford requested a review from lukastaegert March 22, 2018 13:56
@guybedford guybedford mentioned this pull request Mar 22, 2018
@guybedford

Copy link
Copy Markdown
Contributor Author

Also note that the name of the manual chunk is the "chunk alias". So that setting chunkNames: '[alias].js' etc allows controlling the exact output filename.

@guybedford
guybedford force-pushed the manual-chunks branch 3 times, most recently from 8b8429c to 67521f1 Compare March 24, 2018 19:30
@guybedford
guybedford force-pushed the chunk-naming branch 3 times, most recently from edd4931 to 2712e30 Compare March 24, 2018 22:24
@guybedford

Copy link
Copy Markdown
Contributor Author

Rebased and ready for review.

@guybedford
guybedford changed the base branch from chunk-naming to master March 26, 2018 06:40

@lukastaegert lukastaegert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just finished a first review. Definitely a very useful feature. Two things I noticed:

  • Currently, entry points and manual chunks cannot match (see my comment). Maybe this should be changed?
  • I find the new logic to be a little hard to read, also with the (ab)use of chunk aliases as flags. Maybe it is possible to extract some of this logic, especially from buildChunks, into separate functions?

Comment thread src/rollup/index.ts Outdated

export interface InputOptions {
input: string | string[] | { [entryAlias: string]: string };
manualChunks: string[] | { [chunkAlias: string]: string };

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like manualChunks could be an array as well. However when specifying something like manualChunks: ['file.js'] via config file, I get a type error. This should either be fixed (and tested) or removed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, will remove this. Yes manual chunks must be named (and it would be string[][] anyway if we did permit an array form).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Correct the object case to be string[] too!)

Comment thread src/Graph.ts Outdated
let entryAndManualChunkIds = entryModuleIds.concat([]);
if (manualChunks) {
Object.keys(manualChunks).forEach(name => {
const chunk = manualChunks[name];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chunk should probably be something better describing its content like ids or chunkIds.

Comment thread src/Graph.ts
entryModules,
!preserveModules
!preserveModules,
manualChunkModules

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the recent additions, the complex buildChunks has grown a lot more complex with lots of local variables with similar names and unclear scopes floating around. manualChunkModules could be avoided by extracting its generation into e.g. a helper function getManualChunkModules(manualChunks, entryAndManualChunkIds, entryAndChunkModules).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion

Comment thread src/Graph.ts
manualChunkModules
);

if (entryModuleAliases) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One important use case I see for this feature is to inline all modules imported by the "first" entry point in a base chunk so that initially, no secondary chunks need to be loaded. To specify this, one would probably write something like

{
  input: ['main1', 'main2'],
  manualChunks:{
    initial: ['main1']
  },
  ...
}

Currently, this will throw a "duplicate entry points" error since analyseExecution will already assign an alias to this chunk which is considered to be the relevant flag for this error. One solution would be to not list main1 in the input list but then the initial chunk can have exports that are not part of the main1 interface as the chunk is considered internal.

Ideally, I would assume we check for duplicate entry points only within input and manualChunks separately but allow a manual chunk to be the same as an input entry point. In case those match up and no additional facade is necessary, I would further expect the resulting chunk to receive its name via input (otherwise, the non-facade chunk should probably get the manual chunk name).

As this will probably make the existing logic even more complex, maybe there is a way to extract more of this chunking logic out of buildChunks into separate helpers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've brought up an interesting case, and this is really good to poke holes in the model. The difficulty with supporting this case is there are a few questions in relating it to the aliasing model:

  • Should the chunk alias for main1 be called main1 or initial?
  • Should the chunk name for main use the chunkNames rule or the entryNames rule? Since it is defined as both a manual chunk and an entry, we don't currently have a way to deal with this.

Note from a chunking point of view (naming aside), this scenario is equivalent to just having:

{
  input: ['main2'],
  manualChunks:{
    initial: ['main1']
  },
  ...
}

One alternative might be to have the manual chunk called initial get created, and then have the input become simply a reexporting facade module pointing to the manual chunk? That might be a way around the naming model issues here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact the above would provide a general solution to the duplicate entry error, allowing us to have a way of always dealing with this case, removing the error entirely.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To summarize the approach taken in the latest commit:

Should the chunk alias for main1 be called main1 or initial?

The manual chunk is created, and then seen to be an entry module facade - therefore it takes the place of main1, and gets called main1. If the chunk had had other modules in it then a new reexporting facade would have been used to the chunk.

Should the chunk name for main use the chunkNames rule or the entryNames rule? Since it is defined as both a manual chunk and an entry, we don't currently have a way to deal with this.

When a chunk corresponds to an entry point facacde as in this case, the manual chunk name does not apply at all, with the entry naming scheme precedence.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will still require two separate downloads but I guess this is certainly the cleanest approach 👍

@guybedford

guybedford commented Mar 27, 2018

Copy link
Copy Markdown
Contributor Author

I know the abuse of chunk aliases as flags is bad for readability, but that's why I carefully commented and I just hate adding a new property to Module when it can be avoided... memory footprint matters on objects like Module that are instantiated a lot... I know it's not much of a difference but I still prefer not to add a new property when possible. If you think it would be clearer I can add a redundant boolean flag though.

I've added changes with the feedback and to remove the duplicate entry point error in the chunk scenario, replacing this with facade creation, while retaining the duplicate entry point error for actual duplicate entry points.

@lukastaegert lukastaegert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good now! I also very much like the split of buildChunks 👍
Will merge this now.

@lukastaegert
lukastaegert merged commit 4a0ba2d into master Mar 28, 2018
@lukastaegert
lukastaegert deleted the manual-chunks branch March 28, 2018 06:17
@lukastaegert lukastaegert added this to the 0.58.0 milestone Mar 29, 2018
@derrickb

Copy link
Copy Markdown

This looks great! Is there a complete worked example on how to use this for generating a vendor.js?

@lukastaegert

Copy link
Copy Markdown
Member

Unfortunately not yet to my knowledge. This is a config that should be working (also using the new object form for input):

rollup.rollup({
  input: {
    index: './main.js'
  },
  experimentalCodeSplitting: true,
  manualChunks: {
    'vendor': ['react', 'some-other-big-dependency']
  },
  output: {
    dir: 'dist',
    format: 'es'
  }
})

Basically you list all dependencies you want in your vendor bundle in a comma separated list. Note that order matters here i.e. if one of your dependencies creates a global that another dependency relies on, it should go first in the manual chunk. I.e., manual chunks will also slightly change the execution order.

@caub

caub commented Jun 3, 2018

Copy link
Copy Markdown

I tried using this new option here

I also npm link'ed rollup/rollup-plugin-commonjs#322, else it fails sooner

It fails with:

$ NODE_ENV=rollup npx rollup -c rollup.config.demo.chunks.js

demo/index.js → demo/dist...
[!] Error: Could not resolve entry (react)
Error: Could not resolve entry (react)
    at error (/home/caub/dev/color-wheel/node_modules/rollup/dist/rollup.js:199:15)
    at /home/caub/dev/color-wheel/node_modules/rollup/dist/rollup.js:20402:17
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:746:11)
    at findNodeScript.then.existing (/usr/lib/node_modules/npm/node_modules/libnpx/index.js:268:14)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants