handle missing exports in chunked mode - #2118
Conversation
guybedford
left a comment
There was a problem hiding this comment.
As mentioned, this is not about outcome, but ensuring that the exports logic around shimming can entirely be isolated to the src/Module.ts file, so that these exception cases don't spill into other logic which could become a refactoring hazard.
| const module = this.exports.get(variable); | ||
|
|
||
| // skip shimmed exports | ||
| if (!module) continue; |
There was a problem hiding this comment.
The plan in going through with this approach was to be able to avoid having to have special cases like this - otherwise all logic around exports needs to be considered against this case which may lead to bugs.
I really do think we should ensure the shimmed export otherwise behaves exactly like any other, with module being defined here.
There was a problem hiding this comment.
Ok I'll take another stab at it.
There was a problem hiding this comment.
Thanks, will be a great help to ensure the assumptions are minimal.
| { | ||
| missingExport() { | ||
| return true; | ||
| } |
There was a problem hiding this comment.
Rather than have the missingExport hook provide this functionality, how about making this a flag option - shimMissingExports: true?
There was a problem hiding this comment.
That is already the way to stop the default behavior of hard erroring (in non-code-splitting mode). Are you suggesting checking here for shimMissingExports and conditionally not erroring?
There was a problem hiding this comment.
Yes exactly, so that this can be done without a plugin.
a35282d to
57b27a5
Compare
3843f25 to
ffd1fd5
Compare
|
Ready for review again. It turned out pretty clean IMO. While it is more encapsulated, I had to add a little bit more compatibility code to accommodate some bad assumptions, like |
| this.exports[name] = { | ||
| localName: name, | ||
| shim | ||
| }; |
There was a problem hiding this comment.
Would it be possible for ShimVariable to inject itself into originals here on the module itself so that variable lookups play out correctly? Or does that introduce too many cross-cutting issues?
There was a problem hiding this comment.
I thought about it, but I couldn't figure out how to get access to the NamespaceVariable.originals from earlier in the traceExport call stack at this point of shimming.
There was a problem hiding this comment.
I think NamespaceVariables are created after the initial tracing in module right? So that exports should already be shimmed by then and then the getExports() call from the namespace constructor should include the shimmed names?
Let me know if there are any other issues - I know it's a bit to investigate.
There was a problem hiding this comment.
Let me know if you have further questions on this one.
There was a problem hiding this comment.
I was able to remove the NamespaceVariable.originals workaround completely. Something I changed must have made it obsolete.
| name: string, | ||
| _options?: { | ||
| isExportAllSearch?: boolean; | ||
| } |
There was a problem hiding this comment.
Could we make this a default flag over an object rather?
There was a problem hiding this comment.
I figured module.traceExport(name, { isExportAllSearch: true }); would be more clear to the reader than, module.traceExport(name, true);, but I can change it if you want.
There was a problem hiding this comment.
That would be great just to avoid the object allocation, for what it's worth. It doesn't seem like the flags will be extending anytime soon.
| } else { | ||
| exportBlock.push(`export const ${specifier.exported} = null;`); | ||
| } | ||
| } else if (specifier.exported === 'default') { |
There was a problem hiding this comment.
I wonder if we could shift these types of logic out of the finalisers by rendering a line like:
export const $$shim = null;as part of the core rendering process, then setting the specifier.local value to $$shim.
There was a problem hiding this comment.
I'm not sure I follow. Wouldn't it get really complicated if there were multiple missing exports for a given modules? Like:
import { missing1, missing2 } from './foo';
missing1('bar');
missing2('baz');There was a problem hiding this comment.
Sorry I should have clarified rendering just a ${varOrConst} $$shim = null;, so that in foo itself the output would be:
const $$shim = null;
export { $$shim as missing1, $$shim as missing2 }Let me know if that makes sense?
There was a problem hiding this comment.
It does now. But aren't you worried about adding a var after the deconflicting step? Any name we choose, like $$shim, would possibly conflict with an existing $$shim var.
There was a problem hiding this comment.
I made the change, but the conflict issue still worries me.
There was a problem hiding this comment.
$$shim can be added to the list of names to deshadow in the setIdentifierResolutions function of Chunk to ensure proper handling of this.
There was a problem hiding this comment.
I'm now running into the problem where my output is
const $$shim$1 = null;
var $$shim = null;
export { $$shim$1 as $$shim, $$shim };This is a dependency and an entry point. My existing unused $$shim was renamed to $$shim$1 because of the deshadowing, but is still exported as $$shim because of entry point. Now this conflicts with my shimmed export $$shim. It's almost like we need to reverse the renaming by leaving existing $$shims as is, and instead do something like chunk.generateUniqueVarAndExportName() for the shimmed export name. Am I on the right track?
There was a problem hiding this comment.
I got something working.
|
Looking good! Thanks for understanding we just want to keep assumptions in unrelated places to a minimum in the codebase. |
ac832ff to
e669b33
Compare
| const exportDeclaration: string[] = []; | ||
| exports.forEach(specifier => { | ||
| if (specifier.shim) { | ||
| exportBlock.push(`const ${specifier.local} = null;`); |
There was a problem hiding this comment.
This can be done in the preRender of Chunk.ts.
There was a problem hiding this comment.
Also the const should be a ${varOrConst} reference.
There was a problem hiding this comment.
The preRender looks a little foreign to me. Would you think it goes in prepend, append, or something else? How do I get the var before the exports block, but after any module wrappers so that it's not in global scope?
There was a problem hiding this comment.
Just like at
Line 502 in 79ad616
preRender renders the main body source, while render renders the finalizers. I think preRender would be a good place. Note that there is a hoistedSource for bindings that must be available early this could be added to at
Line 778 in 79ad616
if (shimExports) hoistedSource += ``${varOrConst} $$shim = null``;). Note that finalizers will apply wrappers so that there are no global leaks to worry about.
There was a problem hiding this comment.
If I do this, it works for everything except systemjs, where the output is
System.register([], function (exports, module) {
'use strict';
return {
execute: function () {
exports('$$shim$2', $$shim$2);
var $$shim$2 = null;
const $$shim = exports('$$shim$1', null);
}
};
});I may have the change the order of functionExports in systemjs to get it to work this way.
There was a problem hiding this comment.
made the change.
c349eb1 to
4c1b81d
Compare
|
This branch merged with #2277 is what I've been testing. It is looking good so far. Nothing that can't be hotfixed later on, so I would say it's good. |
|
In that case @lukastaegert it would be great if you could offer a final review here. |
|
Sure thing. It's a little late today for a full review today but I added it to the top of the review queue of the 1.0.0 project and will get to it first thing tomorrow |
lukastaegert
left a comment
There was a problem hiding this comment.
Hi, I think we are already quite close but the deconflicting is still faulty unfortunately and I do not know yet what a proper solution would be. Apparently if there already exists a local variable named '_shimmedExport', the shimmed export will be wrongly associated with this variable instead of the global shimmed export variable. Also this produces faulty code if there already were shimmed exports which were actually used.
| export { default, default as d3f1, d3f2 } from './d3'; | ||
| export * from './d3'; | ||
|
|
||
| export const _shimmedExport$2 = null; No newline at end of file |
There was a problem hiding this comment.
This test is a little hard to debug as it tries to do everything in one test while at the same time forgetting some important combinatorics that actually fail. I would suggest to split this up into several more focused tests.
- A golden path test showcasing how this feature actually works without name conflicts (helpful for documentation purposes). I think a single import file should be enough here 😉
- One or even more separate test(s) focusing explicitly on name conflicts. The most important situation that should be covered is that output produced by this feature is feeded back into rollup but there is more. What is actually not working correctly and needs to be fixed is the following (which pretends to be a set of shimmed exports but apparently someone has edited it manually 😜):
// main.js
import { missing, _shimmedExport, _shimmedExport$1 } from './dep.js';
_shimmedExport() // note that we are actually *using* some of the shimmed exports
_shimmedExport$1()
missing();
// dep.js
// added a function with a side-effect here
const _shimmedExport = () => console.log('effect');
export { _shimmedExport, _shimmedExport as _shimmedExport$1 };
// output - main.js
import { _shimmedExport as _shimmedExport$1 } from './d1.js';
_shimmedExport$1();
_shimmedExport$1();
_shimmedExport(); // now hello, who do you belong to!? Isn't this the missing one?
// output - dep.js
const _shimmedExport$1 = () => console.log('effect');
// so apparently there *are* several exports here but they are all wired up the same
export { _shimmedExport$1 as _shimmedExport, _shimmedExport$1, _shimmedExport$1 as missing };There was a problem hiding this comment.
If the combinatorics are checked for the non-compact case, a single compact mode test should suffice, though.
| this.exports[name] = { | ||
| localName: '_shimmedExport' | ||
| }; | ||
| return this.graph.scope.findVariable('_shimmedExport'); |
There was a problem hiding this comment.
Instead of searching we might just use this.graph.exportShimVariable here which would also be a little clearer.
|
I can update the tests to be more clear. As for the rest of the changes, maybe @guybedford has some ideas? |
|
@kellyselden that would be great if you can make the test adjustments. Will aim to look into the output fixes over the weekend. |
|
I actually started looking into this myself and will continue later this afternoon. Will keep you posted if I come up with a good solution. |
|
@lukastaegert let me know if you're still working on this, otherwise I can look into it then as well. |
|
Not at the moment, got a little side-tracked by the release issues. My current impression so far is that we should not rely on name-matching to detect if an export is shimmed but maybe via a flag added to the export description. Furthermore, we should not use But I do not have any results yet and will not find any time to look into this further before tomorrow so please go ahead if you have time today. |
|
@lukastaegert it appears you're right there's no clean way to do this at the variable level due to the fact that we dedupe based on name, not variable name (two identifiers of the same name in the same scope cant reference different underlying variables in the deshadowing logic) Given this, the approach @kellyselden originally took would work here. My major concern for this feature from the start was to find an approach that would use the existing tracing logic and not pollute it with conditional exceptions. This is non-spec behaviour, and I'm not sure how useful it is to the wider user base to make an already complex part of the codebase more complex. Rather I'd prefer if we can use a unique identifier name that would fail deshadowing if attempted to be attacked like you have done, as opposed to doing it the "right" way that has all the branch pollution. To be perfectly honest, given the choice between polluted trace branches and supporting this feature, I'd rather not implement this feature at all if that is the case. I've updated the variable name to |
ad74b63 to
1031582
Compare
lukastaegert
left a comment
There was a problem hiding this comment.
I have added a few more tests of my own here (and removed the obsolete missing export hook from the types). Considering that this feature is meant as a stop-gap measure for fixing broken code-bases, I think this should be adequate. The feature may sometimes wrongly associate the shim variable with other instances of the shim variable. However as long as all instances have the same value of "undefined", there should not be any issues. Furthermore if a shim is not reexported, the shim variable is not created at all. Again I think this is not an issue as the code base would break with a runtime exception anyway if the shim variable is accessed.
I added a warning whenever an export is shimmed as I think this will help with fixing a codebase. This warning call also be watched for by a special handler.
If this is not helpful, we can remove it again. @guybedford @kellyselden please check if this works for you, from my side this is good to be merged.
guybedford
left a comment
There was a problem hiding this comment.
Thanks @lukastaegert, seems great to me.
| // could have already been generated | ||
| if (!this.exports[name]) | ||
| this.graph.warn({ | ||
| message: `Export "${name}" has been shimmed in module ${relativeId(this.id)}.`, |
There was a problem hiding this comment.
Perhaps clarify Missing export here.
| this.graph.warn({ | ||
| message: `Export "${name}" has been shimmed in module ${relativeId(this.id)}.`, | ||
| code: 'SHIMMED_EXPORT', | ||
| missing: name, |
There was a problem hiding this comment.
missingExportName or exportName perhaps?
|
This is performing as intended in my test, so all good on my end. I'd like to reiterate that this would be a temporary, stop-gap solution just to get the ecosystem working. I'm all for huge console warnings and deprecating for Rollup v2. |
|
woohoo! |
This Pull Request updates dependency [rollup](https://github.com/rollup/rollup) from `v0.61.2` to `v0.62.0` <details> <summary>Release Notes</summary> ### [`v0.62.0`](https://github.com/rollup/rollup/blob/master/CHANGELOG.md#​0620) [Compare Source](rollup/rollup@v0.61.2...v0.62.0) *2018-06-27* * Add option to automatically shim missing exports ([#​2118](`https://github.com/rollup/rollup/pull/2118`)) * Inline dynamic imports that are also imported statically and only used in a single chunk ([#​2295](`https://github.com/rollup/rollup/pull/2295`)) * Handle caching and invalidation of assets ([#​2267](`https://github.com/rollup/rollup/pull/2267`)) * Fix plugin related types ([#​2299](`https://github.com/rollup/rollup/pull/2299`)) --- </details> --- This PR has been generated by [Renovate Bot](https://renovatebot.com).
This Pull Request updates dependency [rollup](https://github.com/rollup/rollup) from `v0.61.2` to `v0.62.0` <details> <summary>Release Notes</summary> ### [`v0.62.0`](https://github.com/rollup/rollup/blob/master/CHANGELOG.md#​0620) [Compare Source](rollup/rollup@v0.61.2...v0.62.0) *2018-06-27* * Add option to automatically shim missing exports ([#​2118](`https://github.com/rollup/rollup/pull/2118`)) * Inline dynamic imports that are also imported statically and only used in a single chunk ([#​2295](`https://github.com/rollup/rollup/pull/2295`)) * Handle caching and invalidation of assets ([#​2267](`https://github.com/rollup/rollup/pull/2267`)) * Fix plugin related types ([#​2299](`https://github.com/rollup/rollup/pull/2299`)) --- </details> --- This PR has been generated by [Renovate Bot](https://renovatebot.com).
Revived from #1986
From #1986 (comment)
@guybedford Turns out I only needed to do # 1, and I got the desired outcome.
I added that in a previous feature, which is partly why this feature is so small!