Skip to content

handle missing exports in chunked mode - #2118

Merged
lukastaegert merged 8 commits into
rollup:masterfrom
kellyselden:chunked-missing-export-3
Jun 23, 2018
Merged

handle missing exports in chunked mode#2118
lukastaegert merged 8 commits into
rollup:masterfrom
kellyselden:chunked-missing-export-3

Conversation

@kellyselden

Copy link
Copy Markdown
Contributor

Revived from #1986

From #1986 (comment)

@guybedford Turns out I only needed to do # 1, and I got the desired outcome.

Perhaps we could support the handleMissingExport hook returning true or false to determine whether to create a variable or to throw?

I added that in a previous feature, which is partly why this feature is so small!

@guybedford guybedford left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned, this is not about outcome, but ensuring that the exports logic around shimming can entirely be isolated to the src/Module.ts file, so that these exception cases don't spill into other logic which could become a refactoring hazard.

Comment thread src/Chunk.ts Outdated
const module = this.exports.get(variable);

// skip shimmed exports
if (!module) continue;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan in going through with this approach was to be able to avoid having to have special cases like this - otherwise all logic around exports needs to be considered against this case which may lead to bugs.

I really do think we should ensure the shimmed export otherwise behaves exactly like any other, with module being defined here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I'll take another stab at it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, will be a great help to ensure the assumptions are minimal.

{
missingExport() {
return true;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than have the missingExport hook provide this functionality, how about making this a flag option - shimMissingExports: true?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is already the way to stop the default behavior of hard erroring (in non-code-splitting mode). Are you suggesting checking here for shimMissingExports and conditionally not erroring?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes exactly, so that this can be done without a plugin.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@kellyselden
kellyselden force-pushed the chunked-missing-export-3 branch 3 times, most recently from a35282d to 57b27a5 Compare April 15, 2018 23:42
@kellyselden
kellyselden force-pushed the chunked-missing-export-3 branch 8 times, most recently from 3843f25 to ffd1fd5 Compare April 21, 2018 23:31
@kellyselden

Copy link
Copy Markdown
Contributor Author

Ready for review again. It turned out pretty clean IMO. While it is more encapsulated, I had to add a little bit more compatibility code to accommodate some bad assumptions, like NamespaceVariable.originals and traceExport.isExportAllSearch.

Comment thread src/Module.ts
this.exports[name] = {
localName: name,
shim
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible for ShimVariable to inject itself into originals here on the module itself so that variable lookups play out correctly? Or does that introduce too many cross-cutting issues?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about it, but I couldn't figure out how to get access to the NamespaceVariable.originals from earlier in the traceExport call stack at this point of shimming.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think NamespaceVariables are created after the initial tracing in module right? So that exports should already be shimmed by then and then the getExports() call from the namespace constructor should include the shimmed names?

Let me know if there are any other issues - I know it's a bit to investigate.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you have further questions on this one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to remove the NamespaceVariable.originals workaround completely. Something I changed must have made it obsolete.

Comment thread src/ExternalModule.ts Outdated
name: string,
_options?: {
isExportAllSearch?: boolean;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this a default flag over an object rather?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured module.traceExport(name, { isExportAllSearch: true }); would be more clear to the reader than, module.traceExport(name, true);, but I can change it if you want.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be great just to avoid the object allocation, for what it's worth. It doesn't seem like the flags will be extending anytime soon.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/finalisers/esm.ts Outdated
} else {
exportBlock.push(`export const ${specifier.exported} = null;`);
}
} else if (specifier.exported === 'default') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could shift these types of logic out of the finalisers by rendering a line like:

export const $$shim = null;

as part of the core rendering process, then setting the specifier.local value to $$shim.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow. Wouldn't it get really complicated if there were multiple missing exports for a given modules? Like:

import { missing1, missing2 } from './foo';

missing1('bar');
missing2('baz');

@guybedford guybedford Apr 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I should have clarified rendering just a ${varOrConst} $$shim = null;, so that in foo itself the output would be:

const $$shim = null;

export { $$shim as missing1, $$shim as missing2 }

Let me know if that makes sense?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does now. But aren't you worried about adding a var after the deconflicting step? Any name we choose, like $$shim, would possibly conflict with an existing $$shim var.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the change, but the conflict issue still worries me.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$$shim can be added to the list of names to deshadow in the setIdentifierResolutions function of Chunk to ensure proper handling of this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm now running into the problem where my output is

const $$shim$1 = null;

var $$shim = null;
export { $$shim$1 as $$shim, $$shim };

This is a dependency and an entry point. My existing unused $$shim was renamed to $$shim$1 because of the deshadowing, but is still exported as $$shim because of entry point. Now this conflicts with my shimmed export $$shim. It's almost like we need to reverse the renaming by leaving existing $$shims as is, and instead do something like chunk.generateUniqueVarAndExportName() for the shimmed export name. Am I on the right track?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got something working.

@guybedford

Copy link
Copy Markdown
Contributor

Looking good! Thanks for understanding we just want to keep assumptions in unrelated places to a minimum in the codebase.

@guybedford
guybedford dismissed their stale review April 22, 2018 16:14

outdated

@kellyselden
kellyselden force-pushed the chunked-missing-export-3 branch 3 times, most recently from ac832ff to e669b33 Compare April 25, 2018 01:18
Comment thread src/finalisers/esm.ts Outdated
const exportDeclaration: string[] = [];
exports.forEach(specifier => {
if (specifier.shim) {
exportBlock.push(`const ${specifier.local} = null;`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be done in the preRender of Chunk.ts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the const should be a ${varOrConst} reference.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preRender looks a little foreign to me. Would you think it goes in prepend, append, or something else? How do I get the var before the exports block, but after any module wrappers so that it's not in global scope?

@guybedford guybedford Apr 27, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just like at

used['_setter'] = used['_starExcludes'] = used['_$p'] = 1;
, when needing a shim variable you can add it to the deshadowing list this way.

preRender renders the main body source, while render renders the finalizers. I think preRender would be a good place. Note that there is a hoistedSource for bindings that must be available early this could be added to at

let hoistedSource = '';
. Just writing directly into that string is fine (if (shimExports) hoistedSource += ``${varOrConst} $$shim = null``;). Note that finalizers will apply wrappers so that there are no global leaks to worry about.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I do this, it works for everything except systemjs, where the output is

System.register([], function (exports, module) {
	'use strict';
	return {
		execute: function () {

			exports('$$shim$2', $$shim$2);
			var $$shim$2 = null;

			const $$shim = exports('$$shim$1', null);

		}
	};
});

I may have the change the order of functionExports in systemjs to get it to work this way.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made the change.

@kellyselden
kellyselden force-pushed the chunked-missing-export-3 branch 6 times, most recently from c349eb1 to 4c1b81d Compare April 27, 2018 06:30
@kellyselden

Copy link
Copy Markdown
Contributor Author

This branch merged with #2277 is what I've been testing. It is looking good so far. Nothing that can't be hotfixed later on, so I would say it's good.

@guybedford

Copy link
Copy Markdown
Contributor

In that case @lukastaegert it would be great if you could offer a final review here.

@lukastaegert

Copy link
Copy Markdown
Member

Sure thing. It's a little late today for a full review today but I added it to the top of the review queue of the 1.0.0 project and will get to it first thing tomorrow

@lukastaegert lukastaegert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I think we are already quite close but the deconflicting is still faulty unfortunately and I do not know yet what a proper solution would be. Apparently if there already exists a local variable named '_shimmedExport', the shimmed export will be wrongly associated with this variable instead of the global shimmed export variable. Also this produces faulty code if there already were shimmed exports which were actually used.

export { default, default as d3f1, d3f2 } from './d3';
export * from './d3';

export const _shimmedExport$2 = null; No newline at end of file

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is a little hard to debug as it tries to do everything in one test while at the same time forgetting some important combinatorics that actually fail. I would suggest to split this up into several more focused tests.

  • A golden path test showcasing how this feature actually works without name conflicts (helpful for documentation purposes). I think a single import file should be enough here 😉
  • One or even more separate test(s) focusing explicitly on name conflicts. The most important situation that should be covered is that output produced by this feature is feeded back into rollup but there is more. What is actually not working correctly and needs to be fixed is the following (which pretends to be a set of shimmed exports but apparently someone has edited it manually 😜):
// main.js
import { missing, _shimmedExport, _shimmedExport$1 } from './dep.js';
_shimmedExport() // note that we are actually *using* some of the shimmed exports
_shimmedExport$1()
missing();

// dep.js
// added a function with a side-effect here
const _shimmedExport = () => console.log('effect');
export { _shimmedExport, _shimmedExport as _shimmedExport$1 };

// output - main.js
import { _shimmedExport as _shimmedExport$1 } from './d1.js';

_shimmedExport$1();
_shimmedExport$1();
_shimmedExport(); // now hello, who do you belong to!? Isn't this the missing one?

// output - dep.js
const _shimmedExport$1 = () => console.log('effect');

// so apparently there *are* several exports here but they are all wired up the same
export { _shimmedExport$1 as _shimmedExport, _shimmedExport$1, _shimmedExport$1 as missing };

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the combinatorics are checked for the non-compact case, a single compact mode test should suffice, though.

Comment thread src/Module.ts Outdated
this.exports[name] = {
localName: '_shimmedExport'
};
return this.graph.scope.findVariable('_shimmedExport');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of searching we might just use this.graph.exportShimVariable here which would also be a little clearer.

@kellyselden

Copy link
Copy Markdown
Contributor Author

I can update the tests to be more clear. As for the rest of the changes, maybe @guybedford has some ideas?

@guybedford

Copy link
Copy Markdown
Contributor

@kellyselden that would be great if you can make the test adjustments. Will aim to look into the output fixes over the weekend.

@lukastaegert

Copy link
Copy Markdown
Member

I actually started looking into this myself and will continue later this afternoon. Will keep you posted if I come up with a good solution.

@guybedford

Copy link
Copy Markdown
Contributor

@lukastaegert let me know if you're still working on this, otherwise I can look into it then as well.

@lukastaegert

Copy link
Copy Markdown
Member

Not at the moment, got a little side-tracked by the release issues. My current impression so far is that we should not rely on name-matching to detect if an export is shimmed but maybe via a flag added to the export description. Furthermore, we should not use findVariable to get a reference of the shimmed variable instance (which may be shadowed) but directly retrieve it from the graph. Also if we want to rely on the shim variable having a specific name in the end, we need to make sure that no variable on module level can have this name.

But I do not have any results yet and will not find any time to look into this further before tomorrow so please go ahead if you have time today.

@guybedford

Copy link
Copy Markdown
Contributor

@lukastaegert it appears you're right there's no clean way to do this at the variable level due to the fact that we dedupe based on name, not variable name (two identifiers of the same name in the same scope cant reference different underlying variables in the deshadowing logic)

Given this, the approach @kellyselden originally took would work here.

My major concern for this feature from the start was to find an approach that would use the existing tracing logic and not pollute it with conditional exceptions. This is non-spec behaviour, and I'm not sure how useful it is to the wider user base to make an already complex part of the codebase more complex.

Rather I'd prefer if we can use a unique identifier name that would fail deshadowing if attempted to be attacked like you have done, as opposed to doing it the "right" way that has all the branch pollution.

To be perfectly honest, given the choice between polluted trace branches and supporting this feature, I'd rather not implement this feature at all if that is the case.

I've updated the variable name to _missingExportShim, and integrated the test updates and feedback for now.

@guybedford
guybedford force-pushed the chunked-missing-export-3 branch from ad74b63 to 1031582 Compare June 21, 2018 12:24

@lukastaegert lukastaegert left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a few more tests of my own here (and removed the obsolete missing export hook from the types). Considering that this feature is meant as a stop-gap measure for fixing broken code-bases, I think this should be adequate. The feature may sometimes wrongly associate the shim variable with other instances of the shim variable. However as long as all instances have the same value of "undefined", there should not be any issues. Furthermore if a shim is not reexported, the shim variable is not created at all. Again I think this is not an issue as the code base would break with a runtime exception anyway if the shim variable is accessed.

I added a warning whenever an export is shimmed as I think this will help with fixing a codebase. This warning call also be watched for by a special handler.

If this is not helpful, we can remove it again. @guybedford @kellyselden please check if this works for you, from my side this is good to be merged.

@guybedford guybedford left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lukastaegert, seems great to me.

Comment thread src/Module.ts Outdated
// could have already been generated
if (!this.exports[name])
this.graph.warn({
message: `Export "${name}" has been shimmed in module ${relativeId(this.id)}.`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps clarify Missing export here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/Module.ts Outdated
this.graph.warn({
message: `Export "${name}" has been shimmed in module ${relativeId(this.id)}.`,
code: 'SHIMMED_EXPORT',
missing: name,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missingExportName or exportName perhaps?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@kellyselden

Copy link
Copy Markdown
Contributor Author

This is performing as intended in my test, so all good on my end. I'd like to reiterate that this would be a temporary, stop-gap solution just to get the ecosystem working. I'm all for huge console warnings and deprecating for Rollup v2.

@lukastaegert lukastaegert modified the milestones: 1.0.0, 0.62.0 Jun 22, 2018
@lukastaegert
lukastaegert merged commit 8ba819f into rollup:master Jun 23, 2018
@kellyselden
kellyselden deleted the chunked-missing-export-3 branch June 23, 2018 11:55
@kellyselden

Copy link
Copy Markdown
Contributor Author

woohoo!

calebeby referenced this pull request in Pigmice2733/scouting-frontend Jun 30, 2018
This Pull Request updates dependency [rollup](https://github.com/rollup/rollup) from `v0.61.2` to `v0.62.0`



<details>
<summary>Release Notes</summary>

### [`v0.62.0`](https://github.com/rollup/rollup/blob/master/CHANGELOG.md#&#8203;0620)
[Compare Source](rollup/rollup@v0.61.2...v0.62.0)
*2018-06-27*
* Add option to automatically shim missing exports ([#&#8203;2118](`https://github.com/rollup/rollup/pull/2118`))
* Inline dynamic imports that are also imported statically and only used in a single chunk ([#&#8203;2295](`https://github.com/rollup/rollup/pull/2295`))
* Handle caching and invalidation of assets ([#&#8203;2267](`https://github.com/rollup/rollup/pull/2267`))
* Fix plugin related types ([#&#8203;2299](`https://github.com/rollup/rollup/pull/2299`))

---

</details>




---

This PR has been generated by [Renovate Bot](https://renovatebot.com).
stipsan referenced this pull request in scroll-into-view/compute-scroll-into-view Jul 5, 2018
This Pull Request updates dependency [rollup](https://github.com/rollup/rollup) from `v0.61.2` to `v0.62.0`



<details>
<summary>Release Notes</summary>

### [`v0.62.0`](https://github.com/rollup/rollup/blob/master/CHANGELOG.md#&#8203;0620)
[Compare Source](rollup/rollup@v0.61.2...v0.62.0)
*2018-06-27*
* Add option to automatically shim missing exports ([#&#8203;2118](`https://github.com/rollup/rollup/pull/2118`))
* Inline dynamic imports that are also imported statically and only used in a single chunk ([#&#8203;2295](`https://github.com/rollup/rollup/pull/2295`))
* Handle caching and invalidation of assets ([#&#8203;2267](`https://github.com/rollup/rollup/pull/2267`))
* Fix plugin related types ([#&#8203;2299](`https://github.com/rollup/rollup/pull/2299`))

---

</details>




---

This PR has been generated by [Renovate Bot](https://renovatebot.com).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants