Closed Bug 1315558 Opened 9 years ago Closed 2 months ago

tld service for webextensions

Categories

(WebExtensions :: General, enhancement, P3)

49 Branch
enhancement

Tracking

(firefox153 fixed)

RESOLVED FIXED
153 Branch
Tracking Status
firefox153 --- fixed

People

(Reporter: mixedpuppy, Assigned: mckenfra)

References

(Blocks 2 open bugs)

Details

(Keywords: dev-doc-complete, Whiteboard: [wecg][design-decision-approved])

Attachments

(4 files, 4 obsolete files)

We should consider providing tld for webextensions. The SDK provides it here: https://dxr.mozilla.org/mozilla-central/source/addon-sdk/source/lib/sdk/url.js#313 A simple Host api might be useful, such as https://github.com/NiklasGollenstede/get-tld CookieMonster is one use case.
The `eTLD` service is also used in the UsableHomeButton [1] extension. Namely, the `Services.eTLD.getPublicSuffixFromHost()` function is used. A pure-JS library is not a reasonable equivalent of built-in `eTLD` features: * The huge advantage of the built-in `eTLD` service is that the actual list of top-level domains is automatically always up-to-date thanks to browser's automatic updates. This cannot be achieved in a reasonable way with a library. * Using multiple different implementations of the same feature in the same application leads to inconsistencies and is considered bad practice. * Using an extra library while already having similar functionality in the application results in wasting memory and CPU power and generally have a negative performance impact. [1] https://addons.mozilla.org/firefox/addon/usablehomebutton/
Priority: -- → P5
Whiteboard: [design-decision-needed]
Component: WebExtensions: Untriaged → WebExtensions: General
We've had more than one request for this, I think providing an API to the internal implementation is better than making people use external libraries.
We discussed this and decided that it should probably be implemented as an extension-specific WebIDL interface, probably along the lines of MozURLUtils.
Whiteboard: [design-decision-needed] → [design-decision-approved]
Good to see this feature will be implemented. Is there a schedule? The RequestPolicy add-on extensively uses the public suffix list. According to the WE triage meeting, the priority should be P2, no? Thanks.
A good catch, the meeting did say P2, changing. There's no schedule for this yet, its in the backlog. If its something you'd like to contribute code to, please let us know.
Priority: P5 → P2
Blocks: 1311472
Whiteboard: [design-decision-approved] → [design-decision-approved], triaged
Assignee: nobody → tomica
Status: NEW → ASSIGNED
(In reply to Kris Maglione [:kmag] from comment #4) > We discussed this and decided that it should probably be implemented as an > extension-specific WebIDL interface, probably along the lines of MozURLUtils. To confirm I understand the intent here, the idea is to have a `MozURLUtils` property on the `window` global, available only to web extensions (for now), which would include (for this bug), a method like `getTLD(url)`?
Window and Sandbox globals, yes. Or something along those lines, anyway.
Getting the list of TLDs can be done by the extension itself, an API provided by the browser is not needed. Extensions can cache the list, thereby having offline access to the list.
Blocks: 1226546
+1 for this. Greasemonkey uses nsIEffectiveTLDService in our legacy version.
This would be useful for features I want to add to the tab-stat addon (grouping by domain name). Each addon reimplementing something the browser already has built-in (with automatic updates of the list, at that), would be error prone and suboptimal.
I’m forced to temporarily remove the eTLD-powered feature entirely from WebExtensions version of my UsableHomeButton extension since bundling an own copy of 12000+-entries database with each extension that needs it looks unreasonable at least in terms of performance.
This would be great, millions of users use currently sth. like this extensive Adblock implementation (with other addons using different and even slower approaches): var publicSuffixes = { "0.bg": 1, "1.bg": 1, "1kapp.com": 1 }; function getDomain(hostname) { let bits = hostname.split("."); let cutoff = bits.length - 2; for (let i = 0, bitsLen = bits.length; i < bitsLen; i++){ let offset = publicSuffixes[bits.slice(i).join(".")]; if (typeof offset != "undefined"){ cutoff = i - offset; break; } } if (cutoff <= 0) return hostname; return bits.slice(cutoff).join("."); }
See Also: → 1437098
Product: Toolkit → WebExtensions

ni? Fallen to consider and fit in somewhere.

Assignee: tomica → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(philipp)
Priority: P2 → P3

Thanks for tagging me. I'm fine with having this sort of API available, though I'm not sure about the approach. If we are going to be exposing this on the window I'd prefer if we could work towards standardizing it instead of having a WebExtensions specific window property.

Flags: needinfo?(philipp)

BTW: there are also Rust implementations in an attempt to parse PSL quickly:
https://crates.io/crates/psl

There are hundreds of JS libraries and all big extensions like Adblock Plus parse the PSL multiple times on each link on a every website. However, these are all not very efficient.

I use a Regex in my scripts that is executed on every website link. This Regex checks if a link is a referrer link (e.g. http://www.yahoo.de/redirect.cgi?=www.standard.co.uk). For different conditions, the PSL has to be parsed 4 times on each link on a website.
Adbuster's reference implementation is using an array map which results in an overall performance loss of 20% for each website, but it would be awesome if there would be a faster, FF integrated solution.

Assignee: nobody → 3ecdbelo
Status: NEW → ASSIGNED

I filed an issue to discuss this feature in the WebExtensions Community group:

https://github.com/w3c/webextensions/issues/231
("Extension API to find the public suffix (eTLD) of a given URL/domain")

Whiteboard: [design-decision-approved], triaged → [wecg][design-decision-approved]
Attachment #9281822 - Attachment is obsolete: true

Exposes a call to Services.eTLD.getPublicSuffix(url) as a new
webextensions API method: browser.dns.getPublicSuffix(url)

E.g. getPublicSuffix("https://www.mozilla.co.uk:80") => "co.uk"

The method also takes an optional "additionalParts" integer parameter:

E.g. getPublicSuffix("https://www.mozilla.co.uk:80", 1) => "mozilla.co.uk"

This will save Addon creators from having to reinvent the wheel.
The Mozilla Multi-Account Containers Addon team intends to use this functionality
for a Wildcard Subdomains feature that is currently in development, see PR:
https://github.com/mozilla/multi-account-containers/pull/2352

Attachment #9280827 - Attachment is obsolete: true
Severity: normal → S3

The severity field for this bug is relatively low, S3. However, the bug has 277 votes and 5 See Also bugs.
:3ecdbelo, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Flags: needinfo?(3ecdbelo)

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Flags: needinfo?(3ecdbelo)

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Assignee: 3ecdbelo → nobody
Status: ASSIGNED → NEW
Type: defect → enhancement

:mcfrenka

After some discussions in the WECG there is a generally positive sentiment towards a (cross-browser) extension API for public suffix. Are you interested in creating a proposal in the WECG so that we can finalize the details of the API? Once that is done the patch can be tweaked based on the feedback and landed.

The proposal process is described at https://github.com/w3c/webextensions/blob/main/proposals/proposal_process.md

I mentioned this status at https://github.com/w3c/webextensions/issues/231#issuecomment-2247438680

Flags: needinfo?(3ecdbelo)
Assignee: nobody → 3ecdbelo
Status: NEW → ASSIGNED

:robwu

Thanks for pushing this forward. Yes I would be interested in creating a WECG proposal. What sort of timeframe would be good (for creating the proposal)?

There is no time pressure here. It is totally fine if you need a few weeks to find time. Feel free to contact me on Matrix if you have some quick questions.

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Assignee: 3ecdbelo → nobody
Status: ASSIGNED → NEW

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.

Flags: needinfo?(3ecdbelo)
Assignee: nobody → 3ecdbelo
Status: NEW → ASSIGNED

Depends on D295512

Attachment #9281906 - Attachment is obsolete: true
Attachment #9571987 - Attachment description: Bug 1315558 - Add publicSuffix extension API. 4/4 - tests. r=robwu → Bug 1315558 - Add publicSuffix extension API. 3/4 - tests. r=robwu
Attachment #9571985 - Attachment description: Bug 1315558 - Add publicSuffix extension API. 3/4 - getVersion(). r=robwu → Bug 1315558 - Add publicSuffix extension API. 4/4 - getVersion(). r=robwu
Attachment #9591974 - Attachment description: Bug 1315558 - Patch publicSuffix extension API - rename encoding=unicode. r=robwu → Bug 1315558 - Add publicSuffix extension API. 4/4 - rename encoding=unicode. r=robwu
Attachment #9571985 - Attachment description: Bug 1315558 - Add publicSuffix extension API. 4/4 - getVersion(). r=robwu → Bug 1315558 - Add publicSuffix extension API. DROP - getVersion(). r=robwu
Regressions: 2045150
No longer regressions: 2045150

Thanks a lot Francis for your contribution here. This marks the completion of one of the most voted on bugs in WebExtensions on Bugzilla!

:mckenfra contributed the following significant pieces:

Through the effort he showed a lot of patience, since it took over a year between the start of the proposal and the final approvals. But the cross-browser alignment and community efforts paid off, because we now have a cross-browser publicSuffix extension API implementation, including a late fixup to the API (part 4 patch of the patch stack, renaming "unicode" to "display") following cross-browser implementer experience that highlighted an issue (https://github.com/w3c/webextensions/pull/1013).

Attachment #9571985 - Attachment is obsolete: true
See Also: → 2059819

For those interested, I created a blog post with an overview of how the publicSuffix API was developed, at https://blog.mozilla.org/addons/2026/08/06/how-the-publicsuffix-api-was-created/

I'm working on the Chromium implementation of this API at the moment, and I noticed a (very minor) bug in the Firefox implementation. For a case like browser.publicSuffix.isKnownSuffix("invalid%.domain"), the API is returning false instead of throwing an exception. That seems to contradict the API proposal and the docs in MDN (and soon to be the Chromium implementation).

By the way, I noticed a few edge-cases that Chromium's hostname normalisation logic handled possibly more permissively than spec'd for this API:

  • "subdomain%2edomain.com" -> Considered valid (normalised to "subdomain.domain.com")
  • "foo_bar.com" -> considered valid
  • "domain。com" -> Considered valid (normalised to "domain.com")

I tested those cases against the implementation here, and found Firefox agreed except the "subdomain%2edomain.com" case. So I have proposed we also reject the "subdomain%2edomain.com" case in Chromium.

See Also: → 2066620
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: