Obfuscation

From the PING review:

What value is provided by this obfuscation algorithm? Can this feature be marked at risk, given the uncertainty about whether it satisfies any vendor goals while increasing the complexity of the spec and making the source less easily inspectable by the reader?

Why does the creation of the obfuscation key based on the SHA-1 hash function include a SHOULD requirement rather than a MUST? This relaxation seems primarily to decrease interoperability.

The obfuscation section contains no requirements on reading systems. Maybe they are just implicitly supposed to de-obfuscate these resources in order to render the book as intended. Are reading systems expected not to provide unobfuscated access to these obfuscated files to users?

The obfuscation section contains no requirements on reading systems.

Reading system requirements are specified in the Reading Systems specification. In this case, they are required to reverse the process to deobfuscate: https://w3c.github.io/epub-specs/epub33/rs/#sec-container-res-obfus

The obfuscation section contains no requirements on reading systems.

Reading system requirements are specified in the Reading Systems specification. In this case, they are required to reverse the process to deobfuscate: https://w3c.github.io/epub-specs/epub33/rs/#sec-container-res-obfus

Yes, I did eventually figure that out! Thanks. I'm not clear on whether the user agent/reading system is supposed to not provide the de-obfuscated file to the user, or if that's just a requirement that comes externally from the vendor or the DRM provider and not part of the spec.

Why does the creation of the obfuscation key based on the SHA-1 hash function include a SHOULD requirement rather than a MUST? This relaxation seems primarily to decrease interoperability.

This looks like a bad porting of the original algorithm specified in: http://idpf.org/epub/20/spec/FontManglingSpec_2.0.1_draft.htm

That document is contradictory on this point (sigh), but it says in the Obfuscation Algorithm section:

The key for the algorithm must be a 20 byte (160 bit) SHA-1 digest[3] of the publication's unique identifier.

The "should" comes later but it doesn't make sense how it couldn't also be a must.

It appears when it was integrated in the original 3.0 revision that the "must" was dropped (it's no longer limited to 20 bytes) and the later should retained. But I agree that makes no sense since if it you can't know how to create the key, you can't know how to deobfuscate.

Assuming we keep the section, it definitely needs correcting.

I'm not clear on whether the user agent/reading system is supposed to not provide the de-obfuscated file to the user

The reading system will deobfuscate and use the font, but I'd assume most apps, at least, do that in memory and don't write it out to disk, if that's what you're asking. (I don't write reading systems, so that's just my understanding based on how other resources have been stated to be handled; maybe someone else will correct me.) The user typically isn't going to have any way from within the reading system to access the deobfuscated source regardless of how it's done, though, just as they can't access any other resources.

If the reading system is running in a browser, my guess would be you might be able to access the font (but maybe only as a blob url?). I don't think preventing access completely is realistic in this situation.

But obfuscation was always meant to be trivial, so the requirements have been pretty thin. It's tacitly understood that if the user wants at the font, and they have access to the epub file, they'll be able to get it.

The issue was discussed in a meeting on 2021-10-26

no resolutions were taken

View the transcript

1.2. DRM and obfuscation

See github issue #1873, #1874.

Wendy Reid: next, DRM and obfuscation.
… is obfuscation only for fonts? No. It can also be applied to images, I believe. But most common use-case is for fonts.
… the obfuscation is done primarily by tools.

Dave Cramer: not aware of real world use of obfuscation aside from fonts.
… adobe was heavily involved in original epub. They were concerned about fonts easily accessed from the epub package..

Nick Doty: the obfuscation can be undone easily though.

Dave Cramer: some font vendors have told me that even these very ineffective means are good because then they can say that if you work around them then you violated DMCA etc..
… the legal case is clearer.
… this is anecdotal only. As a publisher we've not obfuscated fonts..
… but i'm open to discussion of whether we need this or not.

Rick Johnson: not a single title in our 2mm title inventory with font obfuscation included.

Wendy Reid: obfuscation tends to break things in RS.

Nick Doty: I'm not sure the goal of making it easier to sue the user is important.

Wendy Reid: DRM is tricky because the spec does not specify the DRM to be applied.
… there are a number out there, with the most popular being Adobe DRM.
… though implementation is also platform specific.
… in our charter we've said that we are not going to talk about DRM, which also complicates things.

Brady Duga: i think you can only obfuscate fonts.
… boils down to whether or not the publisher's license of the font allows it to be used in ebook or not.
… it appears to be okay to font vendors as long as font can be embedded.
… removing this may limit fonts that publishers can use.
… but this was also written before WOFF was a thing.
… so i'm open to recommending WOFF, but leaving it open to RS to do this (esp. for backwards compat).

Nick Doty: deprecating the feature would be helpful in terms of the risk of expanding user-agent-implemented obfuscation to other w3c work.

Rick Johnson: my opinion is that we shouldn't address this in spec.

Matt Garrish: we've never wanted to go into DRM implementation in spec.

Samuel Weiler: but you have the have the hooks for it in the spec, even if you aren't fully specifying the DRM.
… so we care about this.

Nick Doty: and the hooks make it so that the full contents of files are encrypted, rather than some smaller subset.

Tzviya Siegman: epub exists in an ecosystem that has been around for a long time. If we took out those hook we would be ceding our standard to a world that would not accept the lack of it.

Wendy Reid: we wouldn't have the support of most of the publishing industry, RS would be happy, and retailers would also be impacted.

Nick Doty: I think our goal in doing a privacy review is to note the current privacy situation and the potential harms.
But we could also consider harm reduction or mitigation, as W3C has done in the past about DRM.

Dave Cramer: I might disagree. If we took this out of the spec, would this change anything in practice?.
… as far as i know every epub created is done without DRM, because DRM is the responsibility of retailer rather than publisher.
… so removing from spec would not change this.
… and the DRM that exists and is being used may or may not rely on the hooks in the spec.

Matt Garrish: i agree with dauwhe. It doesn't break anything if we take this out..
… it might just make things more difficult for implementors.
… we'd need more information certainly, but not sure that anything breaks by taking this out.

Wendy Reid: okay, good points everyone. This is something we need to assess as part of our privacy threat model.

The issue was discussed in a meeting on 2021-10-28

no resolutions were taken

View the transcript

4.3. Obfuscation (issue epub-specs#1873)

See github issue epub-specs#1873.

Dave Cramer: another thing PING brought up: obfuscation.

Matt Garrish: it would be nice if we didn't have this, but now that we're 10 years in, not sure what we can do about it.
… we're kind of stuck with it.
… there is a SHOULD that should be a MUST.
… we can point out that it is trivial.

Brady Duga: we can't remove it; we could recommend using WOFF.

Brady Duga: WOFF.

Theresa O'Connor: clarification on obfuscation?.

Dave Cramer: how to keep fonts from escaping the EPUB to protect font vendors.

Matt Garrish: See the obfuscation section in the spec.

Dan Lazin: keeping in mind that Adobe is a main exporter of custom ebooks, they have had a strong interest in both DRM and protecting their fonts.
… if you take an InDesign ebook and change the publication ID, the font will no longer work because it is ties to that hash.

Dan Lazin: See extra test commits on obfuscation.

If the obfuscation is trivial, then I'm not sure what value it's providing. If it's there for historical or backcompat only, it could be deprecated and warnings put in place to minimize any future harm.

Perhaps the purpose (as was mentioned at the joint group call) is to make it easier to sue or threaten criminal consequences for anyone who writes a simple script to de-obfuscate or any vendor that implements a Reading System that happens to save the file to disk. If so, that seems inconsistent with ethical Web principles.

Also, if the purpose is to enhance legal threats, we should probably document that risk somewhere: I don't want someone getting sued because they implemented -- or wrote tests for! -- the Reading System specification.

If the obfuscation is trivial, then I'm not sure what value it's providing

Trivial is still going to block most non-technical self publishers from being able to take the font and drop it in their own book, for example. (I'd hope professional publishers would know better.) It provides a measure of defence for the font vendor.

I don't think it matters much legally whether you took an unobfuscated font directly or you figured out how to reverse the obfuscation, but IANAL. You're violating the font licensing agreement by reusing it without paying for it.

Whether the user agent assumes any risk by allowing access to the unobfuscated version isn't something I can answer, either. The theft isn't in deobfuscating but in reusing without a license, so I would expect not.

There's been some discussion about the origins of this and whether it's still needed in the group's email list starting here: https://lists.w3.org/Archives/Public/public-epub-wg/2021Oct/0025.html

As I noted in the meeting minutes above, I think it's still "needed" because most (commercial) EPUBs are exported from InDesign and Adobe cares a lot about font copyright. I would guess that Adobe would not agree to removing obfuscation, and practically speaking we would need them to.

I don't think it matters much legally whether you took an unobfuscated font directly or you figured out how to reverse the obfuscation, but IANAL. You're violating the font licensing agreement by reusing it without paying for it.

Whether the user agent assumes any risk by allowing access to the unobfuscated version isn't something I can answer, either. The theft isn't in deobfuscating but in reusing without a license, so I would expect not.

This isn't theft, but potential copyright infringement, to be clear. Obfuscation doesn't only make it a little more difficult for someone to copy a font into another publication that they would sell without permission (a clear case of copyright infringement), but also often breaks epub files when they're edited on a user's device.

I am also not a lawyer, but anti-circumvention provisions in the DMCA and other laws around the world do make it particularly risky to produce (or distribute or market) de-obfuscation tools, even if you never use it or intend it for copyright infringement.

More background here: https://en.wikipedia.org/wiki/Anti-circumvention

There's been some discussion about the origins of this and whether it's still needed in the group's email list starting here: https://lists.w3.org/Archives/Public/public-epub-wg/2021Oct/0025.html

This is super useful context, thank you! It also recommends a clear way forward, that WOFF or some subsetting proposals could make obfuscation (and the legal risks of de-obfuscation) unnecessary.

Also, if obfuscation is only ever used for font files, that would be a useful limitation to note. Many of the effects (for privacy, accessibility, etc.) would be less severe if the only obfuscated files are ones that don't include contents of the text, active scripts or references to external resources.

Also, if obfuscation is only ever used for font files, that would be a useful limitation to note. Many of the effects (for privacy, accessibility, etc.) would be less severe if the only obfuscated files are ones that don't include contents of the text, active scripts or references to external resources.

We will propose to restrict obfuscation to only fonts, and we can enforce this via EPUBCheck. We can't remove obfuscation entirely as the feature is widely used. Adobe InDesign does this. Forbidding this would break thousands and thousands of existing books.

The situation is quite different in Japan.
Japanese fonts have strict license conditions, so in many cases they are not embedded in EPUB.
There are few cases where EPUB is output from Adobe InDesign. MORISAWA (DTP vendor and font vendor) is the only tool for embedding fonts in EPUB. However, in order to use the font embedded there in a reader that reads with a web browser, it is necessary to deobfuscate it and put it as a file, and since it is unknown whether it is allowed or not, our(Voyager's) RS does not use embedded fonts.

We can't remove obfuscation entirely as the feature is widely used. Adobe InDesign does this. Forbidding this would break thousands and thousands of existing books.

I'm not against above Dave's comments.
Just explained the situation at Japan.

I would certainly recommend requiring limiting this obfuscation technique to only where it's already being used.

Can it also be marked as a deprecated technique, with clear alternatives (to WOFF or something else) to move to something better? If it's known that this feature is generally bad for users and authors and reading systems but is included for backwards compatibility, then we should be able to note it as deprecated and provide better methods going forward.

I think it would help to explain the harms of the font obfuscation technique, in addition to the pointers to better alternatives. (Obfuscation breaks compatibility and interoperability of EPUB files, creates opacity for end users inspecting the files they're reading and introduces complexity and potential legal liability for reading system developers.)

We might also include a warning (in the RS spec) to reading system developers of the potential legal threats if they provide de-obfuscation or access to fonts.

I wonder if we should better explain the limitations of font obfuscation on the authoring side so that it fully removes any expectation that reading systems have any obligation to keep the obfuscated font secure.

The key sentence in the introduction is this:

The hope is that this algorithm will meet the requirements of most vendors who require some assurance that their fonts cannot simply be extracted by unzipping the Container.

The only expectation is that it will help prevent trivial copying out of one container and into another, but this may be something we take for granted. Perhaps we can list ways that obfuscation does not protect the content from copying to better remove any misunderstanding (e.g., that users may be able to gain access to the unobfuscated font through their reading system).

There shouldn't be a threat to reading system developers from using obfuscated fonts. The primary point of concern is between the author and the font vendor -- namely, that the vendor agrees that obfuscation is sufficient protection if that vendor isn't the one protecting the resource.

My understanding is that the DMCA has been used as a legal threat against those distributing open source software that allows for de-obfuscation and saving of font files, and that that threat could also be levied against any reading system that saves the de-obfuscated font file.

@toshiakikoike

Going back to @toshiakikoike's comment, should deobfuscation support be a recommendation and not a requirement? If there are already reading systems ignoring obfuscated fonts, it would be contradictory to compel reading systems to support deobfuscation. You must deobfuscate the font even if you don't use it?

My understanding is that the DMCA has been used as a legal threat against those distributing open source software that allows for de-obfuscation and saving of font files, and that that threat could also be levied against any reading system that saves the de-obfuscated font file.

Obfuscation in EPUB has been around since 2008 or so. I'm not aware of litigation around this, or threats of litigation.

By "saving the de-obfuscated file" do you mean making the font easily available to the end user in its original form?

My understanding is that the DMCA has been used as a legal threat against those distributing open source software that allows for de-obfuscation and saving of font files, and that that threat could also be levied against any reading system that saves the de-obfuscated font file.

Obfuscation in EPUB has been around since 2008 or so. I'm not aware of litigation around this, or threats of litigation.

By "saving the de-obfuscated file" do you mean making the font easily available to the end user in its original form?

Implementations of both the IDPF and Adobe font (de)obfuscation methods have been available "in the open" for some time now. I'm not sure about the legal implications, but practically-speaking font de-obfuscation seems to be a relatively simply hurdle to bypass. A few examples:

Readium (1) Javascript implementation: https://github.com/readium/readium-js/blob/b2c65bf3dd3b47ec716d328479a3c8af51fde0d2/js/epub-fetch/encryption_handler.js#L19-L22
Readium (1) C++ implementation: https://github.com/readium/readium-sdk/blob/a32d900ce337d9ca46dbb029a92f581184ee224c/ePub3/ePub/font_obfuscation.cpp#L58
Readium (2) Javascript implementation: https://github.com/readium/r2-shared-js/blob/c83cfee2cdb2c06f17406a1af3a21788124747f2/src/transform/transformer-obf-idpf.ts#L20 and https://github.com/readium/r2-shared-js/blob/c83cfee2cdb2c06f17406a1af3a21788124747f2/src/transform/transformer-obf-adobe.ts#L18
Readium (2) iOS / Swift implementation: https://github.com/readium/swift-toolkit/blob/b4a9358ad9d0a7cb0072f5c9b1bbb72a7a2d28c7/Sources/Streamer/Parser/EPUB/Resource%20Transformers/EPUBDeobfuscator.swift#L20-L21
Readium (2) Android / Kotlin implementation: https://github.com/readium/kotlin-toolkit/blob/adeb8f452fbd48b6ab7e1132264bcce9ab64ee95/readium/streamer/src/main/java/org/readium/r2/streamer/parser/epub/EpubDeobfuscator.kt#L46-L49
Calibre Python implementation: https://github.com/kovidgoyal/calibre/blob/3f2aa59072ea3f6c68222fe1e704ec281df6391d/src/calibre/ebooks/conversion/plugins/epub_input.py#L10-L26
Sigil Python implementation: https://github.com/Sigil-Ebook/Sigil/blob/3e8ea448ceefbda53e983e6df79a185c239ae4d0/src/Resource_Files/plugin_launchers/python/epub_utils.py#L138-L180
FBReader Java implementation: https://github.com/geometer/FBReader-Android-2/blob/90779f9ed922e12c67b5df835d0aba5f09e9dc70/fbreader/common/src/main/java/org/geometerplus/zlibrary/core/drm/embedding/EmbeddingInputStream.java#L28
Mantano Dart implementation: https://github.com/Mantano/components/blob/588d9abb1c1e111cf029dfa1bee006149db9a9bc/streamer/lib/src/epub/epub_deobfuscator.dart#L21-L25
A few notable Javascript implementations:
etc.

I agree that the obfuscation algorithm is not challenging for a programmer to bypass and that the code is publicly available (as is the algorithm). I believe the DMCA doesn't require protections to be especially strong for it to be illegal to provide circumvention tools.

My understanding is that the DMCA has been used as a legal threat against those distributing open source software that allows for de-obfuscation and saving of font files, and that that threat could also be levied against any reading system that saves the de-obfuscated font file.

I should be more precise here. I don't know for certain that a particular DMCA complaint has been filed, I've just observed someone posting a link to a github repo for a tool that does de-obfuscation and then the link being broken / code not being available. (My recollection was that there was a reference to DMCA or to a legal issue, but if there was, I don't have a link handy any more.)

By "saving the de-obfuscated file" do you mean making the font easily available to the end user in its original form?

Yes, that's what I mean.

I should be more precise here. I don't know for certain that a particular DMCA complaint has been filed, I've just observed someone posting a link to a github repo for a tool that does de-obfuscation and then the link being broken / code not being available. (My recollection was that there was a reference to DMCA or to a legal issue, but if there was, I don't have a link handy any more.)

There's a very recent case where a GitHub repo that had code to completely remove DRM from an ebook was taken down via a DMCA notice.

The issue was discussed in a meeting on 2022-02-03

List of resolutions:

Resolution No. 2: Remove the legal reference from PR 1980, and merge 1980.

View the transcript

2. Updates to Obfuscation.

See github pull request epub-specs#1980.

See github issue epub-specs#1873.

Dave Cramer: this is the PR. There's a lot of discussion in the related issue..
… we've made a bunch of fixes to this.
… npd asked if we could explain harms of font obfuscation techniques, i.e., interop issues, opacity for end users inspecting what they are reading.
… but i've never had to inspect font file of epub i'm reading, so i'm not on board with that.
… could create liability for RS.
… but i'm not aware of legal issues, or threat of legal issues.

Brady Duga: i'm skeptical of even mentioning legal issues without explicit guidance from lawyers.
… even the caution that we're currently proposing, not sure what legal implications we are hinting at with that.
… if I was concerned about legal issues, I would consult my legal team rather than getting it from a spec.

Dave Cramer: i share this concern.

Matt Garrish: yeah, i struggled to come up with a caution that was meaningful.
… there was talk about DMCA.
… there are probably legal issues all over the place, so why pick just one.
… i'm not strongly in favor of this, so I would be okay with removal.
… stuff on content side is a little less controversial.

Dave Cramer: and some of the other limitations are legitimate.
… there can be real interop problems, etc..

Wendy Reid: +1 to general caution.

Brady Duga: i'm fine with the general caution.

Dave Cramer: mgarrish can you just remove the legal reference?.

Matt Garrish: yes.

Brady Duga: one other language issue about "designed to break the obfuscation".
… you're not breaking the obfuscation because its a well defined algorithm.

Wendy Reid: "deobfuscate".

Brady Duga: "intentionally make available"?.

Matt Garrish: agree.
… on the RS side, do we leave it as SHOULD, or should be go back to MUST support deobfuscation?.

Brady Duga: fine with having it as SHOULD support deobfuscation.

Dave Cramer: fine with leaving it at SHOULD, this is not a core feature.

Proposed resolution: Remove the legal reference from PR 1980, and merge 1980. (Wendy Reid)

Shinya Takami (高見真也): +1.

Wendy Reid: +1.

Toshiaki Koike: +1.

Matthew Chan: +1.

Brady Duga: +1.

Masakazu Kitahara: +1.

Dave Cramer: +1.

Resolution #2: Remove the legal reference from PR 1980, and merge 1980.

@npdoty

@npdoty, in view of the recent updates on the spec (#1980) is it o.k. to close this issue? Thx.

The issue was discussed in a meeting on 2022-04-08

List of resolutions:

Resolution No. 1: Close remaining privacy and security issues.

View the transcript

1. Close Privacy & Security Issues.

Dave Cramer: the TAG has reappeared of making a couple comments, I am making a PR to mention that when using web APIs, which have the most dramatic privacy and security implications (geolocations, push notifications) then you should get user consent.

See github issue epub-specs#1959.

Dave Cramer: we have several issues where there was never much discussion in the issue (#1959 for example).
… I think the PR i mentioned earlier would serve to close this issue.
… agree/disagree?

Ivan Herman: we had a lot of discussion with PING, good discussions, after which we made extensive additions to answer the issues they raised.
… and we contacted them several times to get their acknowledgement. So at this point we consider these issues closed..
… they have the right to reopen issues if they like.
… Amy from TAG has closed the issue of epub review on the TAG repo, so that is an indication of how they feel.

Gregorio Pellegrino: so is this passed? it is okay?

See github issue epub-specs#1872.

Ivan Herman: yes, it is okay.

Dave Cramer: risk of exposure and finger printability.
… this was raised before we clarified the threat model, can we close this now?

See github issue epub-specs#1873.

Dave Cramer: obfuscation, which we've discussed extensively, followed by updates to the spec docs.

See github issue epub-specs#1875.

See github issue epub-specs#1876.

Dave Cramer: interactivity, which we've addressed as best we can given that it's ambiguous.
… self-contained packages, this is a case where its appropriate to close because epub is clear that it is largely self-contained, subject to exceptions enumerated in the spec. Not dramatically impacting privacy.

See github issue epub-specs#1957.

Dave Cramer: we enumerated the threat model, which deals with #1957.

See github issue epub-specs#1958.

Dave Cramer: permission prompts, we're dealing with this, strengthened text.

See github issue epub-specs#1959.

Proposed resolution: Close remaining privacy and security issues. (Wendy Reid)

Dave Cramer: broad user expectations issues, which is covered by the other changes we've made.

Ivan Herman: +1.

Matthew Chan: +1.

Shinya Takami (高見真也): +1.

Bill Kasdorf: +1.

Dave Cramer: +7.

Wendy Reid: +1.

Matt Garrish: +1.

Murata Makoto: +1.

Dan Lazin: +1.

Charles LaPierre: +1.

Ben Schroeter: +1.

Masakazu Kitahara: +1.

Resolution #1: Close remaining privacy and security issues.

Ivan Herman: clap, clap.

Dave Cramer: I think the spec is now much more informative/clear about some of these issues, so thanks everyone.

GeorgeK: +1.

Navigation Menu

Repository navigation

Obfuscation #1873

1.2. DRM and obfuscation

4.3. Obfuscation (issue epub-specs#1873)

11 remaining items

2. Updates to Obfuscation.

1. Close Privacy & Security Issues.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Obfuscation #1873

Description

Activity

mattgarrish commented on Oct 26, 2021

npdoty commented on Oct 26, 2021

mattgarrish commented on Oct 26, 2021

mattgarrish commented on Oct 26, 2021

iherman commented on Oct 27, 2021

1.2. DRM and obfuscation

iherman commented on Oct 29, 2021

4.3. Obfuscation (issue epub-specs#1873)

npdoty commented on Oct 29, 2021

mattgarrish commented on Oct 29, 2021

dlazin commented on Oct 29, 2021

npdoty commented on Oct 29, 2021

dauwhe commented on Nov 10, 2021

toshiakikoike commented on Nov 11, 2021

npdoty commented on Nov 11, 2021

11 remaining items

npdoty commented on Jan 28, 2022

mattgarrish commented on Jan 29, 2022

npdoty commented on Jan 29, 2022

mattgarrish commented on Jan 29, 2022

dauwhe commented on Jan 31, 2022

danielweck commented on Jan 31, 2022

npdoty commented on Jan 31, 2022

dauwhe commented on Jan 31, 2022

iherman commented on Feb 4, 2022

2. Updates to Obfuscation.

iherman commented on Feb 4, 2022

iherman commented on Apr 8, 2022

1. Close Privacy & Security Issues.

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions