Repository navigation
Problem with XML content requirements and external entities: make MathML and SVG1.1 invalid #1338
Description
The current text on XML conformance says that an XML-based media MUST meet:
...
External identifiers MUST NOT appear in the document type declaration [XML]
...
This constraint apply regardless of whether the given Publication Resource is a Core Media Type Resource or a Foreign Resource.
However, that creates problems with SVG 1.1. The conformance rules of SVG1.1 require the presence of:
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" SYSTEM "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
The DTD itself, as described in the relevant appendix, is making a heavy use of external entities (via its usage of DTD Modularization). As a consequence, per the current document, no conform SVG 1.1 file is valid for EPUB 3.3.
Although the upcoming SVG 2 removed this requirement, there are lots of perfectly valid SVG 1.1 out there (SVG 2 is not yet a Recommendation, b.t.w., only a CR). Most notably, Adobe Illustrator generates SVG 1.1 files with those DTD-s in them.
A similar problem may occur with MathML 3, whose official DTD relies on the same approach (although MathML is more permissive and it does not require the usage of that DTD).
This problem was originally raised in w3c/epubcheck#1114.
Activity
iherman commentedon Oct 9, 2020
Note that, in w3c/epubcheck#1114 two possible ways forward were already proposed:
murata2makoto commentedon Oct 10, 2020
@iherman
No, it does not. The rules automatically add the DTD before validation.
The DTD heavily uses external parameter entities, which never occur in documents. But if memory is correct, it does not define any parsed entities.
iherman commentedon Oct 10, 2020
@murata2makoto
Checking it again, I stand corrected:-) The validity process requires the presence or the addition of the DTD. So my sentence:
should rather say:
But, taking into account that tools produce SVG files with the DTD, and they have the perfect right to do so, the problem is not less severe for us. (I have just ran into a different application, called draw.io, that can produce nice diagrams in SVG and adds the DTD.)
I believe you are right. But the presence of the external parameter entities in the DTD is enough to raise a red flag for EPUB.
murata2makoto commentedon Oct 10, 2020
@iherman
I do not understand. Since we do not allow external DTD subsets, external parameter entities are never encountered.
I think that you would like to use the output of SVG software that always embed references to the DTD at the W3C web site. But such references have caused troubles to the W3C web site in the past.
iherman commentedon Oct 10, 2020
Indeed. Such software implementations are correct in doing so, they exist, and our users may want to use them. Adobe Illustrator is probably the most prominent, and certainly very important example. I do not think we can expect, say, Adobe Illustrator users to manually edit their generated SVG files by removing that DTD reference.
I.e., this is a very practical deployment issue.
I do not know about the past, but the troubles are not in the present. My original issue came from the fact that most of the logos used in the W3C publications (e.g., all recommendations) in the past 3-4 years (at least) use SVG files that have those DTD-s, and all the Web Browsers accept those without further ado. Converting them into valid EPUB is not possibly without manually modifying the SVG files.
I suspect that this issue never came up in EPUB land so far because Reading Systems tacitly accepted such SVG files (as the core Web engines accepted them). What brought it to the fore is the much stricter newer version of epubcheck that rejects those files. (This started the whole discussion in w3c/epubcheck#1114 after all...)
I cc @rdeltour explicitly here, to call on the epubcheck implementers to give their opinion.
murata2makoto commentedon Oct 10, 2020
@iherman
I still do not think that this is a problem. Yes, human users or software always have to remove external DTD subsets, if any. IDPF prohibited them, since parsing documents containing external DTD subsets has caused unnecessarily network access.
mattgarrish commentedon Oct 10, 2020
But why are we putting this requirement on authors and not reading systems? It's a reading system optimization pushed onto authors to carry out when a developer could suppress any resolution of DTD references. Isn't that pretty standard anyway, as it's been a while since I processed XML documents but W3C used to blacklist you if you tried to continuously resolve their DTDs.
(And just for the record, I don't see that this affects MathML. MathML can only be embedded in HTML so can't have its own DTD declaration.)
iherman commentedon Oct 10, 2020
Well, this is where we disagree. I believe it is a problem that one of the official content format has an unnecessary restriction compared to its official definition.
But, actually, I do not understand your note, @murata2makoto. In your comment in w3c/epubcheck#1114 (comment) you seem to say that there is no such problem, that content with such DTD is o.k., and the bug is by epubcheck. In which case I believe we have to add some explanation in the spec.
murata2makoto commentedon Oct 12, 2020
@mattgarrish
I might have agreed if this proposal was made when EPUB 3.0 was being designed. But the charter for EPUB 3.3 does not allow any changes that might destroy existing reading systems. If this DOCTYPE statement occurs in an SVG document, some existing EPUB 3 reading systems might try to fetch an external DTD subset. This causes network access, which is slow or unavailable.
murata2makoto commentedon Oct 12, 2020
@iherman
My comment might not be clear. But I'm saying that the DOCTYPE statement in SVG must not be allowed by EPUB 3.3.
mattgarrish commentedon Oct 12, 2020
But the only thing that stops this from happening now is the idea that only perfectly valid publications are ever loaded into reading systems. That seems unlikely given that there are often ways to side-load content.
I agree we shouldn't cause backwards incompatibilities, but It would be interesting to survey whether moving the requirement to reading systems actually changes the way any have been implemented.
iherman commentedon Oct 12, 2020
@murata2makoto the backward compatibility issue is something we should check. You say:
and I am not sure such reading system actually exist. What I would think is that reading systems, so far, did not care about the DTD-s (surely all RS-s that relied on one of the web rendering engines) and this whole issue came to the fore because epubcheck reports this issue although it did not care about it before. In other words, I would expect there were a number of, technically, invalid EPUB content in circulation that were never flagged as such and that become invalid now.
In other words, it may well be that the practical deployment of EPUB content as well as RS-s were actually both ignoring a constraint in the document...
murata2makoto commentedon Oct 12, 2020
@iherman
It may well exist. If my memory is correct, a key member of IDPF argued that the DOCTYPE statement for SVG must not be allowed for this reason. This happened during the development of EPUB3.
llemeurfr commentedon Oct 12, 2020
74 remaining items
mattgarrish commentedon Oct 26, 2020
I find it doubtful. No offense, as I grew up on SGML, but we're partying like it's 1999 in this thread... :)
XSLT is the the more usual language of choice these days if publishers want to insert boilerplate, repeated strings, etc.
Your proposal seems similar to where we were somewhere way back in this thread (although I'm not sure how we disagreed on the wording and I'm not going to try and locate why). External identifiers are disallowed except for a controlled list and reading systems ignore (that has to be separately in the RS spec, though).
Works for me.
iherman commentedon Oct 26, 2020
I have just realized that we should not forget to add the ONIX DTD (or DTDs?) to our controlled list:
after all, it should be possible to add the ONIX metadata as part of the package...
iherman commentedon Oct 26, 2020
yep, we made a big circle...
mattgarrish commentedon Oct 26, 2020
Maybe no harm, but not sure how critical it is. My understanding is they're pushing to move everyone to 3.0 now and 3.0 records aren't supposed to be distributed with a doctype.
I don't believe ONIX records actually end up in EPUBs all that often, either.
iherman commentedon Oct 26, 2020
To be honest, I do not know. SOmething to be checked, possibly with Graham...
clapierre commentedon Oct 26, 2020
ONIX files are out-of-band metadata that publishers send out separate from the EPUB. a lot of times aggregators and distributors get only the ONIX files to determine what EPUBs they want. Also publishers can send these ONIX files with updates say pricing changes etc. without updating the EPUB itself.
iherman commentedon Oct 26, 2020
If we go down the road of listing some 'acceptable' XML DTD-s (see #1338 (comment)) then adding an ONIX DTD is painless. I realize that it does not happen often, but we should not make it impossible to add an ONIX file into the EPUB container...
murata2makoto commentedon Oct 27, 2020
I just created #1369, #1370, #1371, and #1372. Hopefully, all sub-issues are easier
murata2makoto commentedon Oct 27, 2020
Created another issue (#1373) for ONIX.
avneeshsingh commentedon Oct 27, 2020
+1 to have the solution that works for ONIX also. Most of times ONIX is external metadata but during ISO process of EPUB Accessibility, we observed interest in having ONIX accessibility metadata also as internal metadata.
murata2makoto commentedon Oct 28, 2020
This article is interesting. It mentions a massive DDoS attack on the W3C web site all the time.
iherman commentedon Nov 6, 2020
This issue was discussed in a meeting.
RESOLVED: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373View the transcript
Wendy Reid: we had resolutions at the F2F, and further discussions on github… and came to a happy place
Matt Garrish: #1368
Matt Garrish: where we ended up was…
… we put in an allowance for a specific set of external identifiers that we have put in an appendix
… we have SVG and MathML that are allowed to be used in content docs or in separate files
… and we made a restriction against external entities in the internal DTD subset
… so it prevents some security issues but eases authoring
… so we’ll no longer force people to remove SVG DTDs from tool-generated files
… I’m hoping this is it :)
Ivan Herman: tech comment
… in fact, the changes are such that
… makes possible something that I’m not sure we really use
… I can define as part of an internal entity something that won’t go out to the network
… I’m not sure if this feature is in use
… formal comment
… there was a formal resolution on the previous version; this PR slightly changes that
… can we get a formal resolution to merge, and also close a bunch of issues which were examples of the problem?
Proposed resolution: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373 (Wendy Reid)
Garth Conboy: +1
Matt Garrish: +1
Ivan Herman: +1
Charles LaPierre: +1
Matthew Chan: +1
Wendy Reid: +1
Brady Duga: +1
George Kerscher: +1
Laura Brady: +1
Bill Kasdorf: +1
Ben Schroeter: +1
Resolution #1: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373