Repository navigation
OCF Document: RFC3986 or RFC3987? #808
Description
(I know this is a huge can of worms, and I regularly get it wrong; maybe it is the case this time, too...)
The issue is what the allowed characters are for file names. More specifically, what the allowed characters are for the value of full-path in the container.xml file.
- In Section 3.3, the reference is to RFC3986 (URI) and RFC3987 (IRI) are both listed. Which sounds about right; this means a file name of the form
/téléphoneis all right, although this would not be acceptable via RFC3986. (Although I wonder whether the reference to RFC3986 is necessary in the first place; in this respect RFC3987 supersedes RFC3986, doesn't it) - In 3.5.3.1, towards the end, only RFC3986 is reference in the defintion of
full-path. This seems to be in contradiction with what is in Section 3.3 - Section 3.4 seems to give a specific definition for file names; it is not clear to me whether this is necessary. Isn't it enough to refer to the same RFC3987 (well, the
pathportion thereof) to avoid mixup? Isn't it enough to quote RFC3987 and, if necessary, list the possible restrictions (I have not checked which of the characters listed in the 4th bullet point are excluded from the path segment of an IRI anyway) when it comes to file name? (Yes, of course, we have to refer to the last portion of an IRI path as the file name.) - And here is the rub: I think there is a discrepancy at this moment. If I correctly interpret either RFC3986 or RFC3987, the space (i.e., U+0020) character is excluded from the path, whereas Section 3.4 does not exclude it. This means that the examples are also wrong, because
full-path="EPUB/Great Expectation.opf"is not a valid value for@full-path, although it is indeed a valid file name per Section 3.4:-(
As i said, it is a can of worm, and one of you guys may prove me wrong in my interpretation... But if I am right, my proposal would be:
- Use RFC3987 only in the references (with a possible note relating it to RFC3986)
- Reduce section 3.4 by saying that a file name should be conform to the path of RFC3987, minus a number of characters (if necessary, ie, check whether all those entries are necessary)
- Change the examples...
Activity
mattgarrish commentedon Aug 16, 2016
I don't believe 3987 supersedes 3986; more that it builds on it. The base URI is defined in 3986, for example. 3987 only makes reference to using the algorithms in 3986 in the relative IRI references section. ODF does similar, which I expect is what OCF drew on (see, for example, http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os-part3.html#__RefHeading__752821_826425813).
The space question is interesting. Most of this prose goes back to EPUB 2.0, and I can't recall it being questioned except for some additions to the list in 3.4.
I just tested percent encoding a space, and epubcheck and Readium couldn't make sense of the rootfile. They do handle unencoded spaces, though. I wonder if the example has come to influence implementations to the point where if we enforce what is supposed to be done what problems will it cause for existing implementations and content?
But this isn't my comfort zone, either.
iherman commentedon Aug 17, 2016
Hm. I am even more lost at this point, probably because I never looked at the OASIS document before. But what the reference you give seems to say that ODF relies solely on 3987 which reinforces my first point, actually: there is no real reasons why our OCF document should refer to 3986 at all!
iherman commentedon Aug 17, 2016
To be more precise: 3987 does refer to 3986 for the various BNF constructions et al. But that is internal to 3987; our own starting point should be 3987.
iherman commentedon Aug 17, 2016
On the issue of space characters, I have made some extra research, for reference.
RFC 3987 says at Relative IRI references:
Earlier in the RFC3987 it says:
The limitations it refers to are in the ABNF Rules. The important entry is the reference to
ipchar, which allows for percent encoded characters, delimiters, and unreserved characters. The next important point is the latter, which isThe last entry is a bunch of extra Unicode characters, irrelevant here; The next point to look at is the definition of ALPHA. The strange thing is that
ALPHAis not formally defined in 3987 as Unicode ranges, but it refers to 3986, where it says (again without Unicode ranges?):I.e., ALPHA does not include the space character. What this means is that the space character is definitely not allowed as a relative URI in RFC3987 (or RFC3986).
More pragmatically, I believe this makes sense in practice. While it is true that a file name on my Mac may include a space character, I am not sure this is true on all Linux systems or Windows (mainly in Windows 10 it is fine, never checked). I know that whenever I push a file from my machine to my Web server, I am careful in exchanging the space character to, say, a
_character. Using this restriction for EPUB sounds like a safe solution anyway.murata2makoto commentedon Aug 17, 2016
I wonder what I have been doing....
First, I think that we should explicitly say that a file or path name matches isegment-nz in RFC 3987. Or, we might want to even use ipath-rootless (but have to disallow a trailing empty segment.)
Second, W3C has Legacy Extended IRIs (LEIRIs). It shows a list of characters allowed by legacy variants of IRIs but disallowed by RF 3987.
Some characters in this list are explicitly disallowed by OCF 3.1 but they do not have to be. They are shown below:
Third, I believe that the characters in the next itemized list are already disallowed by RFC 3987 and do not have to be menteiond in OCF 3.1.
Fourth, we disallow some characters, although they are allowed by RFC 3987. I do not think that we should lift this limitation.
Fifth, we should explicitly allow the use of the space chracter, although we use RFC 3987 as a basis.
murata2makoto commentedon Aug 17, 2016
As far as I know, the only reason for referencing RFC 3986 is to borrow some terms which are not even mentioned in RFC 3987.
iherman commentedon Aug 17, 2016
Just one comment…
iherman commentedon Aug 17, 2016
Let me roll back a little bit: I have just checked and it seems that space is allowed in Linux; my bad. However, if one uses a URL to access a file that is in the 'exploded' version of an EPUB instance, that URL MUST use %20 instead of a space per existing RFC-s. Ie, although my argument on Linux is wrong, I still keep to my conclusion: it may backfire on us later if we allow spaces in file names within a publication...
laudrain commentedon Aug 17, 2016
I have to mention that spaces in files names inside the EPUB package are a subject for rejection of EPUB files by distributors today.
iherman commentedon Aug 17, 2016
At this point, I believe that a radical re-write of the OCF document in terms of references may be too much to do; let us leave this (and flag this for clean-up!) for a later version. But (also in view of @laudrain's comment) I believe the issue of the space character is real and a bug we should not perpetuate.
To solve the issue for EPUB 3.1, I would propose the following: change all the examples in the document (there quite some) by removing the space character, or replacing it with the
_character. Leave everything else unchanged, except that we may add a note to the document somewhere, as a warning, saying something like:GarthConboy commentedon Aug 26, 2016
+1 to Ivan's above. I don't think we should wade into spec changes beyond the examples.
laudrain commentedon Aug 29, 2016
+1 for me, just change the examples.
9 remaining items
iherman commentedon Jan 19, 2019
This is certainly what newer W3C documents refer do (similarly to the references to the WhatWG HTML spec). The W3C and the WhatWG are hammering out an agreement on working together, and the URL spec is definitely part of that agreement.
I am not sure why we look at a reference implementation here. I would think what really count is the implementations in browsers rather than node.js and, in this sense, what counts is the relevant URL test suite...
llemeurfr commentedon Jan 20, 2019
I'm curious: most OSes (Linux, Windows, MacOS) accept spaces in file names, and URI or IRI referencing those files % encode spaces. Nothing special here.
Therefore I don't see why section 3.4 should refuse spaces in file names and blocking this at the level of EPUBCheck does not seem right to me.
mattgarrish commentedon Jan 20, 2019
Ya, they just have a way of tripping up command line tools, piping operations, etc.
Making them illegal is probably a bit much, but I tested back to the epubcheck 3.0.1 release from 2013 and it has been emitting a warning about their use since at least then, so adding a warning to the specification isn't really changing reality in any way.
But if we decide they should be allowed, then conversely epubcheck needs to be modified.
dauwhe commentedon Sep 30, 2020
Yes, to retain compatibility with HTML! For example, here's how HTML defines the
srcattribute forimg:dauwhe commentedon May 4, 2021
I think we need to talk about the space character, and whether we can move to using the WHATWG URL spec.
mattgarrish commentedon May 5, 2021
I don't think this is an issue anymore.
There was a time when the URL specification only defined parsing of URLs (as complained about in the article mentioned above), but it now includes a section that defines the syntax for valid URLs. Looks like that was added sometime around 2017.
Without that syntax, we'd have lost validation, and that would have made for an interoperability mess.
iherman commentedon May 7, 2021
The issue was discussed in a meeting on 2021-05-07
List of resolutions:
View the transcript
5. OCF Document: RFC3986 or RFC3987?
See github issue #808.
Dave Cramer: basically the question is generally how we define URLs in our spec
… essentially how URLs work on the web is determined by the WHAT-WG
Dave Cramer: if we change our spec to refer to that instead of these RFCs then we are better off