Clarify language tag values

As noted in w3c/epubcheck#702, we're not completely clear about whether dc:language has to be well-formed or valid to bcp47 (sounds like it could be either right now):

The metadata section MUST include at least one language element with a value conforming to [BCP47].

https://www.w3.org/publishing/epub/epub-packages.html#sec-opf-dclanguage

Having had this discussion in web pub/pub manifest and ultimately coming to the conclusion that enforcing validity doesn't add a lot of value, I'd suggest we clarify the same requirement applies here. It's all epubcheck has ever enforced, so it won't affect existing content.

Note that the requirements for implementation are not necessarily the same as those for the authors.

Some related guidance in LTLI:

Specifications SHOULD require that language tags be well-formed.
Specifications MAY require that language tags be valid.
Specifications SHOULD require content authors use valid language tags.
Content validators SHOULD check if content uses valid language tags where feasible.

Right, but there isn't a specific requirement to use the language for any purpose in epub (or pub manifest), which is what makes a validity constraint somewhat excessive. As the LTLI document notes:

Features or functions that need to understand the specific semantic content of subtags are the main reason that a specification would normatively require valid tags as part of the protocol or document format.

The reading system may operate on the language, but otherwise is free to ignore it and any subtags.

@xfq

@xfq How mature is LTLI?

I think that we should completely follow LTLI or request changes to LTLI. Inconsistences between two W3C specifications are confusing to users. Moreover, they may cause a significant delay to EPUB 3.3.

The last publication was in 2015, and the document was stagnant for a while.

Recently, because of the development of Unicode Locales and some discussions about the Internationalization Best Practices for Spec Developers document, we restarted our work on LTLI and added a lot of new content. We plan to publish a new WD soon, and feedback is of course welcome.

FWIW, I noticed that the following is “valid” for EPUB:

    <dc:language>en-US</dc:language> <!-- Primary language. -->
    <dc:language>pi-PI</dc:language> <!-- Secondary languages. -->
    <dc:language>foo-BAR</dc:language>

While these strings match the regex used by XSD [a-zA-Z]{1,8}(-[a-zA-Z0-9]{1,8})* they’re not valid RFC 3066 language identifiers. There’s also the problem of valid language identifiers (like pi (Pāli) or la (Latin)) which do not have ISO 15897 locale identifiers (and are not currently included in the Unicode CLDR project project).

We’ve made it a rule that Bookalope generates only valid RFC 3066 language identifiers, skips over invalid ones, but tolerates missing locales for a language.

@jenstroeger

@jenstroeger Why are you referring to RFC 3066? It's way out of date. The latest incarnation of that is BCP 47.

@r12a

@r12a because I sometimes get lost in the jungle of specs and updates and invalidations and supercessions… thanks for the hint!

The issue was discussed in a meeting on 2021-01-07

no resolutions were taken

View the transcript

1. Clarify language tag values.

See github issue #1325.

Wendy Reid: there's an open issue in epubcheck repo re. dc:language, specifically, whether it has to be well formed or valid.
… currently we require one conforming dc:language value in the OPF.
… some earlier discussion resulted in conclusion that well formed dc:language value wasn't of great use.
… this is in line with other specs.
… by well formedness, we are referring to bcp47.

Garth Conboy: is there a specific proposal?.

Wendy Reid: Matt didn't come up with new language.
… but the gist, i think, was to drop the MUST to a SHOULD.

Ben Schroeter: is it a should in LTLI?.

Wendy Reid: yes, and the idea is to align with that language.

Ben Schroeter: there are a11y implications to this change....

Wendy Reid: currently epubcheck doesn't check the value of dc:language, only that you have one.

Ben Schroeter: and under a SHOULD, epubcheck functionality would remain the same.

Garth Conboy: I thought that a SHOULD meant that epubcheck would only throw a warning?.

Wendy Reid: acknowledge point about a11y, but the a11y guideline could be updated to require a valid bcp47 language declaration.
… RS also has some part to play in this, as it may or may not honor the language attribute value, vs preferring what the ONIX says, etc..
… clarifying the issue in epubcheck, the open issue is that epubcheck does not validate the content of the dc:language, only that it is present and is well formed.
… and making this a SHOULD in our spec would not change that behaviour in epubcheck.

Brady Duga: reviewing what the spec actually says.

Wendy Reid: so the spec actually doesn't make a reference to "well formedness", only "valid".

Marisa DeMeglio: reviewing Matt's comments on this issue.
… reviewing the epubcheck codebase, there are checks for both MUST assertions and SHOULD assertions.
… so I think Garth_ is right about the epubcheck warning.
… so if we change the spec to say SHOULD, epubcheck would be expected to throw a warning, not an error; but really it doesn't change the amount of work required for epubcheck, so even a SHOULD could be unrealistic..

Wendy Reid: as opposed to the current behaviour, where it throws an error only if dc:language is missing completely.
… and does nothing otherwise.

Ben Schroeter: we should clarify what Matt is advocating here: either 1) downgrading from MUST to SHOULD, or 2) removing it completely.

Wendy Reid: I think we should give authors something to work with, rather than being silent, but maybe Matt should come up with the specific revision to the spec.

Responding generally to what I see in the meeting minutes:

epubcheck has never checked that the language codes are valid, only that they are well-formed.
to check validity, we would need to maintain a database of all valid language tags, which is probably not realistic to do for epubcheck.
the epub specification doesn't require reading systems to do anything with the dc:language values - the language of the package document and content documents is set by xml:lang/lang attributes. I don't believe there are accessibility implications in keeping things as they are.
we only require well-formed values for the pub manifest and audiobooks specifications. It's metadata for displaying to the user or maybe doing some optimizations; the publication doesn't break if the language code isn't valid. Reading systems will just ignore the value if they don't recognize it or have no use for it.

The W3C i18n self-review also favours well-formedness:

The word "valid" has special meaning in BCP 47. Generally "well-formed" is a better choice.

https://www.w3.org/International/techniques/developing-specs.en?open=resource&open=sec_lang_values#sec_lang_values

I'm not claiming to be an expert in these matters, but these are some of the considerations that went into the similar decision for pub manifest.

I don't believe we can apply a different standard to epub now, either, since we'd very likely create a backwards incompatibility by making validation stricter (assuming it ever got implemented). I believe well-formedness is our only realistic option.

@mattgarrish

@mattgarrish

the epub specification doesn't require reading systems to do anything with the dc:language values

But RSes still rely on dc:language to infer things like the list of fonts to load for the publication, the user settings to provide, and the xml:lang attribute to append if it is missing in a document. See https://github.com/readium/readium-css/blob/master/docs/CSS16-internationalization.md for details.

We believe that the EPUB spec is missing clarity on this aspect. But it may be too late to add anything but recommendations to EPUB authors.

I don't believe there are accessibility implications in keeping things as they are.

But a wrong evaluation of the primary language leads to pronunciation issues in TTS (e.g. fr-FR vs fr-CA).

@llemeurfr

I am not sure I understand this comment, @llemeurfr :

But a wrong evaluation of the primary language leads to pronunciation issues in TTS (e.g. fr-FR vs fr-CA).

In terms of our original issue: even if the validity is required, this problem remains. After all, both of these are valid language tags...

@iherman

@iherman you're right, my comment out of scope for this exact issue: I'm thinking about opening a specific issue about the primary language in relation to the order of dc:language properties, and the declaration of a script in the same dc:language property.

to check validity, we would need to maintain a database of all valid language tags, which is probably not realistic to do for epubcheck.

The number of all possible valid language tags is very large indeed, because several types of subtag can be combined in many ways, but the list of valid subtags is not so large, and their allowed combinations are pretty easy to validate. For example, see my tool at https://r12a.github.io/app-subtags/index?check=de-419-DE-alt which runs validation on the subtag de-419-DE-alt. I'm not arguing for validation here, just addressing the point above to say that you don't need a database of all valid language tags. Hth.

But a wrong evaluation of the primary language leads to pronunciation issues in TTS (e.g. fr-FR vs fr-CA).

The dc:language isn't used by tts, or shouldn't be. The only accessibility case I've heard for it is to preload dictionaries but that's just an optimization.

The lang/xml:lang attributes are used for voicing the text content and those are checked separately for accessibility conformance.

Another use of dc:language is to provide guidance to the reading system if the page-progression-direction attribute is not specified.

In this case, the reading system SHOULD choose a default page-progression-direction value based on the first language element.

I've written a test with dc:language="ar" and no page-progression-direction. Thorium and iBooks do the right thing, and page RTL.

@mattgarrish

@mattgarrish I think the original issue has been solved. dc:language is to well formed. There is an outstanding PR (#1510) for the xml:lang value, but that is a slightly different issue.

There is a comment on tts's usage of language, which sounds to me like a different issue. Would it be possible to raise a separate issue on that, and close this one? @dauwhe @wareid @shiestyle ?

Would it be possible to raise a separate issue on that, and close this one?

I think we can close this without another issue. The RS specification already says:

The language(s) of the EPUB Publication specified in language elements are advisory. Reading Systems MUST use the language information associated with a resource to determine its language.

@dauwhe

I've written a test with dc:language="ar" and no page-progression-direction. Thorium and iBooks do the right thing, and page RTL.

@dauwhe And what happens if you run that test with the language set to az or some other language that uses more than one script, and the text uses the Arabic script?

And what happens if you run that test with the language set to az or some other language that uses more than one script, and the text uses the Arabic script?

FYI, this problem is picked up on in #1482

Navigation Menu

Repository navigation

Clarify language tag values #1325

1. Clarify language tag values.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Clarify language tag values #1325

Description

Activity

xfq commented on Aug 18, 2020

mattgarrish commented on Aug 18, 2020

murata2makoto commented on Oct 2, 2020

xfq commented on Oct 3, 2020

jenstroeger commented on Oct 23, 2020

r12a commented on Oct 23, 2020

jenstroeger commented on Oct 23, 2020

iherman commented on Jan 8, 2021

1. Clarify language tag values.

mattgarrish commented on Jan 11, 2021

llemeurfr commented on Jan 11, 2021

iherman commented on Jan 11, 2021

llemeurfr commented on Jan 11, 2021

r12a commented on Jan 11, 2021

mattgarrish commented on Jan 11, 2021

dauwhe commented on Jan 21, 2021

iherman commented on Feb 15, 2021

mattgarrish commented on Feb 15, 2021

r12a commented on Feb 15, 2021

mattgarrish commented on Feb 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions