Repository navigation
Can you tts an image in media overlays? #1745
Description
According to the embedded media section, you can reference images but you must have an audio clip for them because they contain no audio.
But what about images in sections that are TTSed? If we expect the reading system will use the alt text, then why can't you omit an audio element so images fall over into that category?
Or maybe what I'm wondering is why are images included in the embedded media section at all?
As far as I remember it used to be called "Embedded Audio and Video" and we only changed it because images were also listed as an example of embedded media in the prose of the section. Was that the wrong fix and we should have just deleted images from the section?
/cc @marisademeglio
Activity
OriKuttner commentedon Jul 1, 2021
danielweck commentedon Jul 1, 2021
We recently had a discussion in the Thorium issue tracker about an EPUB Media Overlays feature which is not commonly used / authored, and as far as I know rarely implemented: i.e. "filling the gaps" with TTS synthetic voice when pre-recorded human narration is not available (SMIL text references without audio clips).
We implemented support for this feature in the original Readium SDK, but so far in Readium "2" we haven't tackled this, due to lack of real-world adoption / usage.
The use-case seems obvious at a UX surface level (which is why this feature was added in the specification), but looking into practical details I think there are some concerning downsides, such as the inevitable inconsistencies between a professionally-prerecorded audio track and an arbitrary system-dependent synthetic voice (e.g. volume levels, speech rate, voice selection, etc.). These aspects are largely underspecified and it leaves both reading system implementers and end-users with vague expectations.
Rather than fixing this part of the spec., I would be in favour of deprecating the feature, which to my knowledge is poorly supported in the real world. I don't think it would be a big loss, if any at all.
cc @llemeurfr
danielweck commentedon Jul 1, 2021
Spec links:
https://www.w3.org/publishing/epub/epub-mediaoverlays.html#sec-text-to-speech
https://w3c.github.io/epub-specs/archive/epub32/spec/epub-mediaoverlays.html#sec-text-to-speech
danielweck commentedon Jul 1, 2021
Thorium issue tracker discussion: edrlab/thorium-reader#1515 (comment)
mattgarrish commentedon Jul 1, 2021
@danielweck are you suggesting we deprecate only the TTS part of media overlays or also the automatic playback of audio/video elements? Is the latter used any more often than the former?
As @iherman has noted elsewhere, it's confusing to use a "text" element to initiate audio/video, although I understand we don't have the corresponding smil elements for simplicity.
danielweck commentedon Jul 1, 2021
Good point @mattgarrish I also think that the embedded audio/video playback feature is in the real world not widely implemented (authors and reading systems), if at all (?) - besides Readium SDK first-gen.
I guess EPUBTest.org has some statistics about that?
mattgarrish commentedon Jul 1, 2021
Unfortunately, no. Neither the old site nor the new one dives into either TTS playback or audio/video initiation.
Given the paucity of audio/video content in EPUBs generally, and the relative low use of media overlays, I don't imagine the feature has been used much, if at all, in the real world.
geovls commentedon Jul 2, 2021
@danielweck perhaps people who use the DAISY format would find this feature useful in an epub book.
avneeshsingh commentedon Jul 8, 2021
Media Overlays specifications is derived from DAISY full text full audio specifications which specifies control of TTS reading through SMIL.
Different accessibility needs requires different reading strategies. For example alt text is skipped for people with Dyslexia, but it is essential for visually impaired. And TTS controled through SMIL gives ability to provide right reading experience to people with different needs.
mattgarrish commentedon Jul 8, 2021
Without any clear support information, it might be better to focus first on ensuring the two sections do not pose a contradiction to each other.
We seem to have three concepts rolled up into two sections:
audioelement means to start playback of them.altattribute (specifically images).I think the current embedded media section describes the first part fine, but it gets confusing around the second. The reading system specification is clearer that only audio and video get initiated, but all the authoring spec says is:
The problems with this statement are that:
Assuming those aren't the outcomes we intended, perhaps we can rewrite this to:
"When a text element references audio or video, the audio sibling element is OPTIONAL. The Reading System will initiate playback of the media."
We might even want to add subsections to describe audio/video and image handling separately.
That only leaves a clarification in the TTS section for images, which may only require tweaking this sentence to:
"A Media Overlay par element MAY omit an audio element when its text element references
a non-media elementtext or an image."mattgarrish commentedon Jul 8, 2021
For additional clarification, I'm not suggesting it isn't important to have, for example, an audio description track, but this seems like a requirement for the accessibility specification. I'm just not sure we want to completely forbid playback of such video.
llemeurfr commentedon Jul 8, 2021
To stay focus on the TTS feature when no audio sibling is set, the comments made earlier by @danielweck stay true, i.e. this rule is vaguely specified, is causing user experience issues, is not widely used by authors and is not implemented in RS (except first-gen Readium SDK). Hence the request for deprecation.
Imho, a good media-overlay will contain the audio alternative to the alt-text of an image.
Re @avneeshsingh comment, EPUB 3, contrary to DAISY, does not (clearly) specify the control of TTS via SMIL. Apps like Thorium split ebooks in comprehensive sections via their own algorithm.
avneeshsingh commentedon Jul 8, 2021
I agree with @mattgarrish for focusing on clarifying the specifications at this stage.
The features which have lack of support will be removed from normative specs when we go through the CR stage.
mattgarrish commentedon Jul 8, 2021
Right, there seem to be at least three possibilities we'd have to consider: complete removal, deprecation or making informative. It's a separate issue from what I wanted to fix here.
iherman commentedon Jul 9, 2021
The issue was discussed in a meeting on 2021-07-08
View the transcript
4. Media Overlays and Images
See github issue #1745.
See github pull request #1753.
Wendy Reid: can you TTS an image in a media overlay document?
… question comes down to, in MO we make mention of the fact that there should be an audio alternative to media (i.e. audio, video, AND images)
… and whether TTS and alt-text could fill this requirement
Dave Cramer: this diff preview seems to be working better for me
… basically it splits out embedded images separately from audio and video
… says that audio for embedded images is optional, and TTS reading out alt-text will be used as a fallback
… to me this makes sense
Brady Duga: not sure it makes sense to me. The purpose of MO is to have narrated text associated with the elements of the MO. Not sure why you would reference the image in MO but not have narration, and instead rely on the TTS and alt-text fallback
Matt Garrish: in embedded images we include ability to embed audio, video, and images. This is so that playback of audio, video, etc. could be started via the MO
Brady Duga: my reading of spec is that audio element is required for every text element
Matt Garrish: so what happens when the "text" element is an image?
Brady Duga: well i don't think it would be valid to include a reference to an image without audio. Spec says audio must be included unless it is "content intended to be rendered via TTS". Not sure if image (within a div) counts.
Matt Garrish: in HTML normally TTS would pick up the alt-text of the image
… so why can't the TTS be used to voice the alt-text of the image in this case
Brady Duga: not sure what normal TTS behaviour is. Perhaps this is why use of this fallback TTS feature is rare
Matt Garrish: My guess is that the purpose of having TTS as a fallback to narration is that you might not want to get a professional narrator to narrate some parts of a MO book. e.g. backmatter
… so you could leave this to TTS
Dave Cramer: I think we should wait for more comment. From Marisa, from NNELs, from other users of MO in the wild
marisademeglio commentedon Jul 21, 2021
Agree that it is reasonable to include images with alt text alongside text in the category of "things that can be TTS'd by a reading system". I also think that using TTS in conjunction with MO is more a part of the reading experience than the EPUB creation process. For reference, I commented along similar lines here.
mattgarrish commentedon Sep 15, 2021
Closing this issue as it was addressed in the referenced pull request. It should have been closed after @marisademeglio's comment.