Repository navigation
Section 5.2: centering, scaling, cropping #1283
Description
From EKR: https://lists.w3.org/Archives/Public/public-webrtc/2017May/0166.html
The advice here (center, scale, then crop) differs from a MUST in JSEP S 3.6.
If the original resolution exceeds the size limits in the attribute,
the sender SHOULD apply downscaling to the output of the
MediaStreamTrack in order to satisfy the limits. Downscaling MUST
NOT change the track aspect ratio.
Activity
[-]Section 5.2[/-][+]Section 5.2: centering, scaling, cropping[/+]fluffy commentedon Aug 23, 2017
Related to rtcweb-wg/jsep#828
stefhak commentedon Aug 29, 2017
FWIW, in the minutes from TPAC/Lisbon (search for 'crop' on that page) it says we decided on 'center, scale, crop'. That is also con you get to the discussion (and there is also a discussion in the end of #305 that seems to confirm that decision).
Slide 73 in https://www.w3.org/2011/04/webrtc/wiki/images/1/1e/WebRTCWG-2016-09-22.pdf
is the one shown when we discussed at TPAC (if memory serves).
9 remaining items
alvestrand commentedon Oct 16, 2017
I have changed my opinion on this.
We should make the spec simpler, not more complex.
The simplest answer, when faced with a situation where you can't rescale the video to fit the constraints, is what JSEP currently says: "NO".
No letterboxing, cropping or pillaring. Either you can scale the video to fit the constraints, or sending the video fails.
stefhak commentedon Oct 16, 2017
@alvestrand the 'constraints' you talk about are the ones imposed by the encoder and decoder (via imageattr, right? (We also have track constraints so I'm a bit confused).
Personally I also like JSEP's way of speccing this, but with my chair hat on all I want now is something that gets consensus.
Simple text for scaling issue
alvestrand commentedon Oct 16, 2017
@stefhak Yes. I looked at #1570 and think the requirement can be made even simpler - I cooked up #1636 to show what I mean.
To reiterate: On the call, I thought the complex proposal being floated there (parts of it by me) made sense. I've changed my mind, so I'm no longer part of the apparent consensus on Thursday.
stefhak commentedon Oct 16, 2017
@alvestrand I like #1636 - clearer than #1570.
BTW, do you read the spec in the same way as I do, that scaling down for transmission is allowed regardless of (mandatory) track constraints?
Say you have a track that is set (mandatory constraints) to exactly 1280x720, and it is possible to transmit anything between 1x1 and 640x480, then it would be OK to scale to 640x360 and transmit?
And if the remote application did
getSettings()on the received track it would say 640x360?fluffy commentedon Oct 16, 2017
alvestrand commentedon Oct 17, 2017
@fluffy no, I'm not OK with aspect ratio changes. You agreed in JSEP that aspect ratios should not change; we can't be OK one place and not OK in the other.
Besides, aspect ratio change loses information. If distortion's required, let the recipient do it.
@stefhak I see the constraints as controlling what the track delivers (defined as "track resolution" in JSEP). That's input to the transmission function, and the transmission function is permitted to scale down what it gets, as per JSEP.
Calling getSettings() on the recipient's track will return what came across. That's going to be 640x360 in the case you posit. I don't see any sense in returning any other number.
alvestrand commentedon Oct 17, 2017
Note: I see no reason not to propose the control and the relaxed MUST NOT requirement as an extension to the spec after CR. If users think it useful and implementors want to implement it, it can be added.
But I think the spec at CR is better off with writing cropping, letterboxing and aspect ratio change out of it.
stefhak commentedon Oct 17, 2017
@alvestrand I'm in agreement with the second part of #1283 (comment) (the one addressed to me).
alvestrand commentedon Oct 17, 2017
@stefhak wrt #1283 (comment) - my interpretation of the constraints text in MediaCapture is that a platform is allowed, but not required, to let getUserMedia() return a scaled and/or cropped version of what the physical camera produces. The spec talks about a range of possible configurations - platforms that do scaling & cropping offer a very large number of configurations; platforms that don't, don't. That's certainly how Chrome's code works (it does scale to fit with specified constraints, and opens the device in a mode that will allow it to satisfy the constraints by croping and/or scaling).
Hope that makes sense....
Add note about implementation feedback sought on #1283
stefhak commentedon Oct 18, 2017
In response to #1283 (comment): that is sort of what I expected, but that, to me, also indicates that this issue requires more implementation and use experience before resolving. As an example, say dimensions up to X by Y can be transmitted, and webrtc-pc forbids cropping to match imageattr. We could then have the situation that applying height and width constraints to the track to limit it to X by Y gives another result (cropped) than just feeding the unconstrained track to the sender.
stefhak commentedon Oct 23, 2017
I've opened w3c/mediacapture-main#495 on mediacapture-main, it is related to this discussion.