stereo3d: detect layout signalled in the video stream (frame packing SEI, st3d) - #18490

Follow-up on scope, before it comes up in review.

The description says no container-tagged file changes behaviour. That holds when the container tag and the in-band signal agree, which is the normal case, but not when they disagree:

mkvmerge -o conflict.mkv --stereo-mode 0:3 sei_only.mp4   # tag: top-bottom, SEI: side-by-side

  before: stereo-in=ab2l    (container tag)
  after:  stereo-in=sbs2l   (in-band signal)

This is deliberate — it mirrors the frame rotation handling a few lines below in fix_image_params(), where the frame's value also takes priority over the container — but it is a behaviour change on a tagged file, so it should be stated rather than implied. If you would prefer the container tag to always win when both are present, that is a one-line reordering and I am happy to make it.

Why are you so rude? Your single accepted PR is larger and contains no technical nothing, it's nothing but bad words on people trying to collaborate because, yes, it's a people dream (humans project into others what they are) to be listed as contributor, not to actually contribute with fixes....

Your single accepted PR

$ gol --author="Nicolas F" | wc -l
28

?

The rude part here is you letting clod flood the maintainers with redundant text.

Why are you so rude? Your single accepted PR is larger and contains no technical nothing, it's nothing but bad words on people trying to collaborate because, yes, it's a people dream (humans project into others what they are) to be listed as contributor, not to actually contribute with fixes....

Please read the contribution guidelines. You need to show you're capable of responding to reviews with human-written responses, which the AI-slop written commit messages implies you're not.

Why are you so rude? Your single accepted PR is larger and contains no technical nothing, it's nothing but bad words on people trying to collaborate because, yes, it's a people dream (humans project into others what they are) to be listed as contributor, not to actually contribute with fixes....

Please read the contribution guidelines. You need to show you're capable of responding to reviews with human-written responses, which the AI-slop written commit messages implies you're not.

AI-slop - define this term, please? What about human slop, like being rude to a genuine contribuition proposal? You are assuming that because I leveraged AI to write the PR is slop? HandBreak actually verifyed my claims before that...

This message might contain english errors because it's not my main language and mainteners prefer wrong english than AI assisted writting.

How often the frame-packing SEI is actually attached to a frame?
Runnable evidence for the "why the first frame was not enough": the frame_packing_arrangement SEI reaches libavcodec as AV_FRAME_DATA_STEREO3D on keyframes only, so a player that reads the side data once and then lets fix_image_params() run again on the next frame loses the layout.
That is what the PR's second commit (f_decoder_wrapper: keep in-band stereo 3D layout across frames) fixes.

It needs no patched mpv and no sample downloads.
Stock ffmpeg, four commands, about ten seconds.

The block

# Build a 10 s SBS clip with the frame-packing SEI and a keyframe every 48 frames.
ffmpeg -v error -f lavfi -i "testsrc2=size=960x1080:rate=24:duration=10" \
       -f lavfi -i "smptebars=size=960x1080:rate=24:duration=10" \
       -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" \
       -c:v libx264 -crf 20 -pix_fmt yuv420p -g 48 \
       -x264-params frame-packing=3 sei_multi.mp4

# Same thing without the SEI, as a negative control.
ffmpeg -v error -f lavfi -i "testsrc2=size=960x1080:rate=24:duration=10" \
       -f lavfi -i "smptebars=size=960x1080:rate=24:duration=10" \
       -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" \
       -c:v libx264 -crf 20 -pix_fmt yuv420p -g 48 no_sei.mp4

# How many frames actually carry AV_FRAME_DATA_STEREO3D?
for f in sei_multi.mp4 no_sei.mp4; do
  ffprobe -v error -select_streams v:0 -show_frames "$f" > /tmp/f.$$
  printf "%-16s frames=%-5s keyframes=%-4s stereo3d_side_data=%s\n" "$f" \
    "$(grep -c '^media_type=video'          /tmp/f.$$)" \
    "$(grep -c '^key_frame=1'               /tmp/f.$$)" \
    "$(grep -c '^side_data_type=Stereo 3D'  /tmp/f.$$)"
  rm -f /tmp/f.$$
done

# And which frames they are.
ffprobe -v error -select_streams v:0 -show_frames sei_multi.mp4 | awk '
  /^media_type=video/{n++} /^key_frame=1/{k[n]=1} /^side_data_type=Stereo 3D/{s[n]=1}
  END{for(i=1;i<=n;i++) if(k[i]||s[i]) printf "frame %3d  keyframe=%d  stereo3d=%d\n", i, (k[i]?1:0), (s[i]?1:0)}'

Output

sei_multi.mp4    frames=240   keyframes=5    stereo3d_side_data=5
no_sei.mp4       frames=240   keyframes=5    stereo3d_side_data=0
frame   1  keyframe=1  stereo3d=1
frame  49  keyframe=1  stereo3d=1
frame  97  keyframe=1  stereo3d=1
frame 145  keyframe=1  stereo3d=1
frame 193  keyframe=1  stereo3d=1

Same counts and the same frame indices across three major FFmpeg versions, so the result is a property of the file rather than of one build.

Why it is built this way?
Two details do real work and are easy to leave out.
The negative control. no_sei.mp4 is the same clip encoded without -x264-params frame-packing=3 and reports stereo3d_side_data=0. Without it, a reader cannot tell whether the counter detects the SEI or simply always prints a number.
The keyframe interval. -g 48 over 10 seconds gives five keyframes. A 2-second clip has exactly one, and 1 of 1 cannot distinguish "one per keyframe" from "one per file" — which is precisely the distinction the second commit turns on. Frames 1, 49, 97, 145 and 193 settle it: the side data lands on every keyframe and on nothing else, consistent with frame_packing_arrangement_repetition_period = 1 ("persists until the next IDR").

Another data point on the approach, in case it is useful to the review: Kodi already does this, and has for years.

CDVDVideoCodecFFmpeg takes the stereo mode from the decoded frame's metadata — FFmpeg's H.264 decoder sets a stereo_mode entry from the frame-packing SEI, with the Matroska strings as values (left_right, top_bottom, …):

xbmc/cores/VideoPlayer/DVDCodecs/Video/DVDVideoCodecFFmpeg.cpp:1043
AVDictionaryEntry* entry = av_dict_get(m_pFrame->metadata, "stereo_mode", NULL, 0);

Verified by running Kodi 21.2 on a file whose only 3D signal is the SEI (neutral filename, no container tag): autodetected stereo mode for movie mode left_right.

Sample to check against:

ffmpeg -f lavfi -i testsrc2=size=1920x1080:rate=24 -t 10 -c:v libx264 \
  -x264-params frame-packing=3 -pix_fmt yuv420p sbs.mp4
ffprobe -show_frames -read_intervals %+#1 sbs.mp4 | grep side_data_type   # Stereo 3D
ffprobe -show_entries stream_tags=stereo_mode sbs.mp4                     # nothing

And on why it matters outside players: on 2011–2012 Sony BRAVIA sets this SEI is the only signal that makes the display engage 3D by itself.

Measured through two independent DLNA servers, byte-exact: the SEI-carrying file engages 3D, the identical file with the SEI removed plays flat.

Look, I am not that good with words and english can be complicated to convey, I therefore disclosured the leverage of AI on the PR. The new rule also landed one day after I filed the PR and merge request.
Please, do not stay out of the 3D video playback fix just because I am Brazilian and used AI to help write some text, look at the content for what it is and think on all future generations that will enjoy seamless 3D content on VR and old school 3D hardware.

@kasper93 urging for a neutral technical review here.