Add flex display support (resizable virtual display) - #6772

Glitches

As a result, the system may briefly render at the old size on the new surface, or at the new size on the old surface.

It can be avoided almost entirely with:

diff --git a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
index ebe424f95..e2dca2b44 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
@@ -19,6 +19,7 @@ import com.genymobile.scrcpy.wrappers.ServiceManager;
 import android.graphics.Rect;
 import android.hardware.display.VirtualDisplay;
 import android.os.Build;
+import android.os.SystemClock;
 import android.view.Surface;
 
 import java.io.IOException;
@@ -265,6 +266,7 @@ public class NewDisplayCapture extends SurfaceCapture {
         if (virtualDisplay == null) {
             startNew(surface);
         } else {
+            SystemClock.sleep(300);
             virtualDisplay.setSurface(surface);
         }
 

But it's not a really good solution… Even if instead we wait explicitly for the display event before setting the surface, it still glitches.

Oh, it's not necessarily a Surface issue (or at least not only): when the virtual display is resized, there is a fade-in/fade-out animation between the previous content (old size) and the new content (new size), so the old size is scaled and blended into the resized content for a few frames, causing a visual glitch.

It can be seen by recording and replaying in slow motion:

scrcpy --new-display=1024x768 --start-app=com.android.settings --flex-display --record=file.mp4

Then trigger one resize, then replay the file with VLC or mpv in slow motion (use [ to slow down and ] to speed up).

That's great! I couldn't find any major issues, at least not ones caused by the apps or my phone not handling virtual displays well.

Can it also be used on main display too outside virtual display? Love it btw, Huge thanks.

Can it also be used on main display too outside virtual display?

Nope, the main display cannot be resized that way (and we don't want to change the resolution of the physical display).

The black bars in landscape mode on main display on pc won't go away I guess right even at the same resolution as phone?

I have a dumb question about glitches (#6772 (comment)), it's probably nonsensical: Could you intentionally drop frames between requesting a resize and setting the new surface? Is it equivalent to your tentative sleep? Would it lead to stuttering (instead of glitches) if it takes too much time? And would the next few frames still show the same issue (#6772 (comment))?

By the way, the documentation of VirtualDisplay.setSurface() says:

It is still the caller's responsibility to destroy the surface after it has been detached. 

Thus maybe you should do this in NewDisplayCapture.start():

            Surface oldSurface = virtualDisplay.getSurface();
            virtualDisplay.setSurface(surface);
            if (oldSurface != null) {
                oldSurface.release();
            }

But I'm really not sure because I don't fully understand how and when virtual displays are (re)created.

[EDIT] The doc is not really clear because "detached" is when you set the surface to null, but what happens when you just change it? Is it released automatically?

For physical displays, instead of changing physical display resolution, maybe it can change capture resolution instead. Would be useful when streaming over Internet, or reduce CPU/RAM usages when multiple instances are open (tiled, each one is small)

@anotheruserofgithub I really appreciate the detailed feedback, it's super helpful. I fixed in a new version.

Thus maybe you should do this in NewDisplayCapture.start()

The surface is owned by SurfaceEncoder, it is already released there.

It seems forcing an OpenGL filter improves the result (not sure if it's just a side effect), but I think it is still not perfect:

diff --git a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
index 851be5579..8ad77fb7a 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
@@ -213,6 +213,10 @@ public class NewDisplayCapture extends SurfaceCapture {
         //                    = DISPLAY_FILTER_MATRIX⁻¹ * FILTER_MATRIX⁻¹
         //                    = displayRotationMatrix * eventTransform
         displayTransform = AffineMatrix.multiplyAll(displayRotationMatrix, eventTransform);
+        if (flexDisplay && displayTransform == null) {
+            // Force OpenGL rendering to avoid glitches on resize
+            displayTransform = AffineMatrix.IDENTITY;
+        }
     }
 
     public void startNew(Surface surface) {
@@ -282,6 +286,9 @@ public class NewDisplayCapture extends SurfaceCapture {
             glRunner.stopAndRelease();
             glRunner = null;
         }
+        if (virtualDisplay != null) {
+            virtualDisplay.setSurface(null);
+        }
     }
 
     @Override

@yume-chan

For physical displays, instead of changing physical display resolution, maybe it can change capture resolution instead. Would be useful when streaming over Internet, or reduce CPU/RAM usages when multiple instances are open (tiled, each one is small)

That's an idea for a separate feature (although the quality is lower if you scale the capture because there is no mipmapping), but it won't allow to change the aspect ratio anyway. Not sure this needs to be dynamic (I don't know)?

when the virtual display is resized, there is a fade-in/fade-out animation between the previous content (old size) and the new content (new size)

Apparently you'd need to turn off window animation, either in Developer options or programmatically:

• https://stackoverflow.com/q/17166647
• https://stackoverflow.com/q/10936946

But I couldn't find a way to get the window shown on the virtual display (is there any?) and change its layout params.

Apparently you'd need to turn off window animation, either in Developer options or programmatically

It can be turned off globally (and it improves the visual result):

diff --git a/server/src/main/java/com/genymobile/scrcpy/wrappers/WindowManager.java b/server/src/main/java/com/genymobile/scrcpy/wrappers/WindowManager.java
index 7ba5cc06d..8d2c1de4c 100644
--- a/server/src/main/java/com/genymobile/scrcpy/wrappers/WindowManager.java
+++ b/server/src/main/java/com/genymobile/scrcpy/wrappers/WindowManager.java
@@ -6,6 +6,7 @@ import com.genymobile.scrcpy.util.Ln;
 import android.annotation.TargetApi;
 import android.os.Build;
 import android.os.IInterface;
+import android.provider.Settings;
 import android.view.IDisplayWindowListener;
 
 import java.lang.reflect.Method;
@@ -40,6 +41,7 @@ public final class WindowManager {
 
     private WindowManager(IInterface manager) {
         this.manager = manager;
+        disableAnimations();
     }
 
     private Method getGetRotationMethod() throws NoSuchMethodException {
@@ -264,4 +266,22 @@ public final class WindowManager {
             Ln.e("Could not invoke method", e);
         }
     }
+
+    public void disableAnimations() {
+        try {
+            Method getMethod = manager.getClass().getMethod("getAnimationScale", int.class);
+            // Settings.Global.WINDOW_ANIMATION_SCALE
+            Ln.i("  [0] = " + getMethod.invoke(manager, 0));
+            // Settings.Global.TRANSITION_ANIMATION_SCALE
+            Ln.i("  [1] = " + getMethod.invoke(manager, 1));
+            // Settings.Global.ANIMATOR_DURATION_SCALE
+            Ln.i("  [2] = " + getMethod.invoke(manager, 2));
+            Method method = manager.getClass().getMethod("setAnimationScale", int.class, float.class);
+            method.invoke(manager, 0, 0);
+            method.invoke(manager, 1, 0);
+            method.invoke(manager, 2, 0);
+        } catch (ReflectiveOperationException e) {
+            Ln.e("Could not invoke method", e);
+        }
+    }
 }

(the relevant value is at index 1, the "transition animation scale")

They can also be set via adb:

# get values
adb shell settings get global window_animation_scale
adb shell settings get global transition_animation_scale
adb shell settings get global animator_duration_scale

# disable
adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0

# restore
adb shell settings put global window_animation_scale 1
adb shell settings put global transition_animation_scale 1
adb shell settings put global animator_duration_scale 1

That does not resolve all glitches though.

That does not resolve all glitches though.

This other thing I saw was ROTATION_ANIMATION_JUMPCUT that can be set to WindowManager.LayoutParams.rotationAnimation but I don't know on which LayoutParams instance to do it and whether it's relevant for resizing flex VDs (as it seems to be specifically about rotation).

• virtualDisplay.resize() is called from the same thread as the encoding process (otherwise Android would internally accumulate resize calls)

I finally abandoned this approach: virtualDisplay.resize() itself is an asynchronous request, and multiple requests can accumulate faster than the display can actually update, so it did not make sense.

In the end, I implemented an event debouncer (based on timing, something I initially wanted to avoid) that applies a resize (by calling virtualDisplay.resize() only after a fixed delay (hardcoded to 300 ms), collapsing all resize events within that window into a single update.

In case you're wondering, the debouncer is implemented on the server side (not on the client side) for several reasons:

• if the display size/rotation changes independently, any the pending client resizes must be canceled (this is not possible if the client resize events are sent after a delay)
• what matters is the interval between actual resize() calls on the server side (if the client sends several resizes over wifi and the network lags, those requests could otherwise be applied in rapid succession once they arrive) (this is a minor concern in practice)
• in the future, we may be able to avoid a time-based debouncer and instead trigger resizes precisely when the display is "ready" (whatever that means), which would inherently be a server-side concern

I installed a Genymotion emulator and built this PR. I can resize the window a couple times but it always crashes with a buffer dequeue exception (expand to see the logs). This only happens with --flex-display.

$ ./run x -Vverbose --no-audio --flex-display --new-display=570x1230/220
scrcpy 3.3.4 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO:     --> (tcpip)  127.0.0.1:6555                  device  Phone
DEBUG: Device serial: 127.0.0.1:6555
DEBUG: Using SCRCPY_SERVER_PATH: x/server/scrcpy-server
x/server/scrcpy-server: 1 file pushed, 0 skipped. 337.3 MB/s (730578 bytes in 0.002s)
[server] INFO: Device: [Genymobile] Genymotion Phone (Android 15)
DEBUG: Server connected
DEBUG: Starting controller thread
DEBUG: Starting receiver thread
[server] DEBUG: Using video encoder: 'OMX.google.h264.encoder'
[server] DEBUG: Video codec size alignment requirement: 1px
[server] INFO: New display: 570x1230/220 (id=2)
INFO: Renderer: opengl
INFO: OpenGL version: 4.6 (Compatibility Profile) Mesa 25.2.8-0ubuntu0.24.04.1
INFO: Trilinear filtering enabled
DEBUG: Using icon from SCRCPY_ICON_DIR: app/data/scrcpy.png
DEBUG: Demuxer 'video': starting thread
VERBOSE: resize_display(512, 1104)
VERBOSE: input: resize display 512x1104
[server] VERBOSE: NewDisplayCapture: requestResize(512, 1104)
[server] VERBOSE: NewDisplayCapture: constrained size = 512x1104
INFO: Texture: 512x1104
[server] VERBOSE: DisplayMonitor: onDisplayConfigurationChanged(2)
[server] VERBOSE: DisplayMonitor: 570x1230 [rotation=0] -> 512x1104 [rotation=0]
VERBOSE: resize_display(513, 1104)
VERBOSE: input: resize display 513x1104
[server] VERBOSE: NewDisplayCapture: requestResize(513, 1104)
[server] VERBOSE: NewDisplayCapture: constrained size = 512x1101
VERBOSE: resize_display(506, 1094)
VERBOSE: input: resize display 506x1094
[server] VERBOSE: NewDisplayCapture: requestResize(506, 1094)
[server] VERBOSE: NewDisplayCapture: constrained size = 506x1094
VERBOSE: resize_display(498, 1081)
VERBOSE: input: resize display 498x1081
[server] VERBOSE: NewDisplayCapture: requestResize(498, 1081)
[server] VERBOSE: NewDisplayCapture: constrained size = 498x1081
VERBOSE: resize_display(496, 1077)
VERBOSE: input: resize display 496x1077
[server] VERBOSE: NewDisplayCapture: requestResize(496, 1077)
[server] VERBOSE: NewDisplayCapture: constrained size = 496x1077
[server] VERBOSE: DisplayMonitor: onDisplayConfigurationChanged(2)
[server] VERBOSE: DisplayMonitor: 512x1104 [rotation=0] -> 496x1077 [rotation=0]
[server] DEBUG: Screen streaming stopped
[server] DEBUG: Device message sender stopped
DEBUG: Demuxer 'video': end of frames
DEBUG: Receiver stopped
WARN: Device disconnected
[server] DEBUG: Controller stopped
[server] ERROR: Exception on thread Thread[video,5,main]
java.lang.IllegalStateException: Pending dequeue output buffer request cancelled

	at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
	at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:4057)
	at com.genymobile.scrcpy.video.SurfaceEncoder.encode(SurfaceEncoder.java:171)
	at com.genymobile.scrcpy.video.SurfaceEncoder.streamCapture(SurfaceEncoder.java:136)
	at com.genymobile.scrcpy.video.SurfaceEncoder.lambda$start$0$com-genymobile-scrcpy-video-SurfaceEncoder(SurfaceEncoder.java:265)
	at com.genymobile.scrcpy.video.SurfaceEncoder$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
	at java.lang.Thread.run(Thread.java:1012)
DEBUG: Using icon from SCRCPY_ICON_DIR: app/data/disconnected.png
DEBUG: Controller stopped
Killed 
DEBUG: Server disconnected
DEBUG: Server terminated
DEBUG: Closing after device disconnection
DEBUG: Quit...

Side remark: Maybe the "disconnected" icon should be positioned at the center of the window regardless of the render-fit mode?

I can resize the window a couple times but it always crashes with a buffer dequeue exception

Is it specific to the new version of this PR (flex-display.4), or did it also happen with flex-display.1?

Is it specific to the new version of this PR (flex-display.4), or did it also happen with flex-display.1?

It fails the same way with all versions, including flex-display.1. I just updated #6772 (comment) with verbose logs. Let me know if I can do something more to help.

See SurfaceEncoder from #6350, that seems to be related. I can test the diff later if you want.

[EDIT] The try-catch avoids the crash (the exception message is "Pending dequeue output buffer request cancelled"), but then the issue is that the virtual display is not resized. It gets resized only when there are no such exceptions.

I can resize the window a couple times but it always crashes with a buffer dequeue exception

I don't know if it's an emulator issue, or if it can happen on a real device. If it cannot happen on real devices, it's not a big deal I think.

You edited #6772 (comment) to include verbose logs, but the exception is not exactly the same as the initial one. Initially it included this error message:

java.lang.IllegalStateException: Pending dequeue output buffer request cancelled

Btw, can you reproduce with --min-size-alignment=16?

Time to merge 🚀

Many thanks again to @anotheruserofgithub for the deep-dive review and thorough testing, which helped catch important issues. 👍

Great work! Thanks. :)

Two things left to do now that this is merged:

• Update links in doc/develop.md (changes in protocol implementation) as discussed in Update protocol documentation #6776 (comment).
• Correct the link to virtual-display.md here when merging with master, due to commit bf8bff4.

@anotheruserofgithub

Update links in doc/develop.md

Done in d678988.

Correct the link to virtual-display.md

Done in the merge commit from master into dev: d678988.

this is crazy good for app development