Should we disable commsOverSubshells because it can corrupt plots? #19719

Description

Comm messages are sent to a kernel subshell by default (commsOverSubshells is perCommTarget), so the kernel runs comm handlers on the subshell thread. An ipywidgets callback therefore runs at the same time as the main shell is executing cells. matplotlib.pyplot keeps global state and is not safe under that: one thread renders a figure the other is still drawing. Figures come out with missing axes and only part of the data, and outputs are sometimes lost entirely.

This needs a kernel with subshell support, so ipykernel 7 and later. Setting commsOverSubshells to disabled avoids it.

Seen in our docs test runs in #19716. general.test.ts runs the Lorenz notebook, which wraps a matplotlib figure in interactive. Since switching to ipykernel v7 among twelve re-runs we get five different screenshots.

Reproduce

1. Install ipykernel 7.3.0, ipywidgets 8.1.5 and matplotlib 3.10.0.
2. Create a notebook with these four cells:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interactive

def plot(seed=0):
    fig = plt.figure(figsize=(5, 5))
    ax = fig.add_subplot()
    rng = np.random.RandomState(seed)
    for i in range(60):
        ax.plot(rng.randn(400).cumsum(), lw=1)
    plt.show()

w = interactive(plot, seed=(0, 10))
w

x = np.random.RandomState(0).randn(500)

plt.hist(x)
plt.title('a');

plt.hist(x * 2)
plt.title('b');

3. Run all cells.
4. Restart the kernel and run all cells again, a few times.
5. Observe that the histogram in cell 3 is often drawn without axes and with most bars missing, and that cell 4 sometimes shows no output.

6. Set commsOverSubshells to disabled under Settings, Kernels, and repeat. The histograms are now the same on every run.

Expected behavior

Every input above is seeded, so both histograms should be identical on every run.

Context

• Operating System and version: Linux
• JupyterLab version: 4.7.0a1