1858175 - Camera capture fails to start with new Ubuntu 22.04 Wayland pool

Status: NEW

(Firefox Build System :: Task Configuration, defect, P2)

Description

[Andrew Halberstadt [:ahal]]

I'm working on getting tests running against a new Ubuntu 22.04 test pool with Wayland enabled. Crashtests are hitting a very frequent assertion failure on shutdown (doesn't seem to be associated with any particular test):
https://treeherder.mozilla.org/logviewer?job_id=431465082&repo=try&lineNumber=225000

[task 2023-10-05T16:15:20.766Z] 16:15:20     INFO - Assertion failure: it != mIdMap.end(), at /builds/worker/checkouts/gecko/dom/media/systemservices/VideoEngine.cpp:231

(from this push)

As it isn't tied to a test, I won't be able to skip anything to work around this and it will block bug 1856358.

It's unclear to me whether the assertion is a result of Wayland or using Ubuntu 22.04 or something else.

Comment 1

[Jim Mathies [:jimm]]

Assertion failure: it != mIdMap.end(), at /builds/worker/checkouts/gecko/dom/media/systemservices/VideoEngine.cpp:231

• In webrtc camera capture code
• enumerated devices related
• shutdown crash??

Jib, any thoughts?

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

It's cleanup of active camera capture upon tear-down of the parent in stopCapture, where it's iterating active capture callbacks and closing them with the OS. Somehow the aCaptureId CaptureEntry to be cleaned up in the VideoEngine is already gone? I suspect Wayland, since it has its own way of handling cameras as I recall.

Running with MOZ_LOG=CamerasParent:5,VideoEngine:5 might reveal more information.

Andreas, wdyt?

Comment 3

[Andreas Pehrson [:pehrsons]]

This is a classic assertion that usually goes under the radar because our tests are good at cleaning up after themselves.
This means we failed to start a non-fake video capture source, because content then doesn't try to stop it -- whereas the parent keeps it around waiting for content to stop it. A tad weird, but harmless, and can catch issues in CI apparently.

Seeing that desktop capture crashtests are already disabled due to timeouts, this is probably for a camera. Wayland has no bearing on cameras, and pipewire is off by default. But if we tried starting a device we must have enumerated it.

So do I assume correctly that there is some video input present, and that the machine has v4l2? Does it have the same v4l2loopback setup as the regular linux boxes?

For now probably easiest to disable those crashtests that request non-fake video through getUserMedia:

• dom/media/tests/crashtests/802982.html
• dom/media/tests/crashtests/1429507_1.html
• dom/media/tests/crashtests/1429507_2.html

Comment 4

[Andreas Pehrson [:pehrsons]]

I'm moving this to match the component of bug 1856358, since the assertion isn't the root of the problem. We'll want camera capture working on these machines at some point (ideally through pipewire since we have no coverage there today).

Feel free to shuffle it around if that is wrong.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit BugBot documentation.

Comment 6

[Andrew Halberstadt [:ahal]]

Andrew, do you know the answers to the questions in comment 3?

It sounds like we can disable our way around this for now, but I wonder if lack of camera capture might be causing issues in other test suites as well.

Comment 7

[Andrew Erickson [:aerickson]]

(In reply to Andreas Pehrson [:pehrsons] from comment #3)

So do I assume correctly that there is some video input present, and that the machine has v4l2? Does it have the same v4l2loopback setup as the regular linux boxes?

These tests are being run in a Virtualbox VM and it seems v4l2loopback does work to some extent in Virtualbox (https://www.virtualbox.org/ticket/20176 via https://github.com/umlaeute/v4l2loopback/issues/399). There isn't a video capture device attached or configured in Virtualbox. I'm not sure what v4l setup the VM image has. It's nearly stock Ubuntu 22.04.

Virtualbox aside: We're waiting on a display driver fix from GCP/Canonical to enable running Wayland VMs (Canonical has authored the fix, waiting on GCP verification) and waiting on hardware capacity to run Wayland hardware in MDC1, Virtualbox is what we could get working in the interim.

What would be helpful for debugging further? I have an older snapshot of the VM I can give you a link to if you have an x86 host to run Virtualbox on (I can setup a newer snapshot if we think it's needed, but I don't think any major changes have happened to the base OS) or I can provide output (e.g. lsmod, modprobe v4l*, and v4l configs?).