Closed Bug 1785390 Opened 3 years ago Closed 6 months ago

Intermittent /webrtc/protocol/rtp-demuxing.html | single tracking bug

Categories

(Core :: WebRTC, defect, P3)

Product:
Core
Component:
WebRTC
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: jmaher, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: intermittent-failure, intermittent-testcase)

No description provided.

Additional information about this bug failures and frequency patterns can be found by running: ./mach test-info failure-report --bug 1785390

Depends on: 1818283

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=433088810&repo=autoland

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
Resolution: --- → INCOMPLETE

This test is permanently timing out for me locally on MacOS because the metadataToBeLoaded promise never resolves. This is because only a single loadedmetadata event is recorded so that resolve() never gets called.

I was not able to find any other open bug and the other two were duped to that one. So I'll reopen for further investigation.

Michael, do you have an idea why this is happening?

Flags: needinfo?(mfroman)
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Also the test has a manifest which includes:

  [Can demux two video tracks with different payload types on a bundled connection]
    expected:
      if (os == "mac") and not debug: [PASS, FAIL]
      if os == "android": FAIL

So it looks like that at least in CI the test is not timing out but it can fail - as such we do not get any further failures reported.``

Byron, can you run this test locally on your macOS machine?
./mach wpt testing/web-platform/tests/webrtc/protocol/rtp-demuxing.html

I'm having python versioning issues locally on my new laptop which installed python 3.13.

Flags: needinfo?(mfroman) → needinfo?(docfaraday)

Ok - after dismantling a bit of homebrew installed stuff that resulted in the python 3.13 installation, I can run this test locally.

I've run it both in debug and opt builds and have not seen it fail over 30 repeated runs using ./mach wpt --repeat 30 testing/web-platform/tests/webrtc/protocol/rtp-demuxing.html. For reference, I was running on mozilla-central ffddf6c98dc7.

Note that I'm on MacOS 15.2 and can reproduce this failure all the time which means that I don't even have to use the --repeat flag.

Since this seems to be a very isolated issue, I'm wondering if you'd mind trying it under a different user acct on your machine? Perhaps there are OS permissions that are blocking or some other environment issue.

Flags: needinfo?(docfaraday)

Which test is failing? If the first, which codec are you not getting video for? I'd guess H264 since that depends on GMP and the fakeopenh264 plugin. Could you just s/H264/VP9/g to try and verify? The test is still passing for me with that change.

Note the fakeopenh264 plugin was plugged in for WPT quite recently in bug 1534688.

Flags: needinfo?(hskupin)

Hi Andreas. Yes, it's the first test that is not completing and timing out. And applying your suggestion actually makes it working! So when running the test it's not awaited until the GMP plugin is downloaded. I can see it as well in the addon manager that its download will happen shortly.

Flags: needinfo?(hskupin)

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #15)

Hi Andreas. Yes, it's the first test that is not completing and timing out. And applying your suggestion actually makes it working! So when running the test it's not awaited until the GMP plugin is downloaded. I can see it as well in the addon manager that its download will happen shortly.

How do you run the test? ./mach wpt?

I added plumbing for using the fakeopenh264 plugin for WPT through mach and in our CI. Could you check that your <objdir>/dist/bin has a gmp-fakeopenh264/1.0/libfakeopenh264* shared library?

Also, does ./mach mochitest dom/media/webrtc/tests/mochitests/test_peerConnection_videoCodecs.html work?

Flags: needinfo?(hskupin)

(In reply to Andreas Pehrson [:pehrsons] from comment #16)

How do you run the test? ./mach wpt?

Yes, that is the command that I'm using.

I added plumbing for using the fakeopenh264 plugin for WPT through mach and in our CI. Could you check that your <objdir>/dist/bin has a gmp-fakeopenh264/1.0/libfakeopenh264* shared library?

Yes, it's there:

✗ ls -la ../gecko-obj/opt/dist/bin/gmp-fakeopenh264/1.0/
total 304
drwxr-xr-x  4 henrik  staff     128 Feb 14 17:13 .
drwxr-xr-x  3 henrik  staff      96 Feb 14 17:12 ..
lrwxr-xr-x  1 henrik  staff      72 Feb 14 17:13 fakeopenh264.info -> /Users/henrik/code/gecko/dom/media/gmp-plugin-openh264/fakeopenh264.info
-rwxr-xr-x  1 henrik  staff  155072 Feb 24 12:10 libfakeopenh264.dylib

Please note that I'm using an artifact build and not a full build. Not sure if this might make a difference.

Also, does ./mach mochitest dom/media/webrtc/tests/mochitests/test_peerConnection_videoCodecs.html work?

It's failing as well with hundreds of timeouts reported in the results.

Maybe it would better to file a new bug about this particular issue? But not sure as of this point in which component it should live.

Flags: needinfo?(hskupin)

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #17)

(In reply to Andreas Pehrson [:pehrsons] from comment #16)

How do you run the test? ./mach wpt?

Yes, that is the command that I'm using.

I added plumbing for using the fakeopenh264 plugin for WPT through mach and in our CI. Could you check that your <objdir>/dist/bin has a gmp-fakeopenh264/1.0/libfakeopenh264* shared library?

Yes, it's there:

✗ ls -la ../gecko-obj/opt/dist/bin/gmp-fakeopenh264/1.0/
total 304
drwxr-xr-x  4 henrik  staff     128 Feb 14 17:13 .
drwxr-xr-x  3 henrik  staff      96 Feb 14 17:12 ..
lrwxr-xr-x  1 henrik  staff      72 Feb 14 17:13 fakeopenh264.info -> /Users/henrik/code/gecko/dom/media/gmp-plugin-openh264/fakeopenh264.info
-rwxr-xr-x  1 henrik  staff  155072 Feb 24 12:10 libfakeopenh264.dylib

Please note that I'm using an artifact build and not a full build. Not sure if this might make a difference.

Ah, perhaps that's it. I can give that a shot.

Also, does ./mach mochitest dom/media/webrtc/tests/mochitests/test_peerConnection_videoCodecs.html work?

It's failing as well with hundreds of timeouts reported in the results.

Maybe it would better to file a new bug about this particular issue? But not sure as of this point in which component it should live.

Probably same root cause though, the fakeopenh264 plugin.

I can reproduce with artifact builds on macOS. But the fakeopenh264 plugin is loaded fine. It's the GMP process that we fail to spawn:

 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPParent[137648100|childPid=0] LoadProcess: for /Users/apehrson/Dev/mozilla-central/obj-aarch64-apple-darwin24.3.0/dist/bin/gmp-fakeopenh264/1.0
 1:47.48 GECKO(1550) [Child 1558: GMPThread]: D/GMP GMP Encode: not initted yet
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::Launch() mLaunchArch: 32
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::FillMacSandboxInfo: plugin dir path: /Users/apehrson/Dev/mozilla-central/obj-aarch64-apple-darwin24.3.0/dist/bin/gmp-fakeopenh264/1.0
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::FillMacSandboxInfo: resolved plugin dir path: /Users/apehrson/Dev/mozilla-central/obj-aarch64-apple-darwin24.3.0/dist/bin/gmp-fakeopenh264/1.0
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::FillMacSandboxInfo: IsPackagedBuild()=false
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::FillMacSandboxInfo: repo dir path: /Users/apehrson/Dev/mozilla-central
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPProcessParent::FillMacSandboxInfo: object dir path: /Users/apehrson/Dev/mozilla-central/obj-aarch64-apple-darwin24.3.0
 1:47.48 GECKO(1550) [Parent 1550, IPC Launch #1] WARNING: posix_spawnp failed: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_mac.mm:154
 1:47.48 GECKO(1550) [Parent 1550, IPC I/O Parent] WARNING: Failed to launch gmplugin subprocess @posix_spawnp (Error:0): file /builds/worker/checkouts/gecko/ipc/glue/GeckoChildProcessHost.cpp:805
 1:47.48 GECKO(1550) [Parent 1550: GMPThread]: D/GMP GMPParent[137648100|childPid=0] LoadProcess: Failed to launch new child process
 1:47.48 GECKO(1550) [Child 1558, GMPThread] WARNING: 'NS_FAILED(aResult.result())', file /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:123
 1:47.48 GECKO(1550) [Child 1558: GMPThread]: D/GMP GMPServiceChild failed to launch GMP with error: Process has not loaded.
 1:47.48 GECKO(1550) [Child 1558: GMPThread]: D/GMP GMPServiceChild::RemoveShutdownBlockerIfNeeded mPendingGetContentParents=0 mServiceChild->HaveContentParents()=false mShuttingDownOnGMPThread=false
Depends on: 1950310

Henrik, do note as per my debugging in bug 1950310, ac_add_options --with-branding=browser/branding/nightly is the appropriate workaround until this is fixed.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 2 years ago → 1 year ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=524205074&repo=mozilla-central

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 1 year ago → 6 months ago
Resolution: --- → INCOMPLETE
Depends on: 2011806
No longer depends on: 2011806
You need to log in before you can comment on or make changes to this bug.