Closed Bug 1826526 Opened 3 years ago Closed 2 years ago

Get test_getUserMedia_basicScreenshare.html enabled on Mac

Categories

(Core :: WebRTC: Audio/Video, task, P2)

Product:
Core
Component:
WebRTC: Audio/Video
Platform:
Unspecified
macOS
Type:
task

Tracking

()

Status:
RESOLVED FIXED
Milestone:
114 Branch
Tracking Flags:
Tracking Status
firefox114 --- fixed

People

(Reporter: pehrsons, Assigned: pehrsons)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(4 files)

Bug 1826526 - In test_gUM_basicScreenshare.html draw white instead of grey. r?webrtc-reviewers
48 bytes, text/x-phabricator-request
Details | Review
Bug 1826526 - In test_gUM_basicScreenshare.html adjust thresholds and re-enable on Mac. r?webrtc-reviewers
48 bytes, text/x-phabricator-request
Details | Review
failing test with log for screenshot, etc.
10.26 KB, text/plain
Details
failing test log
10.26 KB, text/plain
Details

test_getUserMedia_basicScreenshare.html is disabled on both apple silicon (some local machines, like mine) and catalina (used in CI) which means we have close to zero coverage of screen capture on mac, and definitely zero coverage of screen capture on mac in CI.

We need to get it enabled again.

As pointed out in bug 1707742 this is failing due to the color profile used on the test machines in CI. On my M2 Macbook Pro I have the preset profile "Apple XDR Dispaly (P3-1600 nits)" and it fails for me too.

The ARGB capture of the rendered canvas that the test drawn red to has these components with the aformentioned profile: A=255, R=234, G=51, B=36.
With the PAL/SECAM preset I get A=255, R=250, G=0, B=0 and with NTSC A=255, R=255, G=0, B=5. But I won't be running with either of those presets locally just so this test can pass, therefore I disagree with bug 1707742 comment 24.

We will have to adjust the thresholds in the test. Which FWIW I think is completely fine, given that we only draw "pure" components of red, green, blue, and sure, also grey. To get a false positive on, say, blue when checking for red the thresholds would have to be close to 100% (in the example given above where B=36 the threshold would have to be 255-36=219 and we won't go anywhere near those numbers.

Status: NEW → ASSIGNED

Even more interesting than red is green which on my machine is captured as A=255, R=117, G=251, B=76.

White is less susceptible to false positives than grey with a high threshold,
given that all its components are at an extreme (255 vs 128).

A catalina based mac mini used on CI fails the test for green with R,G,B,A=135,249,39,255 (we drew 0,255,0,255). This is excruciatingly large, and we should definitely do something about the color profile too.

Even more interesting is this debug job which failed checking for red, got white, and the screenshot has no firefox in it.

Joel, would you know who can fix the screenshot so we can know what is getting captured here?

Flags: needinfo?(jmaher)
fixing the screenshots to capture the browser is difficult. here is what I see in the log for the above failing test and screenshot: ```
Attached file failing test log —
fixing the screenshots to capture the browser is difficult. here is what I see in the log for the above failing test and screenshot: ```

fixing the screenshots to capture the browser is difficult.

here is what I see in the log for the above failing test and screenshot:

[task 2023-04-05T14:40:17.864Z] 14:40:17     INFO - TEST-START | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html
[task 2023-04-05T14:40:17.941Z] 14:40:17     INFO - GECKO(2280) | TEST DEVICES: No test device found in media.audio_loopback_dev, using fake audio streams.
[task 2023-04-05T14:40:17.942Z] 14:40:17     INFO - GECKO(2280) | TEST DEVICES: No test device found in media.video_loopback_dev, using fake video streams.
[task 2023-04-05T14:40:18.201Z] 14:40:18     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: Can't add a range if the end is older that the start.: file /builds/worker/checkouts/gecko/dom/html/TimeRanges.cpp:72
[task 2023-04-05T14:40:18.206Z] 14:40:18     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: Can't add a range if the end is older that the start.: file /builds/worker/checkouts/gecko/dom/html/TimeRanges.cpp:72
[task 2023-04-05T14:40:18.210Z] 14:40:18     INFO - GECKO(2280) | TEST DEVICES: No test device found in media.audio_loopback_dev, using fake audio streams.
[task 2023-04-05T14:40:18.211Z] 14:40:18     INFO - GECKO(2280) | TEST DEVICES: No test device found in media.video_loopback_dev, using fake video streams.
[task 2023-04-05T14:40:18.216Z] 14:40:18     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: Can't add a range if the end is older that the start.: file /builds/worker/checkouts/gecko/dom/html/TimeRanges.cpp:72
[task 2023-04-05T14:40:18.221Z] 14:40:18     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: Can't add a range if the end is older that the start.: file /builds/worker/checkouts/gecko/dom/html/TimeRanges.cpp:72
[task 2023-04-05T14:40:18.418Z] 14:40:18     INFO - GECKO(2280) | [Parent 2280, Main Thread] WARNING: NS_ENSURE_TRUE(root) failed: file /builds/worker/checkouts/gecko/layout/base/nsDocumentViewer.cpp:2638
[task 2023-04-05T14:40:18.468Z] 14:40:18     INFO - GECKO(2280) | [WARN  webrender::device::gl] Missing optimized shader source for gpu_cache_update
[task 2023-04-05T14:40:18.507Z] 14:40:18     INFO - GECKO(2280) | [WARN  webrender::device::gl] Missing optimized shader source for gpu_cache_update
[task 2023-04-05T14:40:18.543Z] 14:40:18     INFO - GECKO(2280) | [Parent 2280, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:371
[task 2023-04-05T14:40:18.549Z] 14:40:18     INFO - GECKO(2280) | [WARN  webrender::device::gl] Cropping texture upload Box2D((0, 0), (0, 1)) to None
[task 2023-04-05T14:40:18.549Z] 14:40:18     INFO - GECKO(2280) | [WARN  webrender::device::gl] Cropping texture upload Box2D((0, 0), (0, 1)) to None
[task 2023-04-05T14:40:18.550Z] 14:40:18     INFO - GECKO(2280) | [WARN  webrender::device::gl] Cropping texture upload Box2D((0, 0), (0, 1)) to None
[task 2023-04-05T14:40:27.119Z] 14:40:27     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: FuncScope not on stack!: file /builds/worker/checkouts/gecko/dom/canvas/WebGLContext.cpp:1774
[task 2023-04-05T14:40:27.119Z] 14:40:27     INFO - GECKO(2280) | [Child 2288, Main Thread] WARNING: FuncScope not on stack!: file /builds/worker/checkouts/gecko/dom/canvas/WebGLContext.cpp:1774
[task 2023-04-05T14:40:34.876Z] 14:40:34     INFO - GECKO(2280) | 2023-04-05 14:40:34.874 firefox[2280:44606] Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightlydebug.savedState/window_1.data: No such file or directory (2)
[task 2023-04-05T14:40:48.773Z] 14:40:48     INFO - GECKO(2280) | [Parent 2280, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:371
[task 2023-04-05T14:40:49.051Z] 14:40:49     INFO - TEST-INFO | started process screencapture
[task 2023-04-05T14:40:49.164Z] 14:40:49     INFO - TEST-INFO | screencapture: exit 0
[task 2023-04-05T14:40:49.164Z] 14:40:49     INFO - Buffered messages logged at 14:40:17
[task 2023-04-05T14:40:49.165Z] 14:40:49     INFO - TEST-PASS | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html | A valid string reason is expected 
[task 2023-04-05T14:40:49.165Z] 14:40:49     INFO - TEST-PASS | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html | Reason cannot be empty 
[task 2023-04-05T14:40:49.166Z] 14:40:49     INFO - Buffered messages logged at 14:40:18

so the test is running at the time 14:40:17, but it is not grabbing a browser in the foreground.

I see on other macosx1015 failures we can see the browser (mochitest 2 failure):
https://firefoxci.taskcluster-artifacts.net/PNUMpzMWT_eyMCe6apGelA/0/public/test_info/mozilla-test-fail-screenshot_0624da9j.png

also a mochitest-media test failure shows the browser:
https://firefoxci.taskcluster-artifacts.net/Q95uIRgyTR6gdOjtpjUZpg/0/public/test_info/mozilla-test-fail-screenshot_q2bo_jts.png

is it possible the test itself is doing something with the browser (minimizing?) or moving it to another display?

Flags: needinfo?(jmaher)

Thanks for checking. I saw after repeating this test failure a number of times it would usually show the browser. Not sure what happened in this case.

This test is entering and leaving fullscreen, but that should be it. For window capture we would focus the window we are capturing, but this is screen capture so we do nothing.

Attachment #9327115 - Attachment description: Bug 1826526 - In test_gUM_basicScreenshare.html increase the threshold and re-enable on Mac. r?webrtc-reviewers → Bug 1826526 - In test_gUM_basicScreenshare.html adjust thresholds and re-enable on Mac. r?webrtc-reviewers
Depends on: 1827606
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/2b8a934ef44c In test_gUM_basicScreenshare.html draw white instead of grey. r=webrtc-reviewers,ng https://hg.mozilla.org/integration/autoland/rev/a6ab77040b7c In test_gUM_basicScreenshare.html adjust thresholds and re-enable on Mac. r=webrtc-reviewers,ng

Backed out for causing failures on test_getUserMedia_basicScreenshare.html

[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - TEST-FAIL | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html | The author of the test has indicated that flaky timeouts are expected.  Reason: WebRTC inherently depends on timeouts 
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - Buffered messages finished
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - TEST-UNEXPECTED-FAIL | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html | Error executing test: Error: Checking 690,270 against red timed out. Got [255,255,255,255]. Threshold 120.
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - verifyAround/<.cancel<@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:77:13
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - promise callback*verifyAround@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:76:31
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - verifyScreenshare@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:87:11
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - doVerify@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:145:15
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - async*@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:166:11
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - async*runTest/<@https://example.com/tests/dom/media/webrtc/tests/mochitests/mediaStreamPlayback.js:194:11
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - runTestWhenReady@https://example.com/tests/dom/media/webrtc/tests/mochitests/head.js:502:11
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - async*runTest@https://example.com/tests/dom/media/webrtc/tests/mochitests/mediaStreamPlayback.js:193:9
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - async*@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:103:10
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO -  
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - SimpleTest.ok@https://example.com/tests/SimpleTest/SimpleTest.js:424:16
[task 2023-04-21T10:43:31.740Z] 10:43:31     INFO - runTestWhenReady@https://example.com/tests/dom/media/webrtc/tests/mochitests/head.js:504:7
[task 2023-04-21T10:43:31.741Z] 10:43:31     INFO - async*runTest@https://example.com/tests/dom/media/webrtc/tests/mochitests/mediaStreamPlayback.js:193:9
[task 2023-04-21T10:43:31.741Z] 10:43:31     INFO - async*@https://example.com/tests/dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html:103:10
[task 2023-04-21T10:43:31.741Z] 10:43:31     INFO - GECKO(6341) | MEMORY STAT | vsize 400983MB | residentFast 130MB | heapAllocated 14MB
[task 2023-04-21T10:43:31.741Z] 10:43:31     INFO - TEST-OK | dom/media/webrtc/tests/mochitests/test_getUserMedia_basicScreenshare.html | took 30753ms
Flags: needinfo?(apehrson)

Seems to be permafailing on MacOS 11 jobs. These run on apple silicon which is perhaps why they behave differently from the MacOS 10.15 jobs. It does work on apple silicon in general though, I have an M2 locally.

The screenshots all show an empty desktop for these failures, not sure if that is a clue.

An odd thing is that it is reporting white when checking for red, while the desktop background is sort of green. If there's more than one display perhaps it went into fullscreen on the display in the screenshot (but before the screenshot was taken), while we were capturing the other display, where Firefox is. The background of the test page would be white.

Flags: needinfo?(apehrson)

I'll disable on apple silicon for now due to the issue outlined in comment 12. I'll file a followup to get it enabled there too.

Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/2d75263cfb92 In test_gUM_basicScreenshare.html draw white instead of grey. r=webrtc-reviewers,ng https://hg.mozilla.org/integration/autoland/rev/e4c1223b324c In test_gUM_basicScreenshare.html adjust thresholds and re-enable on Mac. r=webrtc-reviewers,ng

https://hg.mozilla.org/mozilla-central/rev/2d75263cfb92
https://hg.mozilla.org/mozilla-central/rev/e4c1223b324c

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox114: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 114 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: