1752351 - canvas.captureStream: A TimerDriver that hasn't requested a frame to be captured can make throttled capturing stall

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Andreas Pehrson [:pehrsons]]

STR: Join a whereby.com room and enable a video effect in a tab that you put in the background. Join as a remote peer and observe the video from the background tab. It will appear throttled although they evade the throttling and continue drawing to the canvas.

This is because we exit prior to marking the context CLEAN for frame capture when the context is DIRTY. Thus the next draw operation does not turn the context DIRTY and thus does not trigger the watcher of the FrameCaptureState Watchable.

A TimerDriver can happen to trigger this if the FrameCaptureStateChange handler runs at a shorter interval (because of draws at the shorter interval) than the framerate given to the TimerDriver, thus the TimerDriver won't have requested another frame yet.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Do not rely on a timer for canvas.captureStream's TimerDriver. r?jib!

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Use markers instead of labels for debugging canvas.captureStream with the profiler, and add more. r?jib!

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Fix frame capture stalling with a TimerDriver that is slower than the drawing code. r?jib!

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1344524

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Improve the throttled capture mochitest. r?jib!

Turned out the refresh driver was not at all throttled in the test. rAF was
throttled but these are not always in sync.

This patch changes the approach to causing throttling by being closer to a real
scenario -- put the tab in the background by opening another tab in the
foreground.

This also meant that the method by which we were measuring the captured frame
rate became moot (videoElement.mozPaintedFrames), as the frames are not
composited in the background, and the compositor is what reports painted frames.

This patch changes the frame counting approach to a local peer connection and
the outbound-rtp stats' framesEncoded member.

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Test throttled frame capture with a slower TimerDriver. r?jib!

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1752351 - Fix an assertion failure with throttled capture. r?jib!

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1752351 - Skip test_capture_throttled.html on android which doesn't appear to reliably throttle rAF in background, at least not from window.open() of a new tab. r?mjf

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/34237ea4f82f
Improve the throttled capture mochitest. r=jib
https://hg.mozilla.org/integration/autoland/rev/861e5267cafc
Test throttled frame capture with a slower TimerDriver. r=jib
https://hg.mozilla.org/integration/autoland/rev/801268f30e94
Do not rely on a timer for canvas.captureStream's TimerDriver. r=jib
https://hg.mozilla.org/integration/autoland/rev/1694bfb744ee
Use markers instead of labels for debugging canvas.captureStream with the profiler, and add more. r=jib
https://hg.mozilla.org/integration/autoland/rev/0c171c3b5e27
Fix frame capture stalling with a TimerDriver that is slower than the drawing code. r=jib
https://hg.mozilla.org/integration/autoland/rev/8f75bab2ab54
Fix an assertion failure with throttled capture. r=jib
https://hg.mozilla.org/integration/autoland/rev/7d6d47f9429b
Skip test_capture_throttled.html on android which doesn't appear to reliably throttle rAF in background, at least not from window.open() of a new tab. r=mjf

Comment 10

[Natalia Csoregi [:nataliaCs]]

Backed out for failures on test_capture_throttled.html

• backout: https://hg.mozilla.org/integration/autoland/rev/1c4ad75da1c8aa4b4ef7da2d1db869aac0b03ee5
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=7d6d47f9429b2a31a257c6ab9f0c8e0f6e82fdcf&selectedTaskRun=cWpe5TKfQOaWKjn5hMZMbg.0
• failure log: https://treeherder.mozilla.org/logviewer?job_id=366496874&repo=autoland&lineNumber=3187

[task 2022-02-02T20:04:56.898Z] 20:04:56     INFO - TEST-PASS | dom/canvas/test/test_capture_throttled.html | pc2 connection established (connected) 
[task 2022-02-02T20:04:56.899Z] 20:04:56     INFO - Buffered messages finished
[task 2022-02-02T20:04:56.899Z] 20:04:56     INFO - TEST-UNEXPECTED-FAIL | dom/canvas/test/test_capture_throttled.html | Test timed out. - 
[task 2022-02-02T20:04:57.626Z] 20:04:57     INFO - GECKO(6452) | (stun/INFO) STUN-CLIENT(consent): Received response; processing
[task 2022-02-02T20:04:57.629Z] 20:04:57     INFO - GECKO(6452) | (ice/INFO) ICE(PC:{94a75ab8-df3c-4cb1-b5b4-d000adc9a226} 1643831987443000 (id=4294967474 url=http://mochi.test:8888/tests/dom/canvas/test/test_ca)/STREAM(PC:{94a75ab8-df3c-4cb1-b5b4-d000adc9a226} 1643831987443000 (id=4294967474 url=http://mochi.test:8888/tests/dom/canvas/test/test_ca transport-id=transport_0 - ffb200c0:af6d9b650b918c6ced457187c6e645f7)/COMP(1): Consent refreshed
[task 2022-02-02T20:04:57.779Z] 20:04:57     INFO - GECKO(6452) | MEMORY STAT | vsize 561MB | vsizeMaxContiguous 1293MB | residentFast 68MB | heapAllocated 6MB
[task 2022-02-02T20:04:57.798Z] 20:04:57     INFO - TEST-OK | dom/canvas/test/test_capture_throttled.html | took 312741ms

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1752351 - Fix test to handle intermittent lack of outbound-rtp stats. r?ng

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/38c5e475306e
Improve the throttled capture mochitest. r=jib
https://hg.mozilla.org/integration/autoland/rev/3d4eefe8530c
Test throttled frame capture with a slower TimerDriver. r=jib
https://hg.mozilla.org/integration/autoland/rev/5769242de6e1
Do not rely on a timer for canvas.captureStream's TimerDriver. r=jib
https://hg.mozilla.org/integration/autoland/rev/556bc8547846
Use markers instead of labels for debugging canvas.captureStream with the profiler, and add more. r=jib
https://hg.mozilla.org/integration/autoland/rev/09885bfc61fb
Fix frame capture stalling with a TimerDriver that is slower than the drawing code. r=jib
https://hg.mozilla.org/integration/autoland/rev/4c7d71bca5fc
Fix an assertion failure with throttled capture. r=jib
https://hg.mozilla.org/integration/autoland/rev/199ea766c664
Skip test_capture_throttled.html on android which doesn't appear to reliably throttle rAF in background, at least not from window.open() of a new tab. r=mjf
https://hg.mozilla.org/integration/autoland/rev/571541f6bff0
Fix test to handle intermittent lack of outbound-rtp stats. r=ng

Comment 13

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/38c5e475306e
https://hg.mozilla.org/mozilla-central/rev/3d4eefe8530c
https://hg.mozilla.org/mozilla-central/rev/5769242de6e1
https://hg.mozilla.org/mozilla-central/rev/556bc8547846
https://hg.mozilla.org/mozilla-central/rev/09885bfc61fb
https://hg.mozilla.org/mozilla-central/rev/4c7d71bca5fc
https://hg.mozilla.org/mozilla-central/rev/199ea766c664
https://hg.mozilla.org/mozilla-central/rev/571541f6bff0