Closed Bug 1651569 Opened 4 years ago Closed 4 years ago

Perma [Tier 2] /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | X SNR is not greater than or equal to 88.457. Got 83.7951698233912. - assert_true: expected true got false

Categories

(Core :: Web Audio, defect, P5)

Product:
Core
Component:
Web Audio
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
82 Branch
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox-esr78 --- unaffected
firefox78 --- unaffected
firefox79 --- unaffected
firefox80 --- wontfix
firefox81 --- wontfix
firefox82 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: karlt)

References

(Depends on 1 open bug, Regression)

Details

(Keywords: intermittent-failure, regression, Whiteboard: , [wptsync upstream][stockwell unknown])

Attachments

(1 file)

Bug 1651569 reduce required SNR to accept KissFFT ConvolverNode implementation r?padenot
47 bytes, text/x-phabricator-request
Details | Review

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=309072328&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MnNNUV-3SguHhg9K2Te00Q/runs/0/artifacts/public/logs/live_backing.log

[task 2020-07-09T00:46:17.918Z] 00:46:17     INFO - TEST-OK | /FileAPI/blob/Blob-text.any.worker.html | took 319ms
[task 2020-07-09T00:46:17.919Z] 00:46:17     INFO - TEST-START | /FileAPI/blob/Blob-text.any.html
[task 2020-07-09T00:46:17.920Z] 00:46:17     INFO - Closing window 62
[task 2020-07-09T00:46:18.139Z] 00:46:18     INFO - ........
[task 2020-07-09T00:46:18.139Z] 00:46:18     INFO - TEST-OK | /FileAPI/blob/Blob-text.any.html | took 219ms
[task 2020-07-09T00:46:18.139Z] 00:46:18     INFO - Restarting browser for new test group
[task 2020-07-09T00:46:18.147Z] 00:46:18     INFO - PID 19370 | 1594255578141	Marionette	INFO	Stopped listening on port 39481
[task 2020-07-09T00:46:18.678Z] 00:46:18     INFO - 
[task 2020-07-09T00:46:18.678Z] 00:46:18     INFO - TEST-PASS | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | # AUDIT TASK RUNNER STARTED. 
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - TEST-PASS | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | Executing "test" 
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - TEST-PASS | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | Audit report 
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - TEST-PASS | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | > [test] Test convolver with real-time context 
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - TEST-UNEXPECTED-FAIL | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | X SNR is not greater than or equal to 88.457. Got 83.7951698233912. - assert_true: expected true got false
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - _logFailed/<@http://web-platform.test:8000/webaudio/resources/audit.js:52:18
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1977:25
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - test@http://web-platform.test:8000/resources/testharness.js:535:30
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - _logFailed@http://web-platform.test:8000/webaudio/resources/audit.js:51:9
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - _finalize@http://web-platform.test:8000/webaudio/resources/audit.js:217:19
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - _assert@http://web-platform.test:8000/webaudio/resources/audit.js:230:12
[task 2020-07-09T00:46:18.679Z] 00:46:18     INFO - beGreaterThanOrEqualTo@http://web-platform.test:8000/webaudio/resources/audit.js:579:19
[task 2020-07-09T00:46:18.680Z] 00:46:18     INFO - spn.onaudioprocess@http://web-platform.test:8000/webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html:103:26
[task 2020-07-09T00:46:18.680Z] 00:46:18     INFO - 
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - TEST-UNEXPECTED-FAIL | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | < [test] 1 out of 1 assertions were failed. - assert_true: expected true got false
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - _logFailed/<@http://web-platform.test:8000/webaudio/resources/audit.js:52:18
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1977:25
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - test@http://web-platform.test:8000/resources/testharness.js:535:30
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - _logFailed@http://web-platform.test:8000/webaudio/resources/audit.js:51:9
[task 2020-07-09T00:46:18.681Z] 00:46:18     INFO - done@http://web-platform.test:8000/webaudio/resources/audit.js:1227:19
[task 2020-07-09T00:46:18.682Z] 00:46:18     INFO - spn.onaudioprocess@http://web-platform.test:8000/webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html:105:26
[task 2020-07-09T00:46:18.682Z] 00:46:18     INFO - 
[task 2020-07-09T00:46:18.683Z] 00:46:18     INFO - TEST-UNEXPECTED-FAIL | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | # AUDIT TASK RUNNER FINISHED: 1 out of 1 tasks were failed. - assert_true: expected true got false
[task 2020-07-09T00:46:18.683Z] 00:46:18     INFO - _logFailed/<@http://web-platform.test:8000/webaudio/resources/audit.js:52:18
[task 2020-07-09T00:46:18.683Z] 00:46:18     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1977:25
[task 2020-07-09T00:46:18.683Z] 00:46:18     INFO - test@http://web-platform.test:8000/resources/testharness.js:535:30
[task 2020-07-09T00:46:18.684Z] 00:46:18     INFO - _logFailed@http://web-platform.test:8000/webaudio/resources/audit.js:51:9
[task 2020-07-09T00:46:18.684Z] 00:46:18     INFO - _finish@http://web-platform.test:8000/webaudio/resources/audit.js:1284:19
[task 2020-07-09T00:46:18.685Z] 00:46:18     INFO - run/<@http://web-platform.test:8000/webaudio/resources/audit.js:1345:31
[task 2020-07-09T00:46:18.685Z] 00:46:18     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1977:25
[task 2020-07-09T00:46:18.685Z] 00:46:18     INFO - promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:592:36
[task 2020-07-09T00:46:18.685Z] 00:46:18     INFO - promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:591:20
[task 2020-07-09T00:46:18.685Z] 00:46:18     INFO - TEST-OK | /webaudio/the-audio-api/the-convolvernode-interface/realtime-conv.html | took 1573ms

Set release status flags based on info from the regressing bug 1650429

status-firefox78: --- → unaffected
status-firefox79: --- → unaffected
status-firefox80: --- → affected
status-firefox-esr68: --- → unaffected
status-firefox-esr78: --- → unaffected

Pretty minor list of failures. Wondering if we're going to try and improve this or mark it fix-optional for 80?

status-firefox80: affected → fix-optional
Flags: needinfo?(padenot)

Probably under-runs on the test machines, which degrades the SNR, since this is the real-time convolver test. It's OK to move the threshold. Karl, what do you think?

Flags: needinfo?(padenot) → needinfo?(karlt)

This kind of failure shows up frequently because wpt updates do not generate expected results for linux1804-32-shippable.

It is not a regression in the product, but in the expectation data.

I assume this is failing only on linux1804-32-shippable due to the different FFT code used on that platform.
The consistency of the failure and the small error values imply that the failure is not due to under-runs.

Fixing bug 1176049 may resolve the failure, but there may also be better FFT options than ffmpeg.

If the error in every sample had magnitude equal to one unit in the last place (ULP) for the representation of the peak of the ramp (16384), then we would expect signalSum/errorSum to be (16384 * 2^-23)^-2 ≅ 10^5.4, which would correspond to a minRequiredSNR ≅ 54.

Increasing minRequiredSNR to 83, would permit the quadratic mean of errors to be (10^8.3)^-0.5 ≅ 0.036 * 16384 * 2^-23, implying that most of the samples have error significantly less than one ULP for the representation of the peak of the ramp. So, yes, moving the threshold seems reasonable to me.

There would be no value in updating the expectation data on m-c because changes will be reverted in future wpt updates.
https://hg.mozilla.org/mozilla-central/rev/d59f0a83a7e9#l140.2
But I'm puzzled about how the expectation data was initially added in a wpt update:
https://hg.mozilla.org/mozilla-central/rev/0c19eb0743706a0e69155c90cb47b3cb461776c4#l263.3

Depends on: 1176049
Flags: needinfo?(karlt)

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE

New occurrence: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313131724&repo=mozilla-central&lineNumber=3743

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

There have been 22 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-09-05&endday=2020-09-12&tree=trunk&bug=1651569

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=315494822&repo=mozilla-central&lineNumber=3384

Paul, could you assign this to someone?

Flags: needinfo?(padenot)
Whiteboard: [stockwell needswork:owner]

Karl, would you mind bumping the SNR to an appropriate value here until we figure out a proper solution? I wonder why it's intermittent at all, that said, if it's not under-running.

Flags: needinfo?(padenot) → needinfo?(karlt)

as used in Firefox on some platforms.

minRequiredSNR = 83 permits the quadratic mean of errors to be up to
(10^8.3)^-0.5 ≅ 0.036 * 16384 * 2^-23, implying that most of the samples would
have error significantly less than one ULP for the representation of the peak
of the ramp (16384).

Assignee: nobody → karlt
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0194a0d1befc
reduce required SNR to accept KissFFT ConvolverNode implementation r=padenot
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/25553 for changes under testing/web-platform/tests
Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner], [wptsync upstream]

https://hg.mozilla.org/mozilla-central/rev/0194a0d1befc

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox82: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 82 Branch

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

status-firefox81: --- → ?
Upstream PR merged by moz-wptsync-bot
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: