Closed Bug 1560534 Opened 5 years ago Closed 5 years ago

Multiple wpt failures when Gecko 69 merges to Beta on 2019-07-01

Categories

(Testing :: web-platform-tests, defect)

Product:
Testing
Component:
web-platform-tests
Version:
Version 3
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr60 unaffected, firefox67 unaffected, firefox67.0.1 unaffected, firefox68 unaffected, firefox69+ fixed, firefox70 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla70
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox67 --- unaffected
firefox67.0.1 --- unaffected
firefox68 --- unaffected
firefox69 + fixed
firefox70 --- fixed

People

(Reporter: opoprus, Assigned: impossibus)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file, 10 obsolete files)

WebAudio metadata update based on Comment 24
23.66 KB, patch
Details | Diff | Splinter Review

Also Maja could you look over these Android wpt failures that are from the wpt-sync push:
/2dcontext/shadows/2d.shadow.enable.x.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252954801&repo=try&lineNumber=1399
/media-capabilities/decodingInfo.any.worker.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948390&repo=try&lineNumber=3295
/html/webappapis/scripting/processing-model-2/unhandled-promise-rejections/promise-rejection-events.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948392&repo=try&lineNumber=2685 - followed in Bug 1560511
presentation-api/controlling-ua/PresentationRequest_sandboxing_success.https.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948392&repo=try&lineNumber=2963
/content-security-policy/navigate-to/link-click-cross-origin-allowed.sub.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948403&repo=try&lineNumber=1686
/media-source/mediasource-play-then-seek-back.html - /media-source/mediasource-seek-beyond-duration.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948464&repo=try&lineNumber=3554

Flags: needinfo?(mjzffr)

Tried to disable most of the failing tests on beta with https://hg.mozilla.org/try/rev/be3f1cd83049facd6fd91e6748cc3ff17a0db647 but had no success, there are still a lot of them failing: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=6610df6b0be761b27e405c8a81b479e9b898623d&searchStr=wpt&selectedJob=253021518
Most of them are connected to /webaudio/the-audio-api/ or to /html/dom/interfaces.https.html.

Disabling the tests isn't usually what we do in these cases. Typically the correct patch is either to set the prefs that are disabled on beta or above using the expectation metadata file under testing/web-platform/meta or just set the correct expectation in those files. I'm not sure what the correct prefs to set are in this case; you might need to ask the owners of the relevant code (e.g. the webaudio owners).

Flags: needinfo?(james)
Attached patch webaudio-updates.patch (obsolete) — — Splinter Review

(In reply to Cosmin Sabou [:CosminS] from comment #1)

Also Maja could you look over these Android wpt failures that are from the wpt-sync push:

Note that wpt on Fennec is being turned off in Bug 1558334, so I'm going to ignore any Fennec-only items in your list.

/2dcontext/shadows/2d.shadow.enable.x.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252954801&repo=try&lineNumber=1399

Fennec only.

/media-capabilities/decodingInfo.any.worker.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948390&repo=try&lineNumber=3295

This this mostly behaves as expected (crash), so I don't think there's anything new to look into.

/html/webappapis/scripting/processing-model-2/unhandled-promise-rejections/promise-rejection-events.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948392&repo=try&lineNumber=2685 - followed in Bug 1560511

Fennec only.

presentation-api/controlling-ua/PresentationRequest_sandboxing_success.https.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948392&repo=try&lineNumber=2963

This needs to be looked at by the relevant team. Maybe a pref issue? It behaves as expected on m-c.

/content-security-policy/navigate-to/link-click-cross-origin-allowed.sub.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948403&repo=try&lineNumber=1686

Fennec only.

/media-source/mediasource-play-then-seek-back.html - /media-source/mediasource-seek-beyond-duration.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948464&repo=try&lineNumber=3554

I think it's fine to just update the metadata to expect a pass on android debug.

Flags: needinfo?(mjzffr)

(In reply to Cristina Coroiu [:ccoroiu] from comment #6)

The wpt failures are still failing after importing the patch:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1e9c33aef87443da3b4d9a9cdfb24643ae4eebcd&selectedJob=253707559

Sorry, I'll do a try job this time to make sure I catch all of the failures. Should have an updated patch later today.

Flags: needinfo?(dminor)
Attached patch bug-1560534-fix.patch (obsolete) — — Splinter Review
Attachment #9074217 - Attachment is obsolete: true

This updates the webaudio and html/dom/interfaces expectations.

Attachment #9074584 - Attachment is obsolete: true

Thank you for the answer.
Could you please write a patch with expectations update for beta for the following:
(In reply to Maja Frydrychowicz :maja_zf (she/her) (UTC-4) (maja@mozilla.com) from comment #5)

/media-capabilities/decodingInfo.any.worker.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948390&repo=try&lineNumber=3295
This this mostly behaves as expected (crash), so I don't think there's anything new to look into.

Yes, this is green on m-c but when it gets merged into beta next week it will fail as TEST-UNEXPECTED-OK. https://tinyurl.com/y35cabd3

/media-source/mediasource-play-then-seek-back.html - /media-source/mediasource-seek-beyond-duration.html - https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252948464&repo=try&lineNumber=3554

I think it's fine to just update the metadata to expect a pass on android debug.

Also for this one. https://tinyurl.com/yye34na2
Thank you.

The other fix from Dan holds and the tests are no longer failing: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&revision=05e1c33f877b884e4428c5ca8a4347bcf428fbdb&searchStr=wpt&selectedJob=253917107

Flags: needinfo?(mjzffr)

Importing the latest patch does not, yesterday the revision was empty: https://hg.mozilla.org/try/rev/77e0860e2f1c358eaaacfe782c3130c590a2bde6
When trying to import today I get the following:

applying https://bug1560534.bmoattachments.org/attachment.cgi?id=9074935
abort: failed to synchronize metadata for "testing/web-platform/meta/media-capabilities/decodingInfo.any.js.ini"

I edited the files manually like in the patch and added the same commit message.

Will post the results when the tests finish.

Flags: needinfo?(aryx.bugmail)
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/integration/mozilla-inbound/rev/db11508e43e7
Update wpt test expectations for webaudio. r=aryx
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b348e47e897
Update wpt metadata for when Gecko 69 merges to Beta. r=Aryx

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #16)

Backed out the webaudio patch for wpt failures:

https://hg.mozilla.org/integration/mozilla-inbound/rev/29066d3463d49a9887a533eacbc1d8382e23338c

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=7b348e47e8978f6896bd4c65d790d245adf80865

This patch was intended for a mozilla-central as Beta simulation, so it's not surprising it would fail if landed to mozilla-inbound. I assume we have different prefs or code enabled on Nightly than we do on Beta, and that is why we see different results with the tests, but I'm not familiar with the audio worklets code, or how we handle wpt test updates, for that matter. It probably makes sense to follow up with someone more familiar with the code in question. Karl Tomlinson had audio worklets expertise and I think Maja is familiar with wpt updates.

Flags: needinfo?(dminor)

https://hg.mozilla.org/mozilla-central/rev/7b348e47e897

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox69: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Assignee: nobody → mjzffr
Status: RESOLVED → REOPENED
status-firefox69: fixed → affected
Resolution: FIXED → ---
Target Milestone: mozilla69 → ---

Karl, Maja, could you please take a look over what Dan was referring at in Comment 17? This is failing massively on beta. https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=testfailed%2Cbusted%2Cexception&revision=625005a94d4d324f9e9b9f14ebe8986dc6bdcc39&searchStr=wpt&selectedJob=254319477

Flags: needinfo?(mjzffr)
Flags: needinfo?(karlt)

I've updated Dan's patch to make the expectations conditional on "release_or_beta". There may be a better way to deal with this discrepancy between beta and inbound -- perhaps prefs?

Attachment #9074587 - Attachment is obsolete: true
Flags: needinfo?(mjzffr)

Thank you for reverting those changes, Maja.

James, has something changed in the way wpt imports are performed?

Multiple existing expectation annotations were removed with the import for bug 1558851.
e.g. https://hg.mozilla.org/mozilla-central/rev/a91b29ff03f92e1ad64e2e5339a451351f8c2b16#l538.23
Search also for release_or_beta.

Such annotations have existed since December and have survived previous imports.
https://hg.mozilla.org/mozilla-central/rev/bdda06babfaaed1cd12fec8ffdd107b5ca315cc5

Flags: needinfo?(karlt) → needinfo?(james)

(In reply to Maja Frydrychowicz :maja_zf (she/her) (UTC-4) (maja@mozilla.com) from comment #20)

There may be a better way to deal with this discrepancy between beta and inbound -- perhaps prefs?

Prefs are not always an available or desirable solution. Some code is built only on Nightly, and sometimes we prefer to test the shipping configuration on shipping channels.

This version of the patch properly reverts the metadata changes Karl mentioned.

Attachment #9074935 - Attachment is obsolete: true

Maja, unfortunately the latest version of the patch doesn't work on m-c.
integration: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f39c46e8b3af351d7af47d3675c91cdb0de77037&group_state=expanded&selectedJob=255019952
beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&revision=83d4d9b30ee9cc9f75594df7a758219871bc6d3d&group_state=expanded&selectedJob=255020699

Tried updating the expections because the failures on /webaudio/idlharness.https.window.html were TEST-UNEXPECTED-PASS with these modifications: https://hg.mozilla.org/try/diff/33e9b8745d5b7159c2e16b2b2f2ff5b8f2f04f88/testing/web-platform/meta/webaudio/idlharness.https.window.js.ini but they turned into TEST-UNEXPECTED-FAIL
integration: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=1b14f8333d92bc20647d6b53e0f9397e63f9e36c&selectedJob=255037408
beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=bf3b9989b5f6890ae248224ab038a567e9db1ec2

Also tried to use this pref modification but they still failed on m-c. https://hg.mozilla.org/try/diff/beea09edf7301ee2df4261b0f8150c85477152c9/testing/web-platform/meta/webaudio/idlharness.https.window.js.ini#l1.4
integration: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=0f5d1489924084c77cec6d4b85930072d84271ad&selectedJob=255049171
beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=0def22fb82a205451a14988f1b9c432ecf6348c6&selectedJob=255047840

Also for those wpt10 failures (https://tinyurl.com/yyu3b8y3) not to fail on integration these changes seem to fix them: https://hg.mozilla.org/try/diff/33e9b8745d5b7159c2e16b2b2f2ff5b8f2f04f88/testing/web-platform/meta/webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/no-cors.https.html.ini
Could you please take a look over this? Thank you.

Flags: needinfo?(mjzffr)

Thank you, Cosmin, for the detailed report and for trying out some solutions.

(In reply to Cosmin Sabou [:CosminS] from comment #24)

Tried updating the expections because the failures on /webaudio/idlharness.https.window.html were TEST-UNEXPECTED-PASS with these modifications: https://hg.mozilla.org/try/diff/33e9b8745d5b7159c2e16b2b2f2ff5b8f2f04f88/testing/web-platform/meta/webaudio/idlharness.https.window.js.ini but they turned into TEST-UNEXPECTED-FAIL
integration: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=1b14f8333d92bc20647d6b53e0f9397e63f9e36c&selectedJob=255037408
beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=bf3b9989b5f6890ae248224ab038a567e9db1ec2

I don't think anything turned into TEST-UNEXPECTED-FAIL, actually, just expectations that were already correct were unnecessarily modified in that experiment.

Updating the attachment based on the provided data shortly...

Flags: needinfo?(mjzffr)

Maja, trying to import the new patch I get:
applying https://bug1560534.bmoattachments.org/attachment.cgi?id=9076588
abort: bad hunk #1 @@ -1,9 +1,11 @@
(10 9 11 11)
Could you please fix this?

Flags: needinfo?(mjzffr)

Sorry about that, not sure what went wrong there. This one applies cleanly for me.

Attachment #9076588 - Attachment is obsolete: true
Flags: needinfo?(mjzffr)

I fixed some trailing whitespace, too.

Attachment #9076602 - Attachment is obsolete: true

There are still some errors complaining about idlharness.https.window.js.ini while importing the latest patch:
Beta simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=255469674&resultStatus=testfailed%2Cbusted%2Cexception&revision=d9ec3b7ce965b49f92d98e9d471193e02e12582c

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255469674&repo=try&lineNumber=1032

Log snippet:

[task 2019-07-09T12:32:58.123Z] 12:32:58 CRITICAL - File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptmanifest/parser.py", line 596, in value_block
[task 2019-07-09T12:32:58.123Z] 12:32:58 CRITICAL - "Token '{}' is not a known type".format(self.token[0]))
[task 2019-07-09T12:32:58.124Z] 12:32:58 CRITICAL - wptrunner.wptmanifest.parser.ParseError: Token 'ident' is not a known type: /builds/worker/workspace/build/tests/web-platform/meta/webaudio/idlharness.https.window.js.ini line 297
[task 2019-07-09T12:32:58.339Z] 12:32:58 ERROR - Return code: 1
[task 2019-07-09T12:32:58.340Z] 12:32:58 ERROR - No checks run.
[task 2019-07-09T12:32:58.340Z] 12:32:58 ERROR - No suite end message was emitted by this harness.
[task 2019-07-09T12:32:58.340Z] 12:32:58 CRITICAL - # TBPL FAILURE #
[task 2019-07-09T12:32:58.340Z] 12:32:58 WARNING - setting return code to 2

Flags: needinfo?(mjzffr)

My bad, sorry.

Attachment #9076604 - Attachment is obsolete: true
Flags: needinfo?(mjzffr)

(In reply to Karl Tomlinson (:karlt) from comment #21)

Thank you for reverting those changes, Maja.

James, has something changed in the way wpt imports are performed?

Multiple existing expectation annotations were removed with the import for bug 1558851.
e.g. https://hg.mozilla.org/mozilla-central/rev/a91b29ff03f92e1ad64e2e5339a451351f8c2b16#l538.23
Search also for release_or_beta.

Bleh. Yes this is a regression from bug 1469893 and bug 1545143 where we start to consider a try run against central to invalidate all previous conditions on import. But of course we should exclude conditions depending on properties outside the set of updatable properties from that. Filed bug 1564917 on that regression.

Flags: needinfo?(james)
Attachment #9076810 - Flags: review+
Attachment #9075704 - Attachment is obsolete: true
Flags: needinfo?(mjzffr)

Pushed by apavel@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9fb4c1417a0b
Update wpt test expectations for webaudio

Keywords: checkin-needed

This still fails on integration trees:
Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=9fb4c1417a0b5039ffb8422e383fd9667dff6ff9&searchStr=linux%2Cwpt&selectedJob=255771549

Failure logs: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255770949&repo=autoland&lineNumber=40641
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255761226&repo=autoland&lineNumber=27286

Backout link: https://hg.mozilla.org/integration/autoland/rev/f35c24afa7850fc23017f5ebd9b218c57c75fead

[task 2019-07-10T17:31:17.527Z] 17:31:17 INFO - TEST-PASS | /webaudio/idlharness.https.window.html | AudioParamMap interface object name
[task 2019-07-10T17:31:17.527Z] 17:31:17 INFO - TEST-UNEXPECTED-PASS | /webaudio/idlharness.https.window.html | AudioParamMap interface: existence and properties of interface prototype object - expected FAIL
[task 2019-07-10T17:31:17.527Z] 17:31:17 INFO - TEST-INFO | expected FAIL

Flags: needinfo?(mjzffr)
Attachment #9076810 - Attachment is obsolete: true
Flags: needinfo?(mjzffr)

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)
Pushed by rmaries@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/da0a3ff93142
Update wpt test expectations for webaudio. a=test-only

https://hg.mozilla.org/mozilla-central/rev/da0a3ff93142

Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
status-firefox70: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla70

Yes, let's include it in the uplifts today.

Flags: needinfo?(aryx.bugmail)
Flags: needinfo?(james)
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: