1546662 - webauth tests have become unstable

Status: RESOLVED FIXED

(Core :: DOM: Web Authentication, defect, P2)

Description

[James Graham [:jgraham]]

After the latest wpt import the following tests keep switching between TIMEOUT/NOTRUN and whatever their real status is (PASS or FAIL). I've already been backed out once for this and I really don't want to be again, so as a short term measure I'm going to disable the follwoing list of tests:

/webauthn/createcredential-badargs-authnrselection.https.html
/webauthn/createcredential-badargs-rp.https.html
/webauthn/createcredential-badargs-user.https.html
/webauthn/createcredential-excludecredentials.https.html
/webauthn/createcredential-extensions.https.html
/webauthn/createcredential-passing.https.html
/webauthn/getcredential-badargs-rpid.https.html
/webauthn/getcredential-badargs-userverification.https.html
/webauthn/getcredential-extensions.https.html
/webauthn/getcredential-passing.https.html
/webauthn/getcredential-timeout.https.html

It may just be that they need a longer timeout. It may be that we need some pref to stop a prompt from interfering with the tests. I hope someone working on webauthn can investigate and reenable these as soon as possible.

Comment 1

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/da523e237c3d
Disable webauthn tests for instability, a=testonly

Comment 2

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/da523e237c3d

Comment 3

[J.C. Jones [:jcj] (he/they)]

It's unclear why the WPTs became unstable when our internal tests didn't. Nevertheless, can't get to this in 68.

Comment 4

[Boris Zbarsky [:bzbarsky]]

J.C. do you know what the status is here? There's various stuff that our internal tests do not test that the wpts do, but with them disabled we don't have that test coverage...

Comment 5

[J.C. Jones [:jcj] (he/they)]

I have to rewrite the plumbing for WebAuthnManager and rewrite the soft token to implement the VirtualDriver interface (bug 1587563), when I work on WebAuthn again.

Comment 6

[John Schanck [:jschanck]]

Attachment: Bug 1546662 - reenable webauthn wpts. r=keeler

Depends on D162624

Comment 7

[John Schanck [:jschanck]]

Attachment: Bug 1546662 - reenable webauthn wpts. r=keeler

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/965ced29c013
reenable webauthn wpts. r=keeler

Comment 9

[Narcis Beleuzu [:NarcisB]]

Backed out for bustages on command.rs

Backout link: https://hg.mozilla.org/integration/autoland/rev/5954ce91542de618e7ccfc10bdfc5368b9160cb2
Log link: https://treeherder.mozilla.org/logviewer?job_id=425235238&repo=autoland&lineNumber=5933

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ddb81e7713a7
reenable webauthn wpts. r=keeler

Comment 11

[Cristian Tuns]

Backed out for causing wpt failures in response.py

• Backout link
• Push with failures
• Failure Log
• Failure line: TEST-UNEXPECTED-FAIL | /webdriver/tests/classic/new_session/response.py | test_capability_type[webauthn:extension:credBlob-bool] - AssertionError: assert 'webauthn:extension:credBlob' in {'acceptInsecureCerts': False, 'browserName': 'firefox', 'browserVersion': '118.0a1', 'moz:accessibilityChecks': False, ...}

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f93f57b96053
reenable webauthn wpts. r=keeler

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/f93f57b96053