1399334 - Intermittent dom/u2f/tests/test_register_sign.html | /tests/dom/u2f/tests/frame_register_sign.html: Register attestation signature verified

Status: RESOLVED FIXED

(Core :: DOM: Device Interfaces, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=130526088&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/P6UdKEw8T8-PKtYRtVQZSw/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

39 failures in 1032 pushes (0.038 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 13
* autoland: 12
* mozilla-central: 11
* try: 3

Platform breakdown:
* linux64: 10
* windows7-32: 9
* windows10-64: 5
* windows7-32-stylo-disabled: 3
* windows10-64-stylo-disabled: 3
* osx-10-10: 3
* linux32-stylo-disabled: 2
* linux32: 2
* macosx64-stylo-disabled: 1
* linux64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1399334&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 2

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Hi J.C., is this recently-rose-up intermittent failure on your radar?

Comment 3

[J.C. Jones [:jcj] (he/they)]

It wasn't, but it is now. Thanks!

Comment 4

[J.C. Jones [:jcj] (he/they)]

I've a test to print the whole certificate that is offensive whenever it comes up.

Comment 5

[J.C. Jones [:jcj] (he/they)]

Marking checkin-needed w/ leave-open so we can see what the real failure is

Comment 6

[Ryan VanderMeulen [:RyanVM]]

I have no way of landing patches from Phabricator at the moment. Please attach it here or to MozReview instead.

Comment 7

[J.C. Jones [:jcj] (he/they)]

Attachment: 1399334-intermittent_u2f_sign.patch

Here goes!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0902f7275334
Add more debugging to see why certificates aren't valid. r=ttaubert

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0902f7275334

Comment 10

[Intermittent Failures Robot]

19 failures in 943 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 10
* mozilla-central: 6
* mozilla-inbound: 2
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 5
* windows7-32: 4
* windows10-64-stylo-disabled: 2
* windows10-64: 2
* linux32-stylo-disabled: 2
* linux32: 2
* linux64-nightly: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1399334&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 11

[J.C. Jones [:jcj] (he/they)]

I've dug through this and am still not clear on what the cause is.

This failure is new simply because I added the check of whether the certificate's signature was valid to this test relatively recently. There are other tests that check that out, particularly all of the equivalent webauthn tests. 

I'm tempted to go remove the check again. We don't ship the soft token, it's purely for testing...

Comment 12

[Intermittent Failures Robot]

26 failures in 885 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* mozilla-beta: 4
* mozilla-central: 3
* try: 2

Platform breakdown:
* osx-10-10: 6
* linux64: 4
* linux32-stylo-disabled: 3
* windows7-32-stylo-disabled: 2
* windows7-32: 2
* windows10-64-stylo-disabled: 2
* macosx64-stylo-disabled: 2
* linux64-stylo-disabled: 2
* linux32: 2
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1399334&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 13

[J.C. Jones [:jcj] (he/they)]

I think this bug is in PKI.js, but I haven't isolated where. Most of the certificates reported in brasstacks are truncated by the logger (oops), but some are complete, and those which are complete verify fine with OpenSSL and NSS, so whatever is wrong is a false positive.

I'm going to remove the call to attestationCert.verify() --- it's not even testing anything relevant to U2F.

Comment 14

[J.C. Jones [:jcj] (he/they)]

Examples for posterity:


https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-beta&job_id=133848683&lineNumber=4971

-----BEGIN CERTIFICATE-----
MIIBMTCB2aADAgECAgUAq02MeTAKBggqhkjOPQQDAjAhMR8wHQYDVQQDExZGaXJl
Zm94IFUyRiBTb2Z0IFRva2VuMB4XDTE3MDkyNzE2MDAzN1oXDTE3MDkyOTE2MDAz
N1owITEfMB0GA1UEAxMWRmlyZWZveCBVMkYgU29mdCBUb2tlbjBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABH4RcxsrUCg09aLnYY3Bi6kUqjiQ1WtZkjpRs+gxyTKW
Tg4DIWRket4Ok8gv+a4W6JaIzt//orbJ3bo0uWKDUVAwCgYIKoZIzj0EAwIDRwAw
RAIgAORpKjwu9wKJkhIxMYiTHC0WzTb99G58EkQ5ejteHMUCIFYa6ilDjW3BAtm4
fJNr3yK/K2QnP3EfMqO+c+el9C4O
-----END CERTIFICATE-----

openssl verify -purpose any -CAfile /tmp/cert.pem /tmp/cert.pem
/tmp/cert.pem: /CN=Firefox U2F Soft Token
error 10 at 0 depth lookup:certificate has expired
OK


https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=134282931&lineNumber=2673

-----BEGIN CERTIFICATE-----
MIIBMTCB2aADAgECAgUA55x6LTAKBggqhkjOPQQDAjAhMR8wHQYDVQQDExZGaXJl
Zm94IFUyRiBTb2Z0IFRva2VuMB4XDTE3MDkzMDE5MjIzMloXDTE3MTAwMjE5MjIz
MlowITEfMB0GA1UEAxMWRmlyZWZveCBVMkYgU29mdCBUb2tlbjBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABIWu4L8ky7s8I7qVv+JwMRHpippH4b6h7rN0jlKpFbHK
hnEwaCPLrTx04Eh9xT4GK9JWuuP759hnAxsWD5wk0H0wCgYIKoZIzj0EAwIDRwAw
RAIgRIeRcn6LkwU8VOmX+mdQ3jUQrUOp5f2xH/qBECGi5EcCIADBjsm/EDKkAwLZ
pGdX7+N+kgf9No4uuLV4dsNVJ1pa
-----END CERTIFICATE-----

openssl verify -purpose any -CAfile /tmp/cert2.pem /tmp/cert2.pem
/tmp/cert2.pem: OK

Comment 15

[J.C. Jones [:jcj] (he/they)]

Attachment: Bug 1399334 - Workaround buggy pki.js cert verifier implementation

There's an intermittent on the call attestationCert.verify() to test the self-
signed cert from our not-shipped software U2F implementation. Collection of the
intermittents shows these certs are fine, and should verify correctly, but they
don't. The bug must be in pki.js, which is out-of-scope as we only use it for
mochitests.

This patch removes the offending call to xxxx.verify(), because it doesn't
really matter whether the self-signed-cert looks OK to pki.js; we just need
the public key from inside it to proceed with the rest of the tests.

As an example of a so-called "invalid" self-signed cert that failed, we have:

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=134282931&lineNumber=2673

-----BEGIN CERTIFICATE-----
MIIBMTCB2aADAgECAgUA55x6LTAKBggqhkjOPQQDAjAhMR8wHQYDVQQDExZGaXJl
Zm94IFUyRiBTb2Z0IFRva2VuMB4XDTE3MDkzMDE5MjIzMloXDTE3MTAwMjE5MjIz
MlowITEfMB0GA1UEAxMWRmlyZWZveCBVMkYgU29mdCBUb2tlbjBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABIWu4L8ky7s8I7qVv+JwMRHpippH4b6h7rN0jlKpFbHK
hnEwaCPLrTx04Eh9xT4GK9JWuuP759hnAxsWD5wk0H0wCgYIKoZIzj0EAwIDRwAw
RAIgRIeRcn6LkwU8VOmX+mdQ3jUQrUOp5f2xH/qBECGi5EcCIADBjsm/EDKkAwLZ
pGdX7+N+kgf9No4uuLV4dsNVJ1pa
-----END CERTIFICATE-----

There's nothing wrong with this cert, actually. Checking it with OpenSSL shows
all OK:

openssl verify -purpose any -CAfile /tmp/cert2.pem /tmp/cert2.pem
/tmp/cert2.pem: OK

So this intermittent is a bug outside of our U2F and U2F test soft token code.

Review commit: https://reviewboard.mozilla.org/r/185742/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/185742/

Comment 16

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8914437 [details]
Bug 1399334 - Workaround buggy pki.js cert verifier implementation

https://reviewboard.mozilla.org/r/185742/#review190690

Seems reasonable. Maybe file a bug wherever pki.js tracks its issues with a certificate that fails to verify?

Comment 17

[J.C. Jones [:jcj] (he/they)]

That's https://github.com/PeculiarVentures/PKI.js ; good idea, will do!

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/42e5371334d9
Workaround buggy pki.js cert verifier implementation r=keeler

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/42e5371334d9

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f66ff4d9ddbe

Comment 22

[Intermittent Failures Robot]

5 failures in 824 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2
* mozilla-inbound: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 1
* windows10-64-stylo-disabled: 1
* osx-10-10: 1
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1399334&startday=2017-10-02&endday=2017-10-08&tree=all