Investigate occasional "malformed signature" errors being seen in production

VERIFIED FIXED

Status

Cloud Services
Server: Token
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: rfkelly, Assigned: rfkelly)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

(Assignee)

Description

3 years ago
As noted in Bug 1044532, we are seeing occasional verification failures with the error "malformed signature".  AFAICT this means one of our sync clients is sending badly-formed assertions.  We need to monitor for these are figure out what's going on.

This is a top-level tracking bug for me to keep the search organised.  See https://github.com/mozilla/browserid-verifier/issues/59 and some recent improvements to jwcrypto for how we get at the raw information, which will need to go out in the next browserid-verifier release.
Whiteboard: [qa+]
(Assignee)

Comment 1

3 years ago
:whd created this kibana dash for the verifier logs on response to Bug 1044532:

  https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Bug%201044532

For further investigation of this bug, it would be great to make a more official version of that, and to send the logs from the tokenserver-specific verifier instances along there as well.

Comment 2

3 years ago
I'll work on getting the more official verifier dashboard(s) set up with the stage deploy in Bug 1063350.
(Assignee)

Comment 3

3 years ago
Adding deploy bug as blocker
Depends on: 1066370

Comment 4

3 years ago
POC for prod: https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Token%20Local%20Verifier%20POC

:rfkelly currently verifier logs being indexed in that cluster are limited to "bid.v2" (other types include "bid.summary" "bid.server" "bid.ccverifier"). Let me know if you want any of these other types indexed.
(Assignee)

Comment 5

3 years ago
The current dashboard shows no "malformed" errors, the only reason for failures seems to be "expired".  Let's let it accumulate for a few days and see what shows up.
(Assignee)

Comment 6

3 years ago
I've been checking this every couple of days, and I can not see even a single "malformed" error in the logs.  Perhaps the original problem has since been resolved.  In any case, i don't see further value in keeping this bug open, and we now have the logging in place to debug any future errors like this.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
OK. Thanks.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.