1045502 - Investigate occasional "malformed signature" errors being seen in production

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Token, defect)

Description

[Ryan Kelly [:rfkelly]]

As noted in Bug 1044532, we are seeing occasional verification failures with the error "malformed signature".  AFAICT this means one of our sync clients is sending badly-formed assertions.  We need to monitor for these are figure out what's going on.

This is a top-level tracking bug for me to keep the search organised.  See https://github.com/mozilla/browserid-verifier/issues/59 and some recent improvements to jwcrypto for how we get at the raw information, which will need to go out in the next browserid-verifier release.

Comment 1

[Ryan Kelly [:rfkelly]]

:whd created this kibana dash for the verifier logs on response to Bug 1044532:

  https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Bug%201044532

For further investigation of this bug, it would be great to make a more official version of that, and to send the logs from the tokenserver-specific verifier instances along there as well.

Comment 2

[Wesley Dawson [:whd]]

I'll work on getting the more official verifier dashboard(s) set up with the stage deploy in Bug 1063350.

Comment 3

[Ryan Kelly [:rfkelly]]

Adding deploy bug as blocker

Comment 4

[Wesley Dawson [:whd]]

POC for prod: https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Token%20Local%20Verifier%20POC

:rfkelly currently verifier logs being indexed in that cluster are limited to "bid.v2" (other types include "bid.summary" "bid.server" "bid.ccverifier"). Let me know if you want any of these other types indexed.

Comment 5

[Ryan Kelly [:rfkelly]]

The current dashboard shows no "malformed" errors, the only reason for failures seems to be "expired".  Let's let it accumulate for a few days and see what shows up.

Comment 6

[Ryan Kelly [:rfkelly]]

I've been checking this every couple of days, and I can not see even a single "malformed" error in the logs.  Perhaps the original problem has since been resolved.  In any case, i don't see further value in keeping this bug open, and we now have the logging in place to debug any future errors like this.

Comment 7

[James Bonacci [:jbonacci]]

OK. Thanks.