As noted in Bug 1044532, we are seeing occasional verification failures with the error "malformed signature". AFAICT this means one of our sync clients is sending badly-formed assertions. We need to monitor for these are figure out what's going on. This is a top-level tracking bug for me to keep the search organised. See https://github.com/mozilla/browserid-verifier/issues/59 and some recent improvements to jwcrypto for how we get at the raw information, which will need to go out in the next browserid-verifier release.
:whd created this kibana dash for the verifier logs on response to Bug 1044532: https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Bug%201044532 For further investigation of this bug, it would be great to make a more official version of that, and to send the logs from the tokenserver-specific verifier instances along there as well.
I'll work on getting the more official verifier dashboard(s) set up with the stage deploy in Bug 1063350.
Adding deploy bug as blocker
POC for prod: https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/elasticsearch/Token%20Local%20Verifier%20POC :rfkelly currently verifier logs being indexed in that cluster are limited to "bid.v2" (other types include "bid.summary" "bid.server" "bid.ccverifier"). Let me know if you want any of these other types indexed.
The current dashboard shows no "malformed" errors, the only reason for failures seems to be "expired". Let's let it accumulate for a few days and see what shows up.
I've been checking this every couple of days, and I can not see even a single "malformed" error in the logs. Perhaps the original problem has since been resolved. In any case, i don't see further value in keeping this bug open, and we now have the logging in place to debug any future errors like this.