744892 - 500-level errors not always showing up the pencil graphs for browserid

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[John Morrison [:jrgm]]

I triggered a couple of 500 errors while checking the fix for hotfix 1429 for
browserid. I noticed that 500 errors are not always reflected in the pencil
graphs on stage for _qps_other.

Here are the graphs for that time:
Webhead:
> http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead/bid_webhead_qps_other?start=2012-04-12+11%3A30&duration=30+minutes&width=800&height=200
> http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead_wsapi/bid_webhead_qps_other?start=2012-04-12+11%3A30&duration=30+minutes&width=800&height=200
Dbwriter:
> http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_dbwriter_wsapi/bid_dbwriter_qps_other?start=2012-04-12+11%3A30&duration=30+minutes&width=800&height=200
> http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_dbwriter/bid_dbwriter_qps_other?start=2012-04-12+11%3A30&duration=30+minutes&width=800&height=200

And the raw logs show:

> @@ web1 @@
> {"level":"info","message":"127.0.0.1 - - [Thu, 12 Apr 2012 18:42:17 GMT] \"POST /wsapi/stage_user HTTP/1.0\" 500 - \"-\" \"-\"","timestamp":"2012-04-12T18:42:17.046Z"}
> @@ web2 @@
> {"level":"info","message":"127.0.0.1 - - [Thu, 12 Apr 2012 18:41:51 GMT] \"POST /wsapi/stage_user HTTP/1.0\" 500 - \"-\" \"-\"","timestamp":"2012-04-12T18:41:51.844Z"}
> @@ web3 @@
>  nothing
> 
> @@ sweb1 @@
>  nothing
> @@ sweb2 @@
> {"level":"info","message":"127.0.0.1 - - [Thu, 12 Apr 2012 18:41:51 GMT] \"POST /wsapi/stage_user HTTP/1.0\" 500 - \"-\" \"-\"","timestamp":"2012-04-12T18:41:51.838Z"}
> @@ sweb3 @@
> {"level":"info","message":"127.0.0.1 - - [Thu, 12 Apr 2012 18:42:17 GMT] \"POST /wsapi/stage_user HTTP/1.0\" 500 - \"-\" \"-\"","timestamp":"2012-04-12T18:42:17.041Z"}

web2, sweb2, sweb3 don't show any red for that time period, but should.