1187880 - Reported (actual?) date range doesn't seem to update.

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[David Bolter [:davidb] (NeedInfo me for attention)]

For me, this link:
https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/42.0a1?days=3

Currently reports the date range:
'2015-07-18 through 2015-07-21'

Which seems very stale.

Comment 1

[(Away)]

Same here on https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/40.0b/date_range_type/build?days=7

I see: No crashing signatures found for the period 2015-07-14 to 2015-07-21.

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

TCBS matviews are red on https://crash-stats.mozilla.com/crontabber-state/

Peter, is something wrong with our cron jobs?

Comment 3

[Peter Bengtsson [:peterbe]]

The short answer is because cron jobs have been failing for a couple of days because of Postgres+Unicode bugs. I'm trying to find out what the "official" bug(s) for that is.

Comment 4

[Robert Helmer [:rhelmer]]

We need monitoring for /crontabber-state - could we get a JSON representation of that?

I'll clear out the bad unicode records so this self-heals.

Comment 5

[Robert Helmer [:rhelmer]]

breakpad=> delete from raw_crashes where raw_crash::text like '%\u0000%'::text and date_processed >= '2015-07-21';
DELETE 12

Crontabber should self-heal now, might take a few hours.

Comment 6

[Robert Helmer [:rhelmer]]

Reports are now up to date:

https://crash-stats.mozilla.com/crontabber-state/

We are discussing ways to catch this with automated monitoring (we used to do so, but this changed in our recent AWS move.)

Comment 7

[(Away)]

Looks like it's back? https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/42.0a1/date_range_type/build?days=3 shows me "2015-07-27 through 2015-07-30"

Comment 8

[Robert Helmer [:rhelmer]]

breakpad=> delete from raw_crashes where raw_crash::text like '%\u0000%'::text and date_processed >= '2015-07-30';
DELETE 8

We have a fix for this ready, and are working on monitoring for these failures generally.