639514 - add fixBrokenDumps cron job to work around broken crashes

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

Firefox Linux 4.0b11 and 4.0b12 as well as Fennec Android 4.0b5 had a bug in the breakpad client which caused broken dumps (see bug 637680).

We need to fix these crashes as they come in. This process should be run at least hourly, since the TCBS cron will only look up to two hours prior to it's invocation time for reprocessed crashes.

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: add fix broken crashes cron job

This adds two features to the patch from bug 637680:

* cron shell script wrapper
* use pickle file to mark where processing last completed

Is it a good idea to use date_processed for the latter? I think maybe using the id might make more sense (what we really want to be comparing is the date it was inserted into the table, I think).

Comment 2

[Robert Helmer [:rhelmer]]

Committed revision 2998 + 2999.

Also modifying the queries to be from last_date_processed until (now - 30 minutes) instead of (implicitly) now:

brokenFirefoxLinuxQuery = """
    SELECT uuid,date_processed FROM reports WHERE product = 'Firefox'
      AND (version = '4.0b11' OR version = '4.0b12')
      AND os_name = 'Linux'
      AND date_processed > '%s'
      AND date_processed < (now() - INTERVAL '30 minutes')
"""
brokenFennecQuery = """
    SELECT uuid,date_processed FROM reports WHERE product = 'Fennec'
      AND version = '4.0b5'
      AND date_processed > '%s'
      AND date_processed < (now() - INTERVAL '30 minutes')
"""

These queries are intended to return:

Firefox Linux 4.0b11, 4.0b12
Fennec 4.0b5

jberkus, let me know if you see any problems with this approach (besides what we covered in IRC of course).

Committed revision 3000.

Comment 3

[[:jberkus] Josh Berkus]

Rob,

That looks fine.

Comment 4

[[:jberkus] Josh Berkus]

Rob,

That looks fine.

However, we should make sure (nagios check) that this cron job is executing successfully.  Any way to check?

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> Rob,
> 
> That looks fine.
> 
> However, we should make sure (nagios check) that this cron job is executing
> successfully.  Any way to check?

Yes we should do this... we could perhaps check that all Firefox Linux 4.0b11, 4.0b12 and Fennec 4.0b5 crashes have been reprocessed at least once, and we could check on the hbase side that the dump was replaced (checking that it was actually modified is probably too heavy).

Filed bug 639751 to follow up on this.

Comment 6

[Josh Matthews [:jdm]]

I'm confused by some of the crash reports I'm seeing for Fennec 4.0b5.  If you search for crashes in the past day, a lot of the links to recent crashes that still need reprocessing show entries from a day or even several days earlier. Example: https://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&date=2011-03-15%2008%3A00%3A00&signature=nsDOMSVGZoomEvent%3A%3AnsDOMSVGZoomEvent&version=Fennec%3A4.0b5

Not a single one of the crashes in the above link is valid, but they don't seem to have been reprocessed.