412375 - Older crashreports no longer visible ?

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Carsten Book [:Tomcat]]

<Tomcat> morgamic: do we still have problems with breakpad reports http://crash-stats.mozilla.com/report/index/86805b2e-bfdb-11dc-bd7e-001a4bd43ef6?date=2008-01-11-00 is missing since 4 days :)
<ss> Older reports don't seem to exist, as well.
<ss> Like, http://crash-stats.mozilla.com/report/index/7954f507-b7b7-11dc-9f4e-001a4bd43ef6
<ss> Which is from sometime in December.

also http://crash-stats.mozilla.com/report/index/659118a3-bf65-11dc-9855-001a4bd43ed6?date=2008-01-10-10 is missing and get the 404 Message.

Comment 1

[Michael Morgan [:morgamic]]

We archived reports older than 60 days, it shouldn't have affected reports in January.  Also note that the deletion occurred last Thursday, the evening of the 10th.

Aravind, can you check individual partitions for existence of these ids?

Comment 2

[Michael Morgan [:morgamic]]

The query we used to select the max report id for reports older than 2 months returned an id that was too high because "date" is not reliable.  For example, we have reports august 2008 and august 2005, which is completely messed up.

So, to fix things:
1) recrash and you should have the new things picked up
2) we could restore from the old db from tape if we really want to restore

The problem with 2) is we'd either have to merge the db's or set up an alternate installation of socorro.

Of course the problem with 1) is that we lose data from december and the first week of january.

To prevent this from happening in the future, we plan on adding a processed_time column to the reports table so we have a date field to rely on for future garbage collection that won't screw us.

What do you guys want to do?  Main thing is that we haven't lost the data, just need to know what is important for the release. 

Comment 3

[Samuel Sidler (old account; do not CC)]

I talked this over with morgamic a little bit ago.

Let's do 2) with a separate instance of Socorro and advertise it on the site somewhere (maybe a banner on the top or something). I don't want to worry about merging data since that seems potentially dangerous and overly time consuming.

The reason we want that data available is because some crashes were reported with steps that either don't reproduce or are hard to reproduce. We can still analyze the stacks, but not without having them. ;)

Sound good?

Comment 4

[Aravind Gottipati [:aravind]]

I have bad news.  We moved the databases to a new partition on the netapp when we ran out of space on the db server.  There are only three days worth of backups on the server itself.  I apparently didn't move the db keys and so the backups weren't getting on to tape.  I fixed that now, but I probably will not be able to restore old backups for this.  Sorry guys.

Comment 5

[Michael Morgan [:morgamic]]

So, we had a couple of compound issues.  We're fixing the dates today, and will work to set up tests to verify that backups are working.  This won't happen again.

In addition, we will run cleanup on staging first next time, which might be a challenge because of the size of the db, but will try to make it work.

This time through we ran the cleanup on prod because it was not reasonable to download a 170g DB to my dev area -- last week we were in a rush to get the db operational and dumping/downloading/importing a 170g DB onto a VM would have meant at least 4 days of downtime for the server (instead of 1 day).