909469 - FakeData incorrectly sets a date for processed

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Brandon Savage [:brandon]]

I had a crash report that was dated August 21, 2013 but was "processed" on August 14, 2013. This created a problem where the crashes/ API didn't load the crash. We should make the fakedata always have the processed date after the crash date (since it would be in real life).

Comment 1

[Robert Kaiser]

which is the other date than "processed"? If it's the "crash date" we send with the crash report, things ought to work when it's in the future, as it's taken from the local computer clock, and there's definitely users out there how have skewed clocks.

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

+1 to what KaiRo says, we have no control over the "client_crash_date", so having weird dates should be expected and should not break Socorro. Afaik, we set "date_processed", "completeddatetime" and "startedDateTime" in the processor, and:

date_processed < startedDateTime < completeddatetime

Comment 3

[Brandon Savage [:brandon]]

Okay, so I explained the problem poorly. Here's what I meant.

The UUID for the crash has a date (last six characters of the OOID) of 130821. However, our internal processing date is 2013-08-14 - supposedly more than a week before the crash occurred.

This only affects fakedata as far as I can tell. But the crash/ API expects that the OOID date is correct, and uses that with the utc_day_is() function to determine if it should pull the crash or not.

It's possible the crashes/ API is broken or has wrong assumptions. I think it's more likely that fakedata just has a bug in it.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Brandon Savage [:brandon] from comment #3)
> Okay, so I explained the problem poorly. Here's what I meant.
> 
> The UUID for the crash has a date (last six characters of the OOID) of
> 130821. However, our internal processing date is 2013-08-14 - supposedly
> more than a week before the crash occurred.
> 
> This only affects fakedata as far as I can tell. But the crash/ API expects
> that the OOID date is correct, and uses that with the utc_day_is() function
> to determine if it should pull the crash or not.
> 
> It's possible the crashes/ API is broken or has wrong assumptions. I think
> it's more likely that fakedata just has a bug in it.

Hrm so looking at the code it looks like both date_processed and the crashid generation code uses the same input (a datetime object named "timestamp").

I'll build a local fakedata db and take a peek.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Brandon Savage [:brandon] from comment #0)
> I had a crash report that was dated August 21, 2013 but was "processed" on
> August 14, 2013. This created a problem where the crashes/ API didn't load
> the crash. We should make the fakedata always have the processed date after
> the crash date (since it would be in real life).

So to clarify, are you talking about the "date_processed" column in the reports table? It's actually a misnomer, that's the collection time (the collectors set it)

Comment 6

[Robert Kaiser]

(In reply to Brandon Savage [:brandon] from comment #3)
> The UUID for the crash has a date (last six characters of the OOID) of
> 130821. However, our internal processing date is 2013-08-14 - supposedly
> more than a week before the crash occurred.

OK, that's surely an error. "date_processed" is the collection time, as rhelmer says, and the date part of the crash ID (can we please get rid of the "OOID" moniker?) is taken from the collection date as well. Those never should differ. Sounds definitely like a bug in fakedata.

Comment 7

[Robert Helmer [:rhelmer]]

Yeah I think this is a bug in fakedata.py, I suspect it's using the very last timestamp (derived from the time the script was started) for the crash id (uuid column) instead of the same that is used for the date_processed on that row:

fakedata=# select uuid,date_processed from reports limit 10;
                 uuid                 |        date_processed         
--------------------------------------+-------------------------------
 60597bdc-5dbe-4409-6b38-4309c0130828 | 2013-08-13 06:16:04.630337+00
 d1ca4dc4-edfd-e416-3eff-8b3fc0130828 | 2013-08-13 06:16:04.630337+00
 5e0730b3-cc17-0c31-c7ee-c61bb0130828 | 2013-08-13 06:16:04.630337+00
 b3de08f9-ec98-3704-4692-ba0350130828 | 2013-08-13 06:16:40.630337+00
 7bd9e8a1-ff29-7d0e-4f2e-84fcb0130828 | 2013-08-13 06:16:40.630337+00
 400db00d-d388-1a50-5805-6ed090130828 | 2013-08-13 06:16:40.630337+00
 21fdac3a-f325-ffbe-9065-6c8fb0130828 | 2013-08-13 06:17:16.630337+00
 ec08693c-7401-f5ce-2425-d75a40130828 | 2013-08-13 06:17:16.630337+00
 78573797-4a80-7546-4627-314170130828 | 2013-08-13 06:17:16.630337+00
 7deaab64-16d1-b5ac-2e1c-5d3a50130828 | 2013-08-13 06:17:52.630337+00
(10 rows)

Comment 8

[Robert Helmer [:rhelmer]]

r? https://github.com/mozilla/socorro/pull/1448

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/9cfe503c2cc1d9fd54c37dabf6fba947d9b0eada
fixes bug 909469 - date_processed and crash id (uuid) timestamps should match

https://github.com/mozilla/socorro/commit/afe16d2919f1533b0198a3a08670aee3a3069dea
Merge pull request #1448 from rhelmer/bug909469-fakedata-uuid-date_processed-discrepency

fixes bug 909469 - date_processed and crash id (uuid) timestamps should ...