479256 - crash-stats shows all crash times as 00:00

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Wayne Mery (:wsmwk)]

example my thunderbird crash from 10:22am EST has Time	2009-02-18 00:00:00
bp-3d05cb4a-6435-4b30-b547-373682090218

this is a regression. bug 466242?

there is some thinking that displayed date/time should be actual crash time, not processed time.  I would favor that, if nothing else as a guard against crashes getting bad info when processing is delayed/isn't working.

Comment 1

[Austin King [:ozten]]

report/list and report/index will display
client_crash_date
instead of 
date_processed

The danger her is when processing gets out of sync ( rerunning the processor, processing a deferred job, or long wait times in the job queue ) the query results won't match what is displayed.

Comment 2

[Austin King [:ozten]]

Option 2
report/list
Change label of table heading 
from Date
to Date Processed
and truncate the displayed value to only the year-month-day

report/index
Also truncate the display date_processed here
Change the label from Time to Day Processed

Add field client_crash_date
Labeled "Crash Time"

Optionally we should update '/' and '/query' forms to say
"Occurs before"
to
"Processed before"

Comment 3

[K Lars Lohn [:lars] [:klohn]]

Provenance of the datetime fields in Socorro 'reports' table:

'date_processed' – a date generated from the last 6 digits of a crash' 'uuid'.  That value is assigned by the collector using the local time of the webhead.  This value is the controller of the database partitioning scheme.  

'client_crash_date' – one of three values: the 'CrashTime' or 'timestamp' as reported by the client; or if those values were missing, the datetime reported by the collector that received the crash.  Because there was no mechanism to determine which date source was actually used, this column seems to be of little use for queries.

'started_datetime' – the datetime that the processor actually started working on the the crash as reported by the machine running the processor.

'completed_datetime' – the datetime that the processor actually committed the the completed crash to the database as reported by the machine running the processor.

Commentary:

'date_processed' shed the time portion of the datetime, because it was unneeded for partitioning and was not encoded in the 'uuid'.  This has resulted, from the perspective of the GUI, in making it appear that all crashes happened at the beginning of the day.  Relative ordering of crashes in the day is no longer possible using the 'date_processed' column.

Prior to database partitioning, this value was set by the processor at the moment that the crash was queued for processing. It included the time.  Because of the need for close coupling of the date embedded 'uuid' assigned by the collector and the 'date_processed' column  used for partitioning, it was no longer feasible to allow two separate machines to assign these respective dates.  They had to be guaranteed to match exactly.

It is possible for 'date_processed' to regain the time component.  It would require that an additional field be added to the json file to record the submission datetime as reported by the collector.  Then in the processor, the value for 'date_processed' with its time component would be retrieved from the json file rather than being gleaned from the uuid.  This both guarantees the required match as well as restoring the relative ordering of crashes within a day.

Comment 4

[Austin King [:ozten]]

Socorro UI is updated now.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

the Socorro UI is updated to do what?  I've not pushed the new Socorro Server code that will recapture the time component of the 'date_processed' field.

Comment 6

[Austin King [:ozten]]

Reassigning to Lars.

Comment 7

[K Lars Lohn [:lars] [:klohn]]

this problem is now resolved.  With the push of the latest collector/monitor code to production last night, all new crashes will have a time component on the crash date field.  I've confirmed that it shows up properly in the UI.