1369493 - Bogus total_virtual_memory ("44651^4 _4") in a crash report

Status: RESOLVED WONTFIX

(Socorro :: General, task)

Description

[Marco Castelluccio [:marco]]

The crash correlations job failed because there was one report where int(total_virtual_memory) would throw an exception. The value of total_virtual_memory was "44651^4 _4".

I've never encountered this over the past months, so it must be really rare (or it's a recent regression).

The TotalVirtualMemory annotation is set at https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/nsExceptionHandler.cpp#801.

Comment 1

[Peter Bengtsson [:peterbe]]

We chatted about this in IRC yesterday but unable to update this bug due to bugzilla downtime. 

The TotalVirtualMemory comes to us (via Antenna) as a string (in a multipart/form-data). Antenna turns it into a JSON (still has a string), with the key "TotalVirtualMemory", and saves it as the raw crash in S3. 
Our cron job that uses the JSON Schema to upload to the Telemetry ingestion S3 bucket, but just copies the value from the raw crash JSON as is. No type conversion anywhere. However, before sending it, it renames the key from "TotalVirtualMemory" to "total_virtual_memory". Again, without touching the value, which is still a string.

So if the value is something that not easily turned into a real integer, it's "Firefox's fault". Meaning, when the crash reporter asks Firefox for the number, that's where the fault lies. 

There is no flaw or bug in how Socorro deals with this value. 

Our options:

1) "Deal with it" in metrics processing. I.e. in Redash queries or ipython notebooks, don't assume it can always easily be converted to an integer. 

2) Post process it in Socorro and write some Socorro processor code that attempts to convert it to a real integer and then nullify those that can't be converted. Then we'd start feeding the Telemetry ingestion with a data point from the *processed crash* instead. 

3) Hunt down, where and how in Firefox, this number could be something that isn't a string that is easily turned into an integer.

Comment 2

[Peter Bengtsson [:peterbe]]

Have you had any more ideas about how to progress this forward?
I think it's the nature of the whole stack. Generally we don't try to clean up values. With the processed crash we don't really take a raw crash value and re-write it under processed crash. It's usually more thing convenient things juggling data around into different forms. 

If it becomes a pressing issue that you can't query against it as a number (e.g. SuperSearch for total_virtual_memory > 1000 && total_virtual_memory < 2000) then we should add a new processing rule that makes this number specifically. 

But generally, I think we want to solve these kinds of more complex querying requirements in places like Redash, Zeppelin, iPython Notebook.

Comment 3

[Marco Castelluccio [:marco]]

No new ideas. Dealing with it in the analysis code sounds fine for now, we can re-evaluate if it becomes a problem in the future.