1311697 - b8085b63-cd80-4876-9143-72a4e2161017 fails to be processed

Status: VERIFIED FIXED

(Socorro :: Webapp, task)

Comment 1

[Peter Bengtsson [:peterbe]]

It's been put into the priority queue for over 20 minutes at the time of writing. 
I also, put it into the reprocessing queue too to see if the first priority queue drop had failed silently. No dice. 

It exists as a rawcrash: https://crash-stats.mozilla.com/api/RawCrash/?crash_id=b8085b63-cd80-4876-9143-72a4e2161017
But isn't coming up as a processed crash: https://crash-stats.mozilla.com/api/ProcessedCrash/?crash_id=b8085b63-cd80-4876-9143-72a4e2161017 :(

Comment 2

[Peter Bengtsson [:peterbe]]

JP, can you use the loggly account (mine doesn't work) to search for b8085b63-cd80-4876-9143-72a4e2161017
Especially on the processor nodes. 
Also, can you check that the rabbitmq queue sizes are sane and flowing?

Comment 3

[Peter Bengtsson [:peterbe]]

Ted, 
This crash fails repeatedly to be processed. I currently have no good way to see logging from the processors to see if there are exceptions which are poorly handled. Can you try processing it locally to see if there's something crazy about it?

Comment 4

[Peter Bengtsson [:peterbe]]

What the heck?!
I found a processor that had run the job. It says it was successful
https://gist.github.com/peterbe/c8d89f1532f4e018d28299589d6960a4

I searched the S3 bucket for this UUID and it definitely isn't there. 
It's as if the processor succeeded twice (once when it was originally priority queue submitted and once when I reprocessed it). But both times it can't upload it to S3.

Comment 5

[Peter Bengtsson [:peterbe]]

I copied the raw crash and upload_file_minidump_browser crash to the equivalent in the stage S3 bucket. When I run it locally I get this error: https://gist.github.com/peterbe/9175249a5e88960d3c9597232db38918

Basically, the new `return SocorroDotDict(copy.deepcopy(dict(crash)))` line is causing this error. 

Let's dig into it and figure out what needs to be done.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I suspect this line isn't sufficient:

   return SocorroDotDict(copy.deepcopy(dict(crash)))

I think if doing just dict(crash) was sufficient, then we wouldn't need to do a deepcopy.

Comment 7

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3546

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/c049e63f6bdd0e29acadbe326a68d9890fcd67f0
fixes bug 1311697 - deep copy SocorroDotDict better (#3546)

* fixes bug 1311697 - deep copy SocorroDotDict better

* Fix dictify

This reworks dictify to more obviously traverse an object graph
converting mapping-ish things (including DotDict and DotDict) to dicts
and sequence-ish things to lists.

The guarantee after the conversion is that the resulting thing is
deep-copyable.

This also adds a bunch of tests for the various possible cases.

* Fix a gross error

I am deeply sorry such an egregious error made it into the previous
commit. I hope future Socorro developers can find it in their hearts to
forgive me.

Comment 9

[Peter Bengtsson [:peterbe]]

Code's gone to master but not to prod so I can't verify it yet.

Comment 10

[Peter Bengtsson [:peterbe]]

Now it works! https://crash-stats.mozilla.com/report/index/b8085b63-cd80-4876-9143-72a4e2161017 Yay
Sorry for the delay.

Comment 11

[alex_mayorga]

¡Hola!

Yup! I can see it now.

Unsure what to do with it as it is one of those dreaded [@ IPCError-browser | ShutDownKill ] that seem to be all the rage this days...

¡Gracias!
Alex