1194230 - Allow downloading sets of crashes for analysis

Status: RESOLVED WONTFIX

(Socorro :: Webapp, task)

Description

[Jeff Muizelaar [:jrmuizel]]

Some analysis are not doable with the current tools. It would be very valuable to be able to download all of the crashes for a paticular search and be able to do analysis locally.

Comment 1

[(Away)]

For mass downloads I've found the "cliget" addon to be useful in combination with xargs: https://addons.mozilla.org/en-US/firefox/addon/cliget/

I agree that it's not ideal, but perhaps that can unblock you in the near term.

Comment 2

[Robert Kaiser]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #0)
> Some analysis are not doable with the current tools. It would be very
> valuable to be able to download all of the crashes for a paticular search
> and be able to do analysis locally.

Do you want the minidumps or some metadata/annotations on the crashes (or both)?

Comment 3

[Jeff Muizelaar [:jrmuizel]]

(In reply to Robert Kaiser (:[e-mail]) from comment #2)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #0)
> > Some analysis are not doable with the current tools. It would be very
> > valuable to be able to download all of the crashes for a paticular search
> > and be able to do analysis locally.
> 
> Do you want the minidumps or some metadata/annotations on the crashes (or
> both)?

I think the json raw dumps would be best.

Comment 4

[Lonnen :lonnen]

There's been a few requests for something like this. I'll associate the bugs if I get a few minutes to dig them up. I've seen requests for this sort of bulk downloading, as well as sampling features to pull a fraction of a query.

I'm wary of bulk downloads with PII containing info, but with sufficient monitoring and alarm bells I would probably come around. Also some pagination in the API to prevent OOM on the servers

Comment 5

[Lonnen :lonnen]

There is a `socorro_crash` dataset available on telemetry containing the public subset of crash reports. I encourage people to use that set of tools going forward, and where possible to do this kind of analysis on managed machines.