1037512 - Ability to get all raw dumps from the API

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Benjamin Smedberg]

Currently to download a raw dump, you follow a link such as https://crash-stats.mozilla.com/rawdumps/c0416855-49cc-4c23-ade6-a78182140630.dmp

But this only works if you're logged in and isn't an API endpoint.

There is the RawCrash API that looks like it would also provide raw dump access, but it doesn't appear to work. Links such as https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-ade6-a78182140630&format=raw return an ISE.

In addition, there is no way to get access to the raw dumps other than the main one: so for plugin hangs where there are 1-3 additional minidumps, there is no way to download them.

I propose an additional parameter to the RawCrash API dumpname=string which can be used to get access to the additional minidumps.

Can somebody look up what the current ISE is and confirm that it's supposed to be working?

Comment 1

[Robert Helmer [:rhelmer]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> Currently to download a raw dump, you follow a link such as
> https://crash-stats.mozilla.com/rawdumps/c0416855-49cc-4c23-ade6-
> a78182140630.dmp
> 
> But this only works if you're logged in and isn't an API endpoint.
> 
> There is the RawCrash API that looks like it would also provide raw dump
> access, but it doesn't appear to work. Links such as
> https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-
> ade6-a78182140630&format=raw return an ISE.

Here's the error, the API "cleaner" is trying to treat the response as JSON and redact keys - so it's just assuming the type is 'meta'. We should probably just make a dedicated endpoint for this:

https://errormill.mozilla.org/webtools/socorro-stage/group/170446/

AttributeError: 'str' object has no attribute 'keys'

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "waffle/decorators.py", line 36, in _wrapped_view
    return view(request, *args, **kwargs)
  File "ratelimit/decorators.py", line 95, in _wrapped
    return fn(request, *args, **kw)
  File "crashstats/crashstats/utils.py", line 296, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/crashstats/utils.py", line 53, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/api/views.py", line 284, in model_wrapper
    cleaner.start(result)
  File "crashstats/api/cleaner.py", line 42, in start
    self._scrub(data, self.whitelist)
  File "crashstats/api/cleaner.py", line 49, in _scrub
    self._scrub_list(result, api_whitelist)
  File "crashstats/api/cleaner.py", line 92, in _scrub_list
    self._scrub_item(data, whitelist)
  File "crashstats/api/cleaner.py", line 73, in _scrub_item
    for key in data.keys():

Comment 2

[Benjamin Smedberg]

ok, so completely separate the API like this:

CrashMetadata(uuid) - returns the metadata key/value pairs
CrashDump(uuid, name) - returns the minidump file, pass "" for the default  or "browser" will get you "upload_file_minidump_browser"

rhelmer does that sound ok?

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #2)
> ok, so completely separate the API like this:
> 
> CrashMetadata(uuid) - returns the metadata key/value pairs
> CrashDump(uuid, name) - returns the minidump file, pass "" for the default 
> or "browser" will get you "upload_file_minidump_browser"
> 
> rhelmer does that sound ok?

That looks good to me - I don't think there's a need for "Raw" in the name since the processed version is called "ProcessedCrash", I think that will be clear enough for people looking at the API docs.

Comment 4

[Peter Bengtsson [:peterbe]]

The ISE isn't because of the scrubber but because it incorrectly tries to return the binary content as a JSON string. 

I'm going to tackle that primary problem first and foremost and then we'll extend the functionality to get other dumps other than the "main one". I wasn't aware there were more than one by name.

Comment 5

[Peter Bengtsson [:peterbe]]

PR https://github.com/mozilla/socorro/pull/2217

This just adds the ability to down the raw dump by /api/RawCrash/?crash_id=:crash_id&format=raw

It requires the `view_pii` permission. 

Things that remain to do:

1. Change the link in the /report/index/:crash_id page to use the API link instead. Maybe?
2. Benjamin's feature request to be able to get other dumps other than the "main one" .

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b6da8f6ff46ffe079ed9fabcebc515d3c61279f6
bug 1037512 - ability to get raw dumps from the API, r=@AdrianGaudebert

https://github.com/mozilla/socorro/commit/92296748505704c1d6c68d634181165032802e7e
Merge pull request #2217 from peterbe/bug-1037512-ability-to-get-raw-dumps-from-the-api

bug 1037512 - ability to get raw dumps from the API

Comment 7

[Peter Bengtsson [:peterbe]]

/me clearly forgot to put the "fixes" in the commit message.

Comment 8

[Benjamin Smedberg]

Peter, since mostly the point of this bug was to be able to download all of the dumps, I'm going to reopen this. It's especially important now that I've switched nightly/aurora to use crash-stats instead of the temporary skunky-server.

Comment 9

[Peter Bengtsson [:peterbe]]

We can now query for raw dumps other than the default "upload_file_minidump"
https://github.com/peterbe/socorro/commit/002f1b975a50a6bd6b7832a04e4d85023bb400fa

Comment 10

[Peter Bengtsson [:peterbe]]

New PR just for the new API parameter 
https://github.com/mozilla/socorro/pull/2366

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/fc3fd4c685503fa35bc7a2b0283b678c24ac360c
bug 1037512 - adding 'name' param to /api/RawCrash/

https://github.com/mozilla/socorro/commit/177a10c7c3f370923ee7b0a93840db77106e2ebb
Merge pull request #2366 from peterbe/bug-1037512-adding-name-param-to-apirawcrash

bug 1037512 - adding 'name' param to /api/RawCrash/

Comment 12

[Peter Bengtsson [:peterbe]]

I'm going to close this because it's annoying that it's one bug for multiple different (albeit interlinked) features in one bug. 
The last github commit should now make it totally possible to add do:
https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-ade6-a78182140630&format=raw&name=upload_file_minidump_flash1 as soon as this bug gets deployed.