Ability to get all raw dumps from the API

RESOLVED FIXED in 105

Status

Socorro
Webapp
--
enhancement
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: Benjamin Smedberg, Assigned: peterbe)

Tracking

unspecified
All
Windows 7

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Currently to download a raw dump, you follow a link such as https://crash-stats.mozilla.com/rawdumps/c0416855-49cc-4c23-ade6-a78182140630.dmp

But this only works if you're logged in and isn't an API endpoint.

There is the RawCrash API that looks like it would also provide raw dump access, but it doesn't appear to work. Links such as https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-ade6-a78182140630&format=raw return an ISE.

In addition, there is no way to get access to the raw dumps other than the main one: so for plugin hangs where there are 1-3 additional minidumps, there is no way to download them.

I propose an additional parameter to the RawCrash API dumpname=string which can be used to get access to the additional minidumps.

Can somebody look up what the current ISE is and confirm that it's supposed to be working?
Flags: needinfo?(chris.lonnen)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> Currently to download a raw dump, you follow a link such as
> https://crash-stats.mozilla.com/rawdumps/c0416855-49cc-4c23-ade6-
> a78182140630.dmp
> 
> But this only works if you're logged in and isn't an API endpoint.
> 
> There is the RawCrash API that looks like it would also provide raw dump
> access, but it doesn't appear to work. Links such as
> https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-
> ade6-a78182140630&format=raw return an ISE.

Here's the error, the API "cleaner" is trying to treat the response as JSON and redact keys - so it's just assuming the type is 'meta'. We should probably just make a dedicated endpoint for this:

https://errormill.mozilla.org/webtools/socorro-stage/group/170446/

AttributeError: 'str' object has no attribute 'keys'

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "waffle/decorators.py", line 36, in _wrapped_view
    return view(request, *args, **kwargs)
  File "ratelimit/decorators.py", line 95, in _wrapped
    return fn(request, *args, **kw)
  File "crashstats/crashstats/utils.py", line 296, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/crashstats/utils.py", line 53, in wrapper
    response = f(request, *args, **kw)
  File "crashstats/api/views.py", line 284, in model_wrapper
    cleaner.start(result)
  File "crashstats/api/cleaner.py", line 42, in start
    self._scrub(data, self.whitelist)
  File "crashstats/api/cleaner.py", line 49, in _scrub
    self._scrub_list(result, api_whitelist)
  File "crashstats/api/cleaner.py", line 92, in _scrub_list
    self._scrub_item(data, whitelist)
  File "crashstats/api/cleaner.py", line 73, in _scrub_item
    for key in data.keys():
(Reporter)

Comment 2

4 years ago
ok, so completely separate the API like this:

CrashMetadata(uuid) - returns the metadata key/value pairs
CrashDump(uuid, name) - returns the minidump file, pass "" for the default  or "browser" will get you "upload_file_minidump_browser"

rhelmer does that sound ok?
Flags: needinfo?(chris.lonnen) → needinfo?(rhelmer)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #2)
> ok, so completely separate the API like this:
> 
> CrashMetadata(uuid) - returns the metadata key/value pairs
> CrashDump(uuid, name) - returns the minidump file, pass "" for the default 
> or "browser" will get you "upload_file_minidump_browser"
> 
> rhelmer does that sound ok?

That looks good to me - I don't think there's a need for "Raw" in the name since the processed version is called "ProcessedCrash", I think that will be clear enough for people looking at the API docs.
Flags: needinfo?(rhelmer)
(Assignee)

Updated

3 years ago
Assignee: nobody → peterbe
Component: Middleware → Webapp
(Assignee)

Comment 4

3 years ago
The ISE isn't because of the scrubber but because it incorrectly tries to return the binary content as a JSON string. 

I'm going to tackle that primary problem first and foremost and then we'll extend the functionality to get other dumps other than the "main one". I wasn't aware there were more than one by name.
(Assignee)

Comment 5

3 years ago
PR https://github.com/mozilla/socorro/pull/2217

This just adds the ability to down the raw dump by /api/RawCrash/?crash_id=:crash_id&format=raw

It requires the `view_pii` permission. 

Things that remain to do:

1. Change the link in the /report/index/:crash_id page to use the API link instead. Maybe?
2. Benjamin's feature request to be able to get other dumps other than the "main one" .

Comment 6

3 years ago
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b6da8f6ff46ffe079ed9fabcebc515d3c61279f6
bug 1037512 - ability to get raw dumps from the API, r=@AdrianGaudebert

https://github.com/mozilla/socorro/commit/92296748505704c1d6c68d634181165032802e7e
Merge pull request #2217 from peterbe/bug-1037512-ability-to-get-raw-dumps-from-the-api

bug 1037512 - ability to get raw dumps from the API

Updated

3 years ago
Target Milestone: --- → 95
(Assignee)

Comment 7

3 years ago
/me clearly forgot to put the "fixes" in the commit message.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

3 years ago
Peter, since mostly the point of this bug was to be able to download all of the dumps, I'm going to reopen this. It's especially important now that I've switched nightly/aurora to use crash-stats instead of the temporary skunky-server.
(Reporter)

Updated

3 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 9

3 years ago
We can now query for raw dumps other than the default "upload_file_minidump"
https://github.com/peterbe/socorro/commit/002f1b975a50a6bd6b7832a04e4d85023bb400fa
(Assignee)

Comment 10

3 years ago
New PR just for the new API parameter 
https://github.com/mozilla/socorro/pull/2366

Comment 11

3 years ago
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/fc3fd4c685503fa35bc7a2b0283b678c24ac360c
bug 1037512 - adding 'name' param to /api/RawCrash/

https://github.com/mozilla/socorro/commit/177a10c7c3f370923ee7b0a93840db77106e2ebb
Merge pull request #2366 from peterbe/bug-1037512-adding-name-param-to-apirawcrash

bug 1037512 - adding 'name' param to /api/RawCrash/
(Assignee)

Comment 12

3 years ago
I'm going to close this because it's annoying that it's one bug for multiple different (albeit interlinked) features in one bug. 
The last github commit should now make it totally possible to add do:
https://crash-stats.mozilla.com/api/RawCrash/?crash_id=c0416855-49cc-4c23-ade6-a78182140630&format=raw&name=upload_file_minidump_flash1 as soon as this bug gets deployed.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Blocks: 1067577

Updated

3 years ago
Target Milestone: 95 → 103

Updated

3 years ago
Target Milestone: 103 → 104

Updated

3 years ago
Target Milestone: 104 → 105
You need to log in before you can comment on or make changes to this bug.