1249412 - Include UUID in graphics report csv

Status: RESOLVED FIXED

(Webtools Graveyard :: Air Mozilla, defect)

Description

[Peter Bengtsson [:peterbe]]

We used to yield a "uuid_url" [0] with a hardcoded URL. 
Instead, let's just return the crash_id in that space. 


[0] https://github.com/mozilla/socorro/blob/f1e289789510cbdc2bf184cca121b1708efac200/socorro/external/postgresql/graphics_report.py#L21

Comment 1

[Peter Bengtsson [:peterbe]]

Vlad, by the way, we try to call it "Crash ID" these days. (having said that, I can never decide if it's "crashID", "crash_id", "crash ID" or "crashid" :)

Comment 2

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3206

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Can we also include 'release_channel' and 'branch'?  I can create a PR for it myself, but figure you can probably get it in this one faster :)

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/874b6d90c1221d4f3d90ec1871fbf633d104262f
fixes bug 1249412 - Include UUID in graphics report csv

https://github.com/mozilla/socorro/commit/a51a7766007078fcbb7946190a4d10f252cd278d
Merge pull request #3206 from peterbe/bug-1249412-include-uuid-in-graphics-report-csv

fixes bug 1249412 - Include UUID in graphics report csv

Comment 5

[Peter Bengtsson [:peterbe]]

The graphics report thing was a desperate thing. Ideally everyone should be using the SuperSearch API. It's maintained and it makes us have fewer unique snowflake solutions we have to maintain. 
The reason we put it in, was to make it really easy for the graphics team to get exactly the report they were used to without having to run it over on the ad-hoc crash-analysis box (it's a long-running goal to decommission that server). 

So, together with Adrian (the SuperSearch guru) can we figure out if we can use SuperSearch instead? If not, can we fix SuperSearch?
(it might mean inconvenience for you in that you might have to mess with API tokens and paginate the results etc.)

Comment 6

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

This is work that's related to the graphics team, but with broader scope.  I've been talking with Adrian; unfortunately, the SuperSearch API is not sufficient for processing crashes.  I need to download/process all crash reports in a given date range.  Using the graphics-report CSV API, I can fetch a day's worth of crash reports in something like 20-30 seconds.  Using the SuperSearch API, it takes about 4-5s to fetch a single "page" of 100 reports.  Fetching a day's worth of reports (~165k) would take around 2.5 hours.

I fully agree that we should be using a single API and should be improving the SuperSearch API; however, that's not something that I can do myself, and I don't know when Adrian or you can fit it in.  Fixing up the graphics report CSV generation unblocks the analysis work that I'm doing, and it can get switched over to SuperSearch results when that's fast.  (Current code can handle both inputs, so it'll be easy to test/switch.)

Comment 7

[Peter Bengtsson [:peterbe]]

Adrian, 
Could we, in an ad-hoc django view, loop over the pagination server-side and buffer a big monster report? Any other ideas for how to make this doable in SuperSearch? 

Vladimir,
Thank you for your open feedback. Mucho appreciated. What we're trying to do with Socorro, in 2016, is to greatly simplify it. At the moment we store crashes in ES AND in PostgreSQL. It's our goal to remove all storage of crashes in PostrgreSQL (we're going to keep it around as a general relational store for other non-crashes stuff). That's why I'm eager to find a working solution. 

I'd be happy to hack the existing SQL we have now. I can easily add release_channel. But for branch, I don't even know what column that would be. In the old code, we never returned it. https://github.com/mozilla/socorro/blob/master/socorro/external/postgresql/graphics_report.py#L28

Comment 8

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

release_channel is the more important thing; I figured branch would be handy if we had it, but I don't have good use for it.  I *think* release_channel is the last thing I need now...

Comment 9

[Peter Bengtsson [:peterbe]]

/me adding release_channel

Comment 10

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3214

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/ad4ed7e22755ae7590d665a4e3bf428356447874
fixes bug 1249412 - include release_channel in graphics report csv

https://github.com/mozilla/socorro/commit/bad1a2c60dd42674319468089cc52111bdd900c1
Merge pull request #3214 from peterbe/bug-1249412-include-release_channel-in-graphics-report-csv

fixes bug 1249412 - include release_channel in graphics report csv