1250524 - [Super Search] Offer a way to download a lot of data at once

Status: RESOLVED WONTFIX

(Socorro :: Webapp, task, P4)

Description

[[DEACTIVATED] Adrian Gaudebert]

We currently have a way of paginating over a dataset using `_results_number` and `_results_offset`, but that's not very convenient, and quite slow. Elasticsearch has some features to make it fast and easy to download a lot of data. We might want to integrate that into Super Search, or figure out another way to do that.

Comment 1

[Peter Bengtsson [:peterbe]]

Kairo actually has me quite convinced with his point made in the meeting. 

Before we jump in with a technical solution; we ought to ask, why don't you let us do the aggregation (using SuperSearch). 

After all, people will most likely download a bunch of crashes just to do their own aggregates in their own little python/js scripts. 

Adding the export functionality isn't free. It'll take time. And unlike some providers where the data is held on behalf of someone else, we don't have an obligation to allow people to download all their data.

In other words; I suggest we mark this bug as a low low priority. Or even INVALID. What do you think?

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

I still think that would be a convenient thing to have, for example if we end up doing a spectateur-like feature. I expect some reports won't be doable without iterating through a lot of data. 

However, I agree that this is low priority and we should educate our users instead of trying to solve this.

Comment 3

[Peter Bengtsson [:peterbe]]

Resolve, Invalid??

If someone really wants to download everything they can with pagination. 
If someone really needs all crashes, should they not use re-dash or something and do more complex/advanced aggregations there.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm going to mark this as WONTFIX.

If a real compelling use case comes up, we can re-think this and re-open or create a new bug or something like that. For now we don't have a good reason to keep thinking about this.