1112024 - Allow me to download all Super Search results

Status: RESOLVED WONTFIX

(Socorro :: Webapp, task)

Description

[Georg Fritzsche [:gfritzsche]]

I don't want to deal with pagination and custom scripts etc., i just want all the results of a Super Search for analysis.

An alternative might be bug 1111612 plus proper random sampling (i think bsmedberg suggested that before).
That way we could probably get plausible results into e.g. http://bsmedberg.github.io/crash-stats-api-magic/analyze-crash.html

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't understand what this bug is about. Seems like you're asking for a mode for the Super Search API that returns all the results rather than a single page because paging is inconvenient.

Is that right? If not, can you elaborate more on what it is you're trying to do?

Comment 2

[Georg Fritzsche [:gfritzsche]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #1)
> Seems like you're asking for a
> mode for the Super Search API that returns all the results rather than a
> single page because paging is inconvenient.

Yes. It's been 3 years since i tried this, so i don't remember the exact use-case anymore.
Generally we were trying to do crash analysis / stats / ... and the API was hard to use.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

While I understand the inconvenience, paging is pretty common in APIs that return large numbers of results. I'm loathe to add a mode that lets someone bypass paging.

I think there are a couple of other ways we could alleviate the original issue. Adding sample code to the API documentation would probably help. Writing language-specific libraries for working with Socorro APIs would probably help. There are several libraries for Bugzilla's API.

If either of those would be helpful, we should write up a new bug for that.

In the meantime, I'm going to WONTFIX this because we really don't want to add a mode that allows someone to download infinite results in a single batch.

Comment 4

[Lonnen :lonnen]

Since this was filed we started publishing the public dataset in telemetry. If you really need bulk data you're probably better off using the a.t.m.o dataset.