1276471 - Document that GET /rest/bug returns a maximum of $max_search_results bugs by default (default: 10000) even with limit=0

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Documentation, defect)

Description

[Monika]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36

Steps to reproduce:

Type the URL: https://bugzilla.mozilla.org/rest/bug?product=core&include_fields=id


Actual results:

Only top 1000 entries retrieved 


Expected results:

Should have got a list of all the tickets for product Firefox.

Comment 1

[Monika]

Sorry, its 10000 instead of 1000 and 50000 instead of 5000

Comment 2

[Frédéric Buclin]

If limit=0 is not passed as argument, then the buglist is limited to $max_search_results bugs by default. This limit is set to 10000 bugs by default, unless admins edited this parameter.

Comment 3

[Monika]

(In reply to Frédéric Buclin from comment #2)
> If limit=0 is not passed as argument, then the buglist is limited to
> $max_search_results bugs by default. This limit is set to 10000 bugs by
> default, unless admins edited this parameter.

If I use the URL: https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&limit=0; I still see 10000 ids only. Please correct me if I am passing argument wrongly. However, if I pass value as 5 that is, https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&limit=5, I get top 5 only.

Comment 4

[Frédéric Buclin]

Please leave the severity alone. This bug has been triaged. Thanks!

Comment 5

[Monika]

Sure Frederic. The issue is sort of critical for my project. If you can resolve the issues at the earliest possible, it will be of great help.

Comment 6

[Frédéric Buclin]

This is not an issue. As I said in comment 2, this is the correct behavior if no limit is specified. If you want all bugs, you must pass limit=0 to the request.

Comment 7

[Monika]

(In reply to Frédéric Buclin from comment #6)
> This is not an issue. As I said in comment 2, this is the correct behavior
> if no limit is specified. If you want all bugs, you must pass limit=0 to the
> request.

Please refer my comment 3 (reply to your comment 2). I tried passing limit=0 still got only top 10000 ids list.

Comment 8

[Monika]

Please refer my comment 3 (reply to your comment 2). I tried passing limit=0 still got only top 10000 ids list.

Comment 9

[Monika]

Effectively, there is no way that we can get more that 10000 transactions. I think if we pass limit value less than default, it overwrites the default value. However, if we pass value as 0 or greater than 10000, it gives output according to default limit only. Therefore, I think this is an issues not just documentation point.

Comment 10

[Frédéric Buclin]

OK, I did some testing and the Search.pm code indeed has a hard limit of 10000 bugs when allow_unlimited is not set to 1.

@dkl: what is the desired behavior when limit=0? Returning all bugs would be a good way to DoS Bugzilla.

@Monika: as a workaround, you can pass the offset argument to the URL to get all bugs in a loop:

https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=10000
https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=20000
etc...

Comment 11

[Monika]

Frederic, if I try using the offset URLs you have shared, I see following error and no list is retrieved. 
{
   "code" : 50,
   "documentation" : "http://www.bugzilla.org/docs/tip/en/html/api/",
   "error" : true,
   "message" : "The function Bug.search() requires a limit argument, and that argument was not set."
}

Please try clicking on the links that you have shared. It seems offset should be used in combination with limit which also doesn't work. Therefore, the suggested workaround doesn't help. I request to please resolve this with high priority.

Comment 12

[Monika]

Hey Frederic, the link below works:
https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=10000&limit=10000

Using this, I get a list of next 10000 ids. I think there is no way to get more than 10000 entries irrespective of what is passed as value for limit attribute. The offset workaround helps if used with limit parameter.

Comment 13

[David Lawrence [:dkl]]

(In reply to Frédéric Buclin from comment #10)
> OK, I did some testing and the Search.pm code indeed has a hard limit of
> 10000 bugs when allow_unlimited is not set to 1.
> 
> @dkl: what is the desired behavior when limit=0? Returning all bugs would be
> a good way to DoS Bugzilla.

I agree that allowing all bugs to be retrieved from the DB with a single call is a bad idea. Even most of them is bad. We should keep the max_search_results limit as the *real* limit and encourage the use of offset param if more than max_search_results is needed. We need to update the docs to state that is limit is excluded, or equals 0, or is greater than max_search_results, then max_search_results will be used instead.

dkl

Comment 14

[Kohei Yoshino]

Looks like this has already been documented:

https://bmo.readthedocs.io/en/latest/api/core/v1/bug.html#get-bug
https://bugzilla.readthedocs.io/en/latest/api/core/v1/bug.html#get-bug

Comment 15

[Kohei Yoshino]

Hmm, the doc still has to be updated to reflect the limit=0 case properly.

Comment 16

[Kohei Yoshino]

Attachment: GitHub Pull Request

Comment 17

[Kohei Yoshino]

Merged to master.