Closed Bug 1276471 Opened 5 years ago Closed 2 years ago

Document that GET /rest/bug returns a maximum of $max_search_results bugs by default (default: 10000) even with limit=0

Categories

(bugzilla.mozilla.org :: Documentation, defect)

Production
defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: monikag, Assigned: kohei.yoshino)

References

Details

Attachments

(1 file)

46 bytes, text/x-github-pull-request
Details | Review
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36

Steps to reproduce:

Type the URL: https://bugzilla.mozilla.org/rest/bug?product=core&include_fields=id


Actual results:

Only top 1000 entries retrieved 


Expected results:

Should have got a list of all the tickets for product Firefox.
Severity: normal → major
Summary: https://bugzilla.mozilla.org/rest/bug?product=core&include_fields=id returns only first 1000 entries while total tickets for Firefox product are more than 5000 → https://bugzilla.mozilla.org/rest/bug?product=core&include_fields=id returns only first 10000 entries while total tickets for Firefox product are more than 50000
Sorry, its 10000 instead of 1000 and 50000 instead of 5000
If limit=0 is not passed as argument, then the buglist is limited to $max_search_results bugs by default. This limit is set to 10000 bugs by default, unless admins edited this parameter.
Severity: major → normal
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: https://bugzilla.mozilla.org/rest/bug?product=core&include_fields=id returns only first 10000 entries while total tickets for Firefox product are more than 50000 → Document that GET /rest/bug returns a maximum of $max_search_results bugs by default (default: 10000)
(In reply to Frédéric Buclin from comment #2)
> If limit=0 is not passed as argument, then the buglist is limited to
> $max_search_results bugs by default. This limit is set to 10000 bugs by
> default, unless admins edited this parameter.

If I use the URL: https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&limit=0; I still see 10000 ids only. Please correct me if I am passing argument wrongly. However, if I pass value as 5 that is, https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&limit=5, I get top 5 only.
Severity: normal → major
Please leave the severity alone. This bug has been triaged. Thanks!
Severity: major → normal
Sure Frederic. The issue is sort of critical for my project. If you can resolve the issues at the earliest possible, it will be of great help.
This is not an issue. As I said in comment 2, this is the correct behavior if no limit is specified. If you want all bugs, you must pass limit=0 to the request.
(In reply to Frédéric Buclin from comment #6)
> This is not an issue. As I said in comment 2, this is the correct behavior
> if no limit is specified. If you want all bugs, you must pass limit=0 to the
> request.

Please refer my comment 3 (reply to your comment 2). I tried passing limit=0 still got only top 10000 ids list.
Please refer my comment 3 (reply to your comment 2). I tried passing limit=0 still got only top 10000 ids list.
Effectively, there is no way that we can get more that 10000 transactions. I think if we pass limit value less than default, it overwrites the default value. However, if we pass value as 0 or greater than 10000, it gives output according to default limit only. Therefore, I think this is an issues not just documentation point.
OK, I did some testing and the Search.pm code indeed has a hard limit of 10000 bugs when allow_unlimited is not set to 1.

@dkl: what is the desired behavior when limit=0? Returning all bugs would be a good way to DoS Bugzilla.

@Monika: as a workaround, you can pass the offset argument to the URL to get all bugs in a loop:

https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=10000
https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=20000
etc...
Flags: needinfo?(dkl)
Frederic, if I try using the offset URLs you have shared, I see following error and no list is retrieved. 
{
   "code" : 50,
   "documentation" : "http://www.bugzilla.org/docs/tip/en/html/api/",
   "error" : true,
   "message" : "The function Bug.search() requires a limit argument, and that argument was not set."
}

Please try clicking on the links that you have shared. It seems offset should be used in combination with limit which also doesn't work. Therefore, the suggested workaround doesn't help. I request to please resolve this with high priority.
Hey Frederic, the link below works:
https://bugzilla.mozilla.org/rest/bug?product=firefox&include_fields=id&offset=10000&limit=10000

Using this, I get a list of next 10000 ids. I think there is no way to get more than 10000 entries irrespective of what is passed as value for limit attribute. The offset workaround helps if used with limit parameter.
(In reply to Frédéric Buclin from comment #10)
> OK, I did some testing and the Search.pm code indeed has a hard limit of
> 10000 bugs when allow_unlimited is not set to 1.
> 
> @dkl: what is the desired behavior when limit=0? Returning all bugs would be
> a good way to DoS Bugzilla.

I agree that allowing all bugs to be retrieved from the DB with a single call is a bad idea. Even most of them is bad. We should keep the max_search_results limit as the *real* limit and encourage the use of offset param if more than max_search_results is needed. We need to update the docs to state that is limit is excluded, or equals 0, or is greater than max_search_results, then max_search_results will be used instead.

dkl
Flags: needinfo?(dkl)
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED

Hmm, the doc still has to be updated to reflect the limit=0 case properly.

Assignee: documentation → kohei.yoshino
Status: RESOLVED → REOPENED
Product: Bugzilla → bugzilla.mozilla.org
QA Contact: default-qa
Resolution: FIXED → ---
Summary: Document that GET /rest/bug returns a maximum of $max_search_results bugs by default (default: 10000) → Document that GET /rest/bug returns a maximum of $max_search_results bugs by default (default: 10000) even with limit=0
Version: unspecified → Production
See Also: → 1323709
Attached file GitHub Pull Request
Status: REOPENED → ASSIGNED

Merged to master.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.