1280494 - Ensure all API endpoints protect/validate against unbounded queries

Status: NEW

(Tree Management :: Treeherder: API, defect, P3)

Description

[Ed Morley [:emorley]]

In bug 1280039, omitting the parameters to a GET request made to /api/jobdetails/ causes significant load on the DB.

In addition to fixing that bug, we should:

1) Ensure that endpoints that are not expected to be used with zero parameters return an HTTP 400 if they are omitted. (For example /api/bugscache/ correctly does so if the `search` parameter is missing)

2) Check that all endpoints have pagination enabled (perhaps by using a `DEFAULT_PAGINATION_CLASS` http://www.django-rest-framework.org/api-guide/pagination/#limitoffsetpagination )

3) Check the pagination sizes are appropriately set (overriding any default set in #2 where needed). We should also likely use different default page sizes to max page sizes in most cases.


For example another unbounded endpoint:

$ time curl https://treeherder.mozilla.org/api/machine/
curl: (52) Empty reply from server

real    0m42.662s
user    0m0.031s
sys     0m0.015s


Complete list we should test:

https://treeherder.mozilla.org/api/bugscache/
https://treeherder.mozilla.org/api/buildplatform/
https://treeherder.mozilla.org/api/classified-failure/
https://treeherder.mozilla.org/api/exclusion-profile/
https://treeherder.mozilla.org/api/failureclassification/
https://treeherder.mozilla.org/api/job-exclusion/
https://treeherder.mozilla.org/api/jobdetail/
https://treeherder.mozilla.org/api/jobgroup/
https://treeherder.mozilla.org/api/jobtype/
https://treeherder.mozilla.org/api/machine/
https://treeherder.mozilla.org/api/machineplatform/
https://treeherder.mozilla.org/api/matcher/
https://treeherder.mozilla.org/api/optioncollectionhash/
https://treeherder.mozilla.org/api/performance/alertsummary/
https://treeherder.mozilla.org/api/performance/alert/
https://treeherder.mozilla.org/api/performance/framework/
https://treeherder.mozilla.org/api/product/
https://treeherder.mozilla.org/api/repository/
https://treeherder.mozilla.org/api/text-log-summary/
https://treeherder.mozilla.org/api/text-log-summary-line/
https://treeherder.mozilla.org/api/user/

https://treeherder.mozilla.org/api/project/mozilla-inbound/jobs/
https://treeherder.mozilla.org/api/project/mozilla-inbound/runnable_jobs/
https://treeherder.mozilla.org/api/project/mozilla-inbound/resultset/
https://treeherder.mozilla.org/api/project/mozilla-inbound/artifact/
https://treeherder.mozilla.org/api/project/mozilla-inbound/note/
https://treeherder.mozilla.org/api/project/mozilla-inbound/bug-job-map/
https://treeherder.mozilla.org/api/project/mozilla-inbound/logslice/
https://treeherder.mozilla.org/api/project/mozilla-inbound/job-log-url/
https://treeherder.mozilla.org/api/project/mozilla-inbound/performance/data/
https://treeherder.mozilla.org/api/project/mozilla-inbound/performance/signatures/
https://treeherder.mozilla.org/api/project/mozilla-inbound/performance/platforms/

Comment 1

[Ed Morley [:emorley]]

This would have helped with bug 1386331.

Comment 2

[Ed Morley [:emorley]]

Bug 1437078 will help with this. Several of the unused APIs are the chronic ones that cause timeouts.

For example this one doesn't use pagination, so attempts (and fails) to query/return X million machine names:
https://treeherder.mozilla.org/api/machine/

Comment 3

[moijes]

@emorley Is this still open for work ? I'd like to take a go at it.

Comment 4

[Cameron Dawson [:camd]]

moijes: emorley is no longer with Mozilla.

Are you still interested in this bug? This may not be an easy first bug, but you're welcome to take a crack at it. I would suggest doing one API at a time. They can be a bit tricky.

Alternatively, you could search the "Tree Management" product for bugs that have keyword "good-first-bug" in them. If they're not assigned, you could start there.

Either way.