1221064 - The job list endpoint times out when the result_set_id filter is not used

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, defect)

Description

[Mauro Doglio [:mdoglio]]

I believe the join with the resultset table is taking too much space in memory. We should probably make that join only when that filter is set.

Comment 1

[Mauro Doglio [:mdoglio]]

Attachment: PR 1111

Comment 2

[Ed Morley [:emorley]]

Comment on attachment 8682487 [details] [review]
PR 1111

Cancelling whilst the PR is updated :-)

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/caa8cb0f820084e2d7fb57448bb9cee121a3c664
Bug 1221064 - add new similar jobs endpoint

This endpoint will be used by the similar jobs panel in the ui.
It returns a list of jobs shaped in the same way as the job list
endpoint but ordered by resultset_id DESC. In order to achieve decent
performance the returned list is filtered by the same job type as the
one selected.

https://github.com/mozilla/treeherder/commit/6cdbfa89b154b7767eda92d195f0d4a35d5022b0
Bug 1221064 - change source endpoint for the similar jobs panel

Let's use the new endpoint to keep the sorting in the similar jobs list
right.

https://github.com/mozilla/treeherder/commit/9701ea433b4f4ca5901cf0716c2d33ac596dbe70
Bug 1221064 - improve job list performance

The job list endpoint was joining the job table and the resultset table
with the only purpose of sorting the results by push_timestamp.Also we
don't really need any sorting on that endpoint, so let's remove it.

Comment 4

[Ed Morley [:emorley]]

Hmm I wonder if this is going to affect other consumers of the /jobs/ API?

Comment 5

[Mauro Doglio [:mdoglio]]

Attachment: PR 1124

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/761ae7f163d0550b124045125255cd0c3b7bd415
Bug 1221064 - make the job list order explicit

Comment 7

[Ed Morley [:emorley]]

(In reply to Treeherder Bugbot from comment #6)
> Commit pushed to master at https://github.com/mozilla/treeherder
> 
> https://github.com/mozilla/treeherder/commit/
> 761ae7f163d0550b124045125255cd0c3b7bd415
> Bug 1221064 - make the job list order explicit

I ended up reverting this since it seems to be a contributing factor to DB load at the moment. The first and last deploy markers here are when it landed and was reverted:

https://rpm.newrelic.com/accounts/677903/applications/4180461/datastores?tw%5Bend%5D=1447189515&tw%5Bstart%5D=1447167915#/overview/All/drilldown?metric=Datastore%252Fstatement%252FMySQL%252Fjob%252Fselect&value=total_call_time_per_minute

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0ecb647f795c15604b191c5b4010956068b9fb8f
Revert "Bug 1221064 - make the job list order explicit"

This reverts commit 761ae7f163d0550b124045125255cd0c3b7bd415.