1435023 - Requests to the JobsViewSet.retrieve have increased from 10rpm to 5000rpm in the last 3 hours

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Screenshots say all.

Looking at the requests they are:
* all to the "retrieve" endpoint (ie single jobs, not a list)
* from a variety of legitimate browser looking User Agents
* all with referrer https://treeherder.mozilla.org/

They appear to start shortly after the deploy, and have been ramping up since.

The deploy changes were:
https://github.com/mozilla/treeherder/compare/d0bfec4c9ea1...a4b48832d38d

I think the cause is the new listener on `thEvents.applyNewJobs` which is calling `getGeckoDecisionTaskId()` many many times:
https://github.com/mozilla/treeherder/commit/d41c3181737a6fc3aa9b8863cae70e7f21b3c189#diff-adda36d471aeae2f9f74fedd4bfa0dccR112

This can be seen by opening:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

...and watching for the 50+ requests of form:
https://treeherder.mozilla.org/api/project/mozilla-inbound/jobs/NNNNNNNNN/

On the plus side, our API and DB held up to a 2 orders of magnitude increase in these requests :-)

Comment 1

[Ed Morley [:emorley]]

Attachment: New Relic graph of production

Comment 2

[Ed Morley [:emorley]]

Fixed by reverting the PR in bug 1401518:
https://github.com/mozilla/treeherder/commit/8e4c7db6c50e0f1432d6c79d039a178b511b7d8d

Comment 3

[Jonathan French (:jfrench)]

Fascinating. I would have thought it would have showed up stage prior to a prod deploy?

Comment 4

[Ed Morley [:emorley]]

Since it's a UI only change, it will only be triggered by people using the site, and not many people use stage.

Comment 5

[Ed Morley [:emorley]]

Also meant to say the ~1hr delay on the graphs before the initial spike and the main spike is due to people not refreshing the Treeherder UI until they get the "new production deploy occurred, click to refresh" notification thing.