Closed Bug 1343599 Opened 8 years ago Closed 8 years ago

[Alert] treeherder-prod: Response times (web) 2017-03-01

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: emorley)

References

Details

Fired twice in the last hour (only sent to my email at present): """ The 95th percentile response time for treeherder-prod has exceeded your threshold setting of 1000ms. """ Investigating.
This appears to be due to increased activity to: * SetaJobPriorityViewSet.list * RunnableJobsViewSet.list ...and since the response time for both of these is pretty high this raises the 95th percentile average response time, which is what the Heroku alert monitors. The increased activity is presumably due to the switching over to Treeherder's SETA implementation in bug 1326102. The reason those API requests are slow is that they make requests to both index.taskcluster.net and queue.taskcluster.net as part of the API call, which is bug 1339829. It would be good to improve the response times for these endpoints, since: * otherwise we'll have to raise the alert thresholds, which could mask issues with non-SETA API calls * slow requests on webheads means lower capacity (or increased request queuing times) for the same number of dynos
Blocks: 1326102
Depends on: 1339829
Rob/Joel, would you mind taking a look at this sometime in the next few days? (Doesn't need to be today, I'll just ignore the alerts for now. I'm also fine with bug 1326102 staying landed.)
Flags: needinfo?(rwood)
Flags: needinfo?(jmaher)
/me passes this to rwood...
Flags: needinfo?(jmaher)
I'll see what I can do!
Flags: needinfo?(rwood)
Resolved via bug 1339829.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.