1947852 - [trees closed] internal server errors for requests to https://firefox-ci-tc.services.mozilla.com , e.g. hooks

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://firefox-ci-tc.services.mozilla.com returns internal server errors, e.g. for retriggers from Treeherder which fetches the actions, or during the execution of the on-push tasks:

"Retrigger failed with Decision task: C8ZuGswLT0ehopKUQc4ftQ: Error: While calling createTask: InternalServerError Internal Server Error, incidentId 4cc4d26e-aeb0-4cc4-a738-deaa15a6cd9d."
https://firefox-ci-tc.services.mozilla.com/hooks/hg-push/autoland GJOyUY4RSCax1pSnHuZdCA

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The issue disappeared without human intervention.

From Gabriel: "looks like it's a message queue timeout, maybe pulse was unreachable or slow in responding"

Comment 2

[Yarik Kurmyza [:yarik] (he/him) (UTC+1)]

Linked incidentID is about Pulse publisher deadline exceeded message.

Turns out there was a ~20min downtime and spike of errors between 20:38 and 20:59 CET (12 February)