TaskCluster builds mostly dying while getting 500s from quay.io

RESOLVED FIXED

Status

Taskcluster
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: philor, Unassigned)

Tracking

Details

Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 47

3 years ago
Looks like it got better after a couple of hours.

Probably still a https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy violation, though.
Severity: blocker → normal
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Depends on: 1146197
A few lines from: http://status.quay.io/
--------------------
Completed - The database upgrade has completed and service should be running normally. We thank you for your patience during this upgrade process. 
Mar 22, 18:03 EDT

In progress - We're beginning the second stage of the upgrade process. We expect an intermittent downtime of no more than 20-30 minutes during this period. 
Mar 22, 17:01 EDT
...

Scheduled - A database upgrade will occur from 4PM - 6PM on Sunday, March 22nd, 2015. We expect some intermittent periods of downtime during this upgrade. 
Mar 17, 13:13 EDT
--------------------

That might explain it, though my timezone math would have to be off by one hour :)

Anyways, this only makes me more interested in looking at using S3 for storing docker images.
Comment hidden (Treeherder Robot)
Blocks: 1153584
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
We're not using quay anymore, right? There's still the matter of bug 1147867 where a lot of these failures were also getting dumped, but I think this can be closed now.
Blocks: 1080265
Flags: needinfo?(jlal)
Closing this bug as it is now being tracked mostly in 1165759 and 1147867.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(jlal)
Resolution: --- → FIXED
Also, the issue is more of being able to retry failed pulls with any repository, not specific to just quay.  Bug 1170999 hopes to help with that.
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.