SM-tc jobs on Beta permafailing due to "Error: cannot GET /DeveloperServices/HistoricalVCSbuild/tooltool.git/info/refs?service=git-upload-pack (404)" errors

RESOLVED FIXED

Status

Firefox Build System
Task Configuration
P1
blocker
RESOLVED FIXED
2 years ago
5 months ago

People

(Reporter: RyanVM, Assigned: gps)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
Appears to only affect Beta. Aurora has recent green runs.

https://public-artifacts.taskcluster.net/dUxAa3LHS1WuKYMTWZXVMw/0/public/logs/live_backing.log
Flags: needinfo?(gps)
(Assignee)

Comment 1

2 years ago
This is a consequence of HTTP redirecting git.mozilla.org as of ~9 hours ago in bug 1290260 (in preparation for its decom).

I guess we didn't realize we had Firefox automation still connecting to git.mozilla.org. If beta is impacted, presumably release and esr are impacted. That means there are release implications.

Technically these are tier-1 jobs. But according to RyanVM they "probably" don't block release since we still have buildbot SM automation running.

If they do block release, we either need a workaround ASAP or to revert the change in bug 1290260.
Severity: normal → blocker
Depends on: 1290260
Flags: needinfo?(gps)
Priority: -- → P1

Comment 2

2 years ago
20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 20

Platform breakdown:
* linux64: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296150&startday=2016-08-17&endday=2016-08-17&tree=all
(Assignee)

Comment 3

2 years ago
I guess the git.mozilla.org landed more than 9 hours ago. Relative timestamps are hard.
(Reporter)

Comment 4

2 years ago
AFAICT, no SM-tc jobs run on m-r or esr45.
(Assignee)

Comment 5

2 years ago
(In reply to Ryan VanderMeulen [:RyanVM] from comment #4)
> AFAICT, no SM-tc jobs run on m-r or esr45.

Excellent. That makes this a "someone needs to fix it now" problem to "it can probably wait several hours" problem.
(Assignee)

Comment 6

2 years ago
Commits we may want to cherry pick:

2ffda4181e7f (bug 1295900) - currently only on autoland
8e93f47555f5 (bug 1290685)
13fa96518284 (bug 1277957)

Of those, the last one is the one that will unbust the SM tasks.

It can't be grafted easily though since paths were renamed.
Depends on: 1277957
Comment hidden (mozreview-request)
(Assignee)

Updated

2 years ago
Assignee: nobody → gps
Status: NEW → ASSIGNED
(Reporter)

Comment 8

2 years ago
mozreview-review
Comment on attachment 8782219 [details]
Bug 1296150 - Obtain tooltool from github.com;

https://reviewboard.mozilla.org/r/72426/#review70036
Attachment #8782219 - Flags: review?(ryanvm) → review+
(Reporter)

Comment 10

2 years ago
Had to push one more follow-up fix, but things are green now.
https://hg.mozilla.org/releases/mozilla-beta/rev/ffe315faba1726473b709697314c13e2da071a62
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Comment 11

2 years ago
30 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 30

Platform breakdown:
* linux64: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296150&startday=2016-08-15&endday=2016-08-21&tree=all

Updated

5 months ago
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.