1296150 - SM-tc jobs on Beta permafailing due to "Error: cannot GET /DeveloperServices/HistoricalVCSbuild/tooltool.git/info/refs?service=git-upload-pack (404)" errors

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

Appears to only affect Beta. Aurora has recent green runs.

https://public-artifacts.taskcluster.net/dUxAa3LHS1WuKYMTWZXVMw/0/public/logs/live_backing.log

Comment 1

[Gregory Szorc [:gps]]

This is a consequence of HTTP redirecting git.mozilla.org as of ~9 hours ago in bug 1290260 (in preparation for its decom).

I guess we didn't realize we had Firefox automation still connecting to git.mozilla.org. If beta is impacted, presumably release and esr are impacted. That means there are release implications.

Technically these are tier-1 jobs. But according to RyanVM they "probably" don't block release since we still have buildbot SM automation running.

If they do block release, we either need a workaround ASAP or to revert the change in bug 1290260.

Comment 2

[Intermittent Failures Robot]

20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 20

Platform breakdown:
* linux64: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296150&startday=2016-08-17&endday=2016-08-17&tree=all

Comment 3

[Gregory Szorc [:gps]]

I guess the git.mozilla.org landed more than 9 hours ago. Relative timestamps are hard.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

AFAICT, no SM-tc jobs run on m-r or esr45.

Comment 5

[Gregory Szorc [:gps]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #4)
> AFAICT, no SM-tc jobs run on m-r or esr45.

Excellent. That makes this a "someone needs to fix it now" problem to "it can probably wait several hours" problem.

Comment 6

[Gregory Szorc [:gps]]

Commits we may want to cherry pick:

2ffda4181e7f (bug 1295900) - currently only on autoland
8e93f47555f5 (bug 1290685)
13fa96518284 (bug 1277957)

Of those, the last one is the one that will unbust the SM tasks.

It can't be grafted easily though since paths were renamed.

Comment 7

[Gregory Szorc [:gps]]

Attachment: Bug 1296150 - Obtain tooltool from github.com;

This is effectively a graft of 13fa96518284 (bug 1277957). I had to
make the changes manually since there were a lot of file renames
and refactoring between central and beta.

I'm not a fan of relying on github.com in our release automation.
I've been actively vendoring things to not rely on external services.
I have bug 1296148 on file to track removing references to github.com
from our automation. But with git.mozilla.org now offline, we can
either bite the bullet and connect to github.com (like our aurora
and central automation currently do) or do an invasive refactor
to vendor tooltool in beta. I'm inclined to let the eventual fixes
to stop relying on github.com to ride the trains.

Review commit: https://reviewboard.mozilla.org/r/72426/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/72426/

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8782219 [details]
Bug 1296150 - Obtain tooltool from github.com;

https://reviewboard.mozilla.org/r/72426/#review70036

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/9883ff680618

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Had to push one more follow-up fix, but things are green now.
https://hg.mozilla.org/releases/mozilla-beta/rev/ffe315faba1726473b709697314c13e2da071a62

Comment 11

[Intermittent Failures Robot]

30 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 30

Platform breakdown:
* linux64: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1296150&startday=2016-08-15&endday=2016-08-21&tree=all