1277838 - [taskcluster-vcs] Move caches from git.m.o to other repositories

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Wander Lairson Costa]

git.m.o will shutdown. The following repositories are cached by taskcluster-vcs, we need to either remove them or redirect to appropriate alternative repos:

https://git.mozilla.org/b2g/B2G.git
https://git.mozilla.org/b2g/B2G (yes, it is duplicated, marked here to remember to fix that)
https://git.mozilla.org/build/tooltool.git
https://git.mozilla.org/external/google/gerrit/git-repo.git

Comment 1

[Hal Wine [:hwine] use NI!]

(In reply to Wander Lairson Costa [:wcosta] from comment #0)
> git.m.o will shutdown. The following repositories are cached by
> taskcluster-vcs, we need to either remove them or redirect to appropriate
> alternative repos:

If you do want to redirect, here are the upstream URLs used to mirror
these repositories that were being mirrored.

> https://git.mozilla.org/b2g/B2G.git
> https://git.mozilla.org/b2g/B2G 
git://github.com/mozilla-b2g/B2G.git -> git+ssh://git.m.o/b2g/B2G.git

> https://git.mozilla.org/build/tooltool.git
Er, that version has been out of date for a long time (May 2015). The
"live" version is at https://github.com/mozilla/build-tooltool

Per that README, only the one file is needed - ni :dustin to see if he has a suggestion on 'proper usage'

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I don't know if anything is still using that cache?  I think we bake tooltool into the image now:

testing/docker/desktop-build/Dockerfile
# install tooltool directly from github where tooltool_wrapper.sh et al. expect
# to find it
RUN wget -O /builds/tooltool.py https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
RUN chmod +x /builds/tooltool.py

However, I don't know if there are other things outside of Gecko using the tc-vcs repo.

Comment 3

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> I don't know if anything is still using that cache?  I think we bake
> tooltool into the image now:
> 
> testing/docker/desktop-build/Dockerfile
> # install tooltool directly from github where tooltool_wrapper.sh et al.
> expect
> # to find it
> RUN wget -O /builds/tooltool.py
> https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
> RUN chmod +x /builds/tooltool.py
> 
> However, I don't know if there are other things outside of Gecko using the
> tc-vcs repo.

We do have a few things specifying git.m.o in tree though for tooltool:

https://dxr.mozilla.org/mozilla-central/search?q=git.mozilla.org%2Fbuild%2Ftooltool&redirect=true

Such as: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/install-packages.sh#10

Comment 4

[Hal Wine [:hwine] use NI!]

(In reply to Wander Lairson Costa [:wcosta] from comment #0)
> git.m.o will shutdown. The following repositories are cached by
> taskcluster-vcs, we need to either remove them or redirect to appropriate
> alternative repos:
> https://git.mozilla.org/external/google/gerrit/git-repo.git

Okay, this one has some lore. We stopped mirroring when upstream broke our builds:
  https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/rev/68c767f8bebb
I can't find a bug that was filed with more details - I do recall we did a hard reset on the server repo to pin the version (otherwise the 'repo' script tried to update itself). iirc, trees were closed, but I find no bug for that, either. It was hectic!

The current heads on the server are:
[[e-mail]3 git-repo.git]# git branch -av
  maint  34acdd2 Fix ManifestParseError when first child node is comment
* master 503d66d Merge "project.RemoteFetch: Handle depth cases more robustly"
  stable 4ccad75 Fix substitution err for schemeless manifest urls

Original upstream URL:
  https://gerrit.googlesource.com/git-repo
  
ni: Catlee for more on the lore side of this. He was in the thick of it.

Comment 5

[Hal Wine [:hwine] use NI!]

(In reply to Justin Wood (:Callek) from comment #3)
> 
> We do have a few things specifying git.m.o in tree though for tooltool:
> 
> https://dxr.mozilla.org/mozilla-central/search?q=git.mozilla.
> org%2Fbuild%2Ftooltool&redirect=true
> 
> Such as:
> https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/
> builder/install-packages.sh#10

This seems not-a-taskcluster issue, so made it's own bug 1277957

Comment 6

[Wander Lairson Costa]

This leaves only gerrit repo. It feels like the official repository is this: https://gerrit.googlesource.com/gerrit/

I am not sure why do need the gerrit repository, however. I guess this is a Connected Devices dependency.

Comment 7

[Chris AtLee [:catlee]]

Bug 1007259 has some of the context, and links to other bugs where we were having problems due to 'repo' updating.

I don't think we need it now.

Comment 8

[Jonathan Griffin (:jgriffin)]

So can this bug be closed now?

Comment 9

[Wander Lairson Costa]

(In reply to Jonathan Griffin (:jgriffin) from comment #8)
> So can this bug be closed now?

Hrm, good question, I will assign to myself and check it.

Comment 10

[Wander Lairson Costa]

Attachment: Remove git.m.o references from cache update

git.m.o will be gone, so we move repo caches to other repositories.
Additionally, we remove the deprecated cache.js module.

Comment 11

[Greg Arndt [:garndt]]

Comment on attachment 8765967 [details] [review]
Remove git.m.o references from cache update

r+, deployed 2.3.38-rc.1 to test this out.

Comment 12

[Wander Lairson Costa]

leaving the bug open because we need to release a new version of taskcluster-vcs and ship in the relevant docker images.

Comment 13

[Wander Lairson Costa]

Attachment: Update docker image

Comment 14

[Greg Arndt [:garndt]]

Comment on attachment 8766343 [details] [review]
Update docker image

Looks good to me, just needs to be built and pushed before the tc-hook is updated to use that new image.  Let me know if you need me to assist with any of that.