1629115 - vcssync: dev/staging git-cinnabar switchover

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[:dhouse]

To help setup the CI system for KaiOS, we need to switch vcs-sync to use git-cinnabar. To do that, we need to prepare vcssync for development/staging and lock-down the secrets and repo once it is in production.

Comment 1

[:dhouse]

copied from email from Hal:

Please setup a new key for the KaiOS project, and share that one with glandium. The public part will need to be added to the repo to allow pushes.

Glandium: for dev/staging work, I'm assuming you'll use a new repo, which you'll have admin rights to, and can install the key.

Dave: once we go live, we'll use a new key in production, and I'll lock down the production repo. Can you set up a tracker bug so we don't lose sight of the steps, please?

Comment 2

[:dhouse]

Friday afternoon, I add :glandium to the jumphost and vcssync host auth_keys, and two ip's for him to access through the security group to the jumphost.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Add support for git-cinnabar conversions to the vcs-sync codebase

The mozharness repo is not on phabricator, so I didn't know where to attach this.

This doesn't take care of the from-scratch conversion, and doesn't handle the steps to actually do the switch from hg-git to git-cinnabar, nor the installation of git-cinnabar. These will have to be manual steps as outlined by email.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Add support for git-cinnabar conversions to the vcs-sync codebase

Comment 5

[Mike Hommey [:glandium]]

Attachment: Use virtualenv python for git-cinnabar

Comment 6

[Mike Hommey [:glandium]]

Attachment: Also run `git cinnabar fsck` after conversion

Comment 7

[Mike Hommey [:glandium]]

Attachment: Use virtualenv python for git-cinnabar

Comment 8

[Mike Hommey [:glandium]]

Attachment: Add support for git-cinnabar conversions to the vcs-sync codebase

Comment 9

[Mike Hommey [:glandium]]

Attachment: Switch gecko-dev and gecko-projects to git-cinnabar

And upgrade mercurial.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Add support for git-cinnabar conversions to the vcs-sync codebase

Comment 11

[:dhouse]

Comment on attachment 9140649 [details] [diff] [review]
Add support for git-cinnabar conversions to the vcs-sync codebase

r+ this looks great. Thank you for making the mappings work. I don't think we need more comments explaining this.

Comment 12

[:dhouse]

I turned on aws monitoring of downloads for the mappings files in s3 ("Access analyzer for S3"). So we can see if anyone is using those, and then turn off the mapping creation if they are unused for a month.

Related to finding who is using the output, we had previously checked if github provided logs or stats of repository clones/pulls and they do not.

Comment 13

[Mike Hommey [:glandium]]

Attachment: Properly use environment set from configuration for git-cinnabar

Comment 14

[Mike Hommey [:glandium]]

Attachment: Remove references to pypi.pvt.build.mozilla.org

The host doesn't exist in the DNS server vcs-sync uses, and that causes long pauses when installing/upgrading python packages.

Comment 15

[Mike Hommey [:glandium]]

Attachment: Switch gecko-dev and gecko-projects to git-cinnabar

This is an updated patch with the git-cinnabar path added to the config.

Comment 16

[Mike Hommey [:glandium]]

https://hg.mozilla.org/build/mozharness/rev/9e1dce7755d0a6b1ba7e88140d1fe523f89cf283
https://hg.mozilla.org/build/mozharness/rev/f9e15479984a94a7cdd03fa6fc6c0b6b94ac2852
https://hg.mozilla.org/build/mozharness/rev/904f897a1305a6a2dc02cfe754f006a7da37605d
https://hg.mozilla.org/build/mozharness/rev/187ceee0ce97ce7e1a7ee1dd68dfe58fa68f0584
https://hg.mozilla.org/build/mozharness/rev/86d4c7fc2280c4d4350047ad46fb86f40960476f
https://hg.mozilla.org/build/mozharness/rev/765bcb44ac00cb574769f2d6d9dae0dd974cb4a9

Comment 17

[:dhouse]

NI to myself to change the log archiving to leave the log archives locally (not deleted after upload to s3).

Comment 18

[:dhouse]

(In reply to Dave House [:dhouse] from comment #17)

NI to myself to change the log archiving to leave the log archives locally (not deleted after upload to s3).

I had changed it keep the logs locally and we put the 13,14,15 archives onto the filesystem also so April13 -> now logs are collected in vcssync1/. (and April16+ for gecko-projects(vcssync2)).

Comment 19

[:dhouse]

The conversion and mappings have been created without problems since the switchover.

Comment 20

[Kartikaya Gupta (email:[e-mail])]

Are there any plans to also switch the nss repo over to using cinnabar? That's the only other one that searchfox is currently pulling from github and then grafting cinnabar data to.

Comment 21

[:dhouse]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #20)

Are there any plans to also switch the nss repo over to using cinnabar? That's the only other one that searchfox is currently pulling from github and then grafting cinnabar data to.

:jcj just a heads-up: we're planning to switch the github sync of nss to use git-cinnabar. This will have no impact unless someone tries to reproduce the git repo from hg.m.o with hg-git. From what I understand when we talked in Berlin, it may not be used anymore: https://github.com/nss-dev/nss

Comment 22

[J.C. Jones [:jcj] (he/they)]

As I understand it, it should have no impact for the changeover. I know people use that mirror, but we only use it as a mirror. hg.m.o is the authoritative source, and even if github.com/nss-dev/nss lost its history we'd be perfectly fine.

Comment 23

[:dhouse]

(In reply to J.C. Jones [:jcj] (he/him) [increased latency due to COVID-19] from comment #22)

As I understand it, it should have no impact for the changeover. I know people use that mirror, but we only use it as a mirror. hg.m.o is the authoritative source, and even if github.com/nss-dev/nss lost its history we'd be perfectly fine.

Thank you ;jcj ! I'll plan to move this over this in the coming month.

:kats, cc from #vcs: I have no update on moving the nss repo to cinnabar. I haven't worked on it this month. I'd like to postpone that for a few more weeks (until other pieces in ci are quieter), unless that causes problems or work on your side.

Comment 24

[Kartikaya Gupta (email:[e-mail])]

:dhouse, thanks for the update! No rush on my side. It's more of a "crossing off the last item on a checklist" feeling kind of thing, rather than any particular actual use-case that we need this for.