Open Bug 1629115 Opened 3 months ago Updated 1 month ago

vcssync: dev/staging git-cinnabar switchover

Categories

(Developer Services :: General, task)

task
Not set
normal

Tracking

(Not tracked)

People

(Reporter: dhouse, Assigned: dhouse, NeedInfo)

References

(Blocks 1 open bug)

Details

Attachments

(6 files, 5 obsolete files)

1.24 KB, patch
dhouse
: review+
Details | Diff | Splinter Review
2.11 KB, patch
dhouse
: review+
Details | Diff | Splinter Review
8.70 KB, patch
dhouse
: review+
Details | Diff | Splinter Review
1.31 KB, patch
dhouse
: review+
Details | Diff | Splinter Review
1.29 KB, patch
dhouse
: review+
Details | Diff | Splinter Review
2.42 KB, patch
dhouse
: review+
Details | Diff | Splinter Review

To help setup the CI system for KaiOS, we need to switch vcs-sync to use git-cinnabar. To do that, we need to prepare vcssync for development/staging and lock-down the secrets and repo once it is in production.

copied from email from Hal:

Please setup a new key for the KaiOS project, and share that one with glandium. The public part will need to be added to the repo to allow pushes.

Glandium: for dev/staging work, I'm assuming you'll use a new repo, which you'll have admin rights to, and can install the key.

Dave: once we go live, we'll use a new key in production, and I'll lock down the production repo. Can you set up a tracker bug so we don't lose sight of the steps, please?

Friday afternoon, I add :glandium to the jumphost and vcssync host auth_keys, and two ip's for him to access through the security group to the jumphost.

The mozharness repo is not on phabricator, so I didn't know where to attach this.

This doesn't take care of the from-scratch conversion, and doesn't handle the steps to actually do the switch from hg-git to git-cinnabar, nor the installation of git-cinnabar. These will have to be manual steps as outlined by email.

Attachment #9140322 - Flags: review?(dhouse)
Attachment #9140322 - Attachment is obsolete: true
Attachment #9140322 - Flags: review?(dhouse)
Attachment #9140331 - Flags: review?(dhouse)
Attachment #9140582 - Flags: review?(dhouse)
Attachment #9140582 - Attachment is obsolete: true
Attachment #9140582 - Flags: review?(dhouse)
Attachment #9140586 - Flags: review?(dhouse)
Attachment #9140331 - Attachment is obsolete: true
Attachment #9140331 - Flags: review?(dhouse)
Attachment #9140620 - Flags: review?(dhouse)

And upgrade mercurial.

Attachment #9140622 - Flags: review?(dhouse)
Attachment #9140620 - Attachment is obsolete: true
Attachment #9140620 - Flags: review?(dhouse)
Attachment #9140649 - Flags: review?(dhouse)
Attachment #9140583 - Flags: review?(dhouse) → review+
Attachment #9140586 - Flags: review?(dhouse) → review+
Attachment #9140622 - Flags: review?(dhouse) → review+
Comment on attachment 9140649 [details] [diff] [review]
Add support for git-cinnabar conversions to the vcs-sync codebase

r+ this looks great. Thank you for making the mappings work. I don't think we need more comments explaining this.
Attachment #9140649 - Flags: review?(dhouse) → review+

I turned on aws monitoring of downloads for the mappings files in s3 ("Access analyzer for S3"). So we can see if anyone is using those, and then turn off the mapping creation if they are unused for a month.

Related to finding who is using the output, we had previously checked if github provided logs or stats of repository clones/pulls and they do not.

The host doesn't exist in the DNS server vcs-sync uses, and that causes long pauses when installing/upgrading python packages.

Attachment #9140949 - Flags: review?(dhouse)

This is an updated patch with the git-cinnabar path added to the config.

Attachment #9140622 - Attachment is obsolete: true
Attachment #9140950 - Flags: review?(dhouse)
Attachment #9140948 - Flags: review?(dhouse) → review+
Attachment #9140949 - Flags: review?(dhouse) → review+
Attachment #9140950 - Flags: review?(dhouse) → review+

NI to myself to change the log archiving to leave the log archives locally (not deleted after upload to s3).

Flags: needinfo?(dhouse)

(In reply to Dave House [:dhouse] from comment #17)

NI to myself to change the log archiving to leave the log archives locally (not deleted after upload to s3).

I had changed it keep the logs locally and we put the 13,14,15 archives onto the filesystem also so April13 -> now logs are collected in vcssync1/. (and April16+ for gecko-projects(vcssync2)).

Flags: needinfo?(dhouse)

The conversion and mappings have been created without problems since the switchover.

Are there any plans to also switch the nss repo over to using cinnabar? That's the only other one that searchfox is currently pulling from github and then grafting cinnabar data to.

Flags: needinfo?(dhouse)

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #20)

Are there any plans to also switch the nss repo over to using cinnabar? That's the only other one that searchfox is currently pulling from github and then grafting cinnabar data to.

:jcj just a heads-up: we're planning to switch the github sync of nss to use git-cinnabar. This will have no impact unless someone tries to reproduce the git repo from hg.m.o with hg-git. From what I understand when we talked in Berlin, it may not be used anymore: https://github.com/nss-dev/nss

Flags: needinfo?(jjones)

As I understand it, it should have no impact for the changeover. I know people use that mirror, but we only use it as a mirror. hg.m.o is the authoritative source, and even if github.com/nss-dev/nss lost its history we'd be perfectly fine.

Flags: needinfo?(jjones)

(In reply to J.C. Jones [:jcj] (he/him) [increased latency due to COVID-19] from comment #22)

As I understand it, it should have no impact for the changeover. I know people use that mirror, but we only use it as a mirror. hg.m.o is the authoritative source, and even if github.com/nss-dev/nss lost its history we'd be perfectly fine.

Thank you ;jcj ! I'll plan to move this over this in the coming month.

:kats, cc from #vcs: I have no update on moving the nss repo to cinnabar. I haven't worked on it this month. I'd like to postpone that for a few more weeks (until other pieces in ci are quieter), unless that causes problems or work on your side.

:dhouse, thanks for the update! No rush on my side. It's more of a "crossing off the last item on a checklist" feeling kind of thing, rather than any particular actual use-case that we need this for.

You need to log in before you can comment on or make changes to this bug.