997708 - git.m.o mirroring

Status: RESOLVED DUPLICATE of bug 1163749

(Developer Services :: Git, defect)

Description

[Kendall Libby [:fubar] (he/him)]

We need to set up mirroring for gitmo, both for performance reasons (e.g. 985864) and for DR/BC (Q/YR goal). Initial scope should just be within SCL3, but we should keep an eye towards PHX1/AWS. 

While this is a Q goal, the next major merge day is June 9. We should use that as our need-before date.

Comment 1

[Kendall Libby [:fubar] (he/him)]

git[23].stage.dmz.scl3 kickstarted on a pair of seamicro xeons, puppetized, and added to puppet.

zeus set up for git-stage-zlb.vips.scl3; ssh pool to git1, http/https pools to git[23].

git.allizom.org reclaimed and pointed at git-stage-zlb.

should be ready for setting up gitolite to mirror frmo git1 to git[23].

Comment 2

[Peter Radcliffe [:pir]]

git is still seeing occasional high load spikes taking out git.m.o port 80 and 443. Hopefully this work can be finished sooner rather than later.

Comment 3

[Ben Kero [:bkero]]

To get this done I'll need to:

1) Put the changes present in git staging hosts (that currently have this set up) into puppet
2) Deploy this on the newly built git mirror hosts
3) Have releng QA the new infra

Comment 4

[Laura Thomson :laura]

bkero: any updates?

Comment 5

[Ben Kero [:bkero]]

The git mirrors have been built out and have puppet configs applied.

The next step is to coordinate with release engineering to get the hook into production. The scripts are written, and are currently being run on the staging infrastructure.

Comment 6

[hwine]

Catlee: based on our last meeting, is there any testing you wish to see performed on the staging area before this is rolled to production?

bkero: once catlee signs off, you are good to proceed with go-live

Comment 7

[Chris AtLee [:catlee]]

Is there an URL we can use for testing?

If so, I'd like to do a basic smoketest that we can do a few B2G builds at once against the new mirrors.

Comment 8

[Ben Kero [:bkero]]

https://git.allizom.org/ is the URL that can be used for testing

Comment 9

[Ben Kero [:bkero]]

catlee,

This is tied to one of our quarterly goals. Could I get a response one way or another with this by the 19th, that way we have time to get this done before quarter end?

Comment 10

[Chris AtLee [:catlee]]

We can't test this because it doesn't have any of the B2G repos I need on it.

Please have the full contents of git.m.o mirrored for testing.

Comment 11

[hwine]

So we have competing priorities here. Let's see if we can find a reasonable path.

Catlee: the smoke test would only catch something 'by chance', many eyes would still be needed when rolled out to production. It also would require significant effort, including support from the vcs-sync system, adding yet-another dependency. Which of the following 2 approaches would meet your needs:
 a) See the detailed deploy and rollback plan, and thus feel comfortable with skipping the smoke test. (Worst case, it will prove the process of rolling back to address any can-be-found-only-in-production issues.)
 b) Working out the specifics of a smoke test that doesn't require vcssync to support.

bkero: we'll need (a) no matter what. Can you open a separate bug for "deploy mirroring to production" with all the puppet changes, etc. needed please? Along with rollback operations?

Comment 12

[Chris AtLee [:catlee]]

The staging setup has none of the repos I need to do even basic testing. The only one it has is gecko.git, which isn't used by our automation.

Can we rsync over the repos to staging so I can test some b2g builds?

Comment 13

[hwine]

(In reply to Chris AtLee [:catlee] from comment #12)
> The staging setup has none of the repos I need to do even basic testing. The
> only one it has is gecko.git, which isn't used by our automation.
> 
> Can we rsync over the repos to staging so I can test some b2g builds?

bkero: do we have enough disk for that? Catlee means everything under:
 b2g/*
 external/*
 releases/*

Comment 14

[Ben Kero [:bkero]]

We have space for those repositories, so I am mirroring all of them over now.

Comment 15

[Ben Kero [:bkero]]

One gotcha is to keep them in sync we're going to need the deployment hook installed on the SSH master. Otherwise you're going to have to do smoke testing with the content that is being mirrored over there now.

Comment 16

[hwine]

(In reply to Ben Kero [:bkero] from comment #15)
> One gotcha is to keep them in sync we're going to need the deployment hook
> installed on the SSH master. Otherwise you're going to have to do smoke
> testing with the content that is being mirrored over there now.

Right - we are not going to keep them in sync with the live content. That is a major difference between production and staging. Deploying the hook on the ssh host would change the config from production.

Catlee: I am assuming that, if you want to push any changes, you'll want to do that in a controlled fashion to uncover the race condition behavior. 

Bkero: on staging after doing the rsync, can you set all these repos to be writable by scm level 1 (so anyone can test), and update catlee on how to push to staging, please?

Comment 17

[Greg Cox [:gcox]]

FYI: git1.dmz.phx1 and gitweb3.dmz.scl3 are out of warranty and will be looked at for p2v.  I'm fine waiting til this is done and we hit more robust/steady-state, but wanted to mention in case that has any bearing on how you approach this bug.

Comment 18

[Greg Cox [:gcox]]

And, in more full disclosure, the staging environment has been p2v'ed in bug 1079089, due to the seamicros closing in on being out of warranty.

Comment 19

[Ben Kero [:bkero]]

gitweb[123].dmz.scl3 are set up and receiving updated contents of the repositories on a minutely cronjob.

Adding these hosts to the zeus git.m.o node pool was determined to be too large of an impact. To fix this we should determine whether we'd like to add a push-to-pull model to the cronjob, or to set up the hosts as mirror.git.mozilla.org and determine that the 1 minute sync delay is acceptable to us.

Comment 20

[hwine]

Appears to have been done in bug 1163749