758003 - Releng needs for testing production git

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[hwine]

From email thread with cshields & joduinn. Most needed items are at top.

From: Hal Wine <[e-mail]>
Subject: Re: Testing access for git.m.o requirements
Date: May 23, 2012 13:49:07 PDT
To: Corey Shields <[e-mail]>
Cc: John O'Duinn <[e-mail]>

Corey,

I may have obfuscated my needs at this point, so wanted to clarify. To start testing, I need:
 - url to new host, and any relevant connection info (vpn, etc.)
 - 4 repos to test against: 1 each r/o & r/w on hg.m.o and the new git server

The the non-technical issues can be worked out in parallel.

Please let me know who will be working on this, and where the work is tracked. My interest is in having a smooth handoff, so I'll be following the old "over communicate" rule.

--Hal

P.S. I know the dc move was the main focus for quite a while - here's an earlier email on this topic (subject: Touching base on git.m.o; date: 2012-04-20)

On May 22, 2012, at 13:49 , Hal Wine wrote:

Corey,

Before I get into the technical questions, let me ask how we're going to handle the policy questions. 

My understanding at the time I did the original git on allizom eval was that the server would be somehow partitioned into "release managed" and "IT managed" parts. Releng would be responsible for all hooks, etc. installed for the "release managed repos". My current understanding is that IT will be managing all parts of git.m.o, just as they do for hg.m.o.

If IT is managing all of git.m.o, then I mostly need to understand how the requirements John has previously relayed will be implemented, so I can verify they match our expectations. As I understand the requirements:
release owned repositories are all those under hg.m.o/{releases,build,projects,integration,l10n,l10n-central} + mozilla-central + try
a similar "name space" structure will be available on git.m.o for (possible) future git versions of these repositories
any "equivalent" repo will be set up to have only one repo (hg or git) be committable (except by an internal release user to maintain the mirroring) (i.e. one will be the "repository of record" and the other a read-only version)
for release owned repositories, the present rule is hg is the repository of record

At the technical level, to verify the above, I'll need 4 repos created to fill the matrix of permissions (2 on each server, one r/o, the other r/w). I can execute various workflows against those test repositories to validate expectations about commit levels, etc. It would be helpful to be able to read the hook code, just to ensure we don't go planning in an incompatible direction without knowing it's going to be a major headache.

Once the validation work is complete, then work can begin adding repos to git.m.o and make them live (in r/o mode) and see how things go from a load perspective.

--Hal

Comment 1

[Ben Kero [:bkero]]

The host running these is not currently routed to the internet, so a VPN connection to our SCL3 datacentre is required.  The hostname is git1.dmz.scl3.mozilla.com, and the gitweb interface is http://git1.dmz.scl3.mozilla.com/git/

git cloning can be done by doing 'git clone [e-mail]:test_hwine.git'

These repos were created for hg:

drwxr-sr-x 3 [e-mail] scm_level_2 4.0K May 29 14:12 test_hwine
drwxrwsr-x 3 hg                scm_level_2 4.0K May 29 14:13 test_scm2

and these repos were created for git.  Please note that the presence of the owner/desc params means that the repositories are public and will be listed on gitweb:

gitolite::repo {
        "test_hwine":
            rwplus => "[e-mail]",
            owner  => "Hal Wine",
            desc   => "Hal's git/hg test plaything";
 
        "test_scm2":
            rwplus => "@scm_level_2",
            owner  => "Hal Wine",
            desc   => "Hal's git/hg test plaything";
}

Comment 2

[Shyam Mani [:fox2mike]]

Hal, please let us know if you're blocked on anything else at this point.

Comment 3

[hwine]

Shyam - will do - I have a release to get started this morning, then I'll be looking at the setup for the first time, since bug 759517 was fixed last night.

Comment 4

[hwine]

confirmed I have write access to the repos, so can begin full testing Friday.

Note: something isn't quite right with the web access (not needed for my initial testing):
Thu May 31 16:54:58 PDT 2012
0 [Hal@Hals-MacBook-Air test_1]
$ curl http://git1.dmz.scl3.mozilla.com/git/
<h1>Software error:</h1>
<pre>syntax error at /etc/gitweb.conf line 35, near &quot;$feature&quot;
</pre>
<p>
For help, please send mail to the webmaster (<a href="mailto:root@localhost">root@localhost</a>), giving this error message 
and the time and date of the error.

</p>
0 [Hal@Hals-MacBook-Air test_1]

Comment 5

[Shyam Mani [:fox2mike]]

Yes, the web access is a known issue and Ben is looking into it.

Comment 6

[hwine]

(In reply to Ben Kero [:bkero] from comment #1)
> These repos were created for hg:
> 
> drwxr-sr-x 3 [e-mail] scm_level_2 4.0K May 29 14:12 test_hwine
> drwxrwsr-x 3 hg                scm_level_2 4.0K May 29 14:13 test_scm2
> 
> and these repos were created for git.  Please note that the presence of the
> owner/desc params means that the repositories are public and will be listed
> on gitweb:
> 
> gitolite::repo {
>         "test_hwine":
>             rwplus => "[e-mail]",
>             owner  => "Hal Wine",
>             desc   => "Hal's git/hg test plaything";
>  
>         "test_scm2":
>             rwplus => "@scm_level_2",
>             owner  => "Hal Wine",
>             desc   => "Hal's git/hg test plaything";
> }

Confirmed that access works as desired w.r.t. restricting write access to named user in test_hwine on both git & hg.

Comment 7

[hwine]

Attachment: public key for writing mirror versions of repos

Per email exchange, please create a new account with no SCM level permissions, and attached ssh key as auth.

Then change ownership on the two test repos (test_hwine on hg & git) to be writable only by this key.

Thanks!

Comment 8

[Justin Dow [:jabba]]

What username for this account?

Comment 9

[hwine]

Shoot - thought a username suggestion was in there, but not.

How about: mirror-repo or repo-sync or hgsyncgit. Ping me in channel if none of those work.

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to Justin Dow [:jabba] from comment #8)
> What username for this account?

Let's go with github-sync-test for now.

Comment 11

[Justin Dow [:jabba]]

I've created:

uid=github-sync-test,ou=logins,dc=mozilla
mail=[e-mail]

with the attached SSH key and added the hgAccount objectClass to it, but did not put the user in any SCM_level groups. I believe this is all that is required on my end. Punting back to bkero for the repo work.

Comment 12

[Ben Kero [:bkero]]

I've created a repo for you with the following permissions:

repo    test_github-sync
    RW+     = [e-mail]
    R       = @all gitweb
    test_github-sync "Hal Wine" = "Github sync testing"

In English that means the name of the repo is 'test_github-sync', which is writable only by the [e-mail] user, and is readable by @all and gitweb.

Comment 13

[Ben Kero [:bkero]]

What is still required to close this bug out?

Comment 14

[hwine]

I believe this one is ready to close.

Comment 15

[hwine]

reopening due to bug 769148 - that will be part of the test case now

Comment 16

[Ben Kero [:bkero]]

769148 is fixed

Comment 17

[Shyam Mani [:fox2mike]]

Hal,

I'd like a new account or this account name changed before we go "live". It should just be [e-mail]. Thoughts? Let's discuss on IRC before we go live.

Comment 18

[hwine]

(In reply to Shyam Mani [:fox2mike] from comment #17)
> Hal,
> 
> I'd like a new account or this account name changed before we go "live". It
> should just be [e-mail]. Thoughts? Let's discuss on IRC
> before we go live.

discussed - not a go live blocker, so moved to bug 770576

rest good