528360 - Set up a git server

Status: RESOLVED FIXED

(Developer Services :: General, task, P2)

Description

[Wil Clouser [:clouserw]]

We talked about this in the meeting today with IT and it didn't sound like it would be much work.  The AMO team is interested in trying out git for Zamboni (amo:v4).

jbalogh recommends using http://github.com/res0nat0r/gitosis for permissions and management but that's up to y'all.

Comment 1

[matthew zeier [:mrz]]

Going to roll this into the whole hudson CI stuff.

Comment 2

[Fred Wenzel [:wenzel]]

As a side note, we'll probably be tracking Bouncer in git soon as well. I am merging the current trunk and production code into my git repository at http://github.com/fwenzel/tuxedo/ , but once we have a production git server at Mozilla, we can use that as the reference copy of the code.

Comment 3

[Fred Wenzel [:wenzel]]

If we don't want to use the ugliness that is gitweb, gitorious (http://gitorious.org/gitorious) seems to be a nice little open source alternative that provides some of the features github does. Maybe we want to use that so that our git.m.o upstream repositories aren't too sad-looking.

Comment 4

[Jeff Balogh (:jbalogh)]

(In reply to comment #3)
> Maybe we want to use that so that our git.m.o upstream repositories aren't too sad-looking.

I think an ugly gitmo is good, it'll push people back to github. ;)

Comment 5

[Jeremy Orem [:oremj]]

Aravind, will you create that netapp mount we talked about yesterday. Please assign back to me when it is ready.

Comment 6

[Jeff Balogh (:jbalogh)]

More Gitosis docs: http://progit.org/book/ch4-7.html

http://progit.org/book/ has a whole chapter of "Git on the Server."

Comment 7

[Jeremy Orem [:oremj]]

Unfortunately, I don't think gitosis will work since it needs to work with LDAP. Aravind already wrote a bunch of scripts for Mercurial, so I should be able to mostly copy those.

Comment 8

[Fred Wenzel [:wenzel]]

FWIW, this would help IT stay independent on github, especially now that we set up vendor repositories per application (and thus shifted our dependence from pypi to github).

Comment 10

[Laura Thomson :laura]

Can somebody in IT give us an update on this?

Comment 11

[Jeremy Orem [:oremj]]

I haven't looked at this, because webdev has been happy with github. I think Aravind could get this up and running the fastest since he's done a bunch of work on mercurial.

Comment 12

[Zandr Milewski [:zandr]]

Github has been in pain this morning, underscoring that we shouldn't have dependencies on external resources. This would, to use the vernacular, "close the tree".

I have an instance running in labs, but this really should be on HA hardware with good backups.

Comment 13

[Laura Thomson :laura]

I'm guessing this should be assigned to somebody other than aravind.

Comment 14

[Justin Dow [:jabba]]

Pete, I think we talked about this last week and you had mentioned it is something you could tackle? This might be a dupe though.

Comment 15

[Jeff Balogh (:jbalogh)]

Alternatively, we can mirror github repos automatically whenever there's an update (bug 649483).

Comment 16

[Pete Fritchman [:petef]]

cshields, you mentioned you guys have a machine somewhere I can run a local copy of gitorious on. Can you point me at that machine, and what kind of storage we should be using?

Comment 17

[Fred Wenzel [:wenzel]]

(In reply to comment #15)
> Alternatively, we can mirror github repos automatically whenever there's an
> update (bug 649483).

I don't think that's an alternative. In fact, I am quite sure this is exactly what's intended.

Comment 18

[Jeremy Orem [:oremj]]

This bug might be switching to what jabba is talking about comment 14, which is a git server for internal git repos that aren't hosted on github.

I'm not sure if we should combine the internal git repos and the git mirror from bug 649483. I don't have a strong opinion either way, but prefer internal repos go on a completely separate server i.e., this bug.

Comment 19

[Myk Melez [:myk] [@mykmelez]]

For the Jetpack project, we'd like a Mozilla-maintained public Git server like git.mozilla.org that serves as the canonical host for the Add-on SDK and Add-on Builder repositories.

We'll continue to use GitHub as a hub for collaboration, but our mirroring should be from gitmo to GitHub, so the only way to land a change into one of the canonical repos is to check it into that repo using a Mozilla-authorized account, which provides greater assurance to both us and RelEng as to the origins of changes.

Thus, when a principal contributor has a change to make, he lands the change on gitmo, and a mirroring script then pushes it to GitHub.

And when a non-principal contributor issues a pull request on GitHub, a principal contributor pulls it into his local repo for testing, then lands it on gitmo, whereupon the mirroring script pushes it to GitHub (which will mark the pull request closed, even though the push happened via the mirroring script).

We have no interest in a web interface to gitmo like Gitorious, as we intend to use GitHub as that interface.  And we can even set up the mirroring script ourselves to begin with, although eventually we'd like to get that onto hosted hardware.

Really, all we need to get started is a Git server hosted by Mozilla on which repositories can be created and to which SSH key-based commit privileges can be extended just as they are for our other hosted RCSes.

Comment 20

[Corey Shields [:cshields]]

Is there any reason you guys can think of that this git server would need to be accessible from the public internet (iow, requires vpn to push).  Pushing from there to github would still work as an outbound connection.

Comment 21

[Gervase Markham [:gerv]]

(In reply to comment #20)
> Is there any reason you guys can think of that this git server would need to be
> accessible from the public internet (iow, requires vpn to push). 

Because Mozilla projects have contributors outside MoCo, and (at least in Myk's model) our git repository is the master?

Why would you want to set it up as internal-only?

Gerv

Comment 22

[Myk Melez [:myk] [@mykmelez]]

What Gerv said!  In the Jetpack project, we want to use Git like we've previously used Hg and like other Mozilla projects have used Hg, SVN, and CVS: as the Mozilla-hosted RCS for our community-centric software development.

That means the RCS must be public, or principal contributors who aren't MoCo employees won't be able to push to it.  GitHub doesn't obviate the need to provide access to such contributors, it just enhances our ability to collaborate with the larger circle of peripheral contributors, i.e. folks who follow along and contribute a change here and there.

Comment 23

[Corey Shields [:cshields]]

(In reply to comment #21)
> (In reply to comment #20)
> > Is there any reason you guys can think of that this git server would need to be
> > accessible from the public internet (iow, requires vpn to push). 
> 
> Because Mozilla projects have contributors outside MoCo, and (at least in Myk's
> model) our git repository is the master?
> 
> Why would you want to set it up as internal-only?

Because you are asking for yet another production level VCS system when we already have to support cvs, svn, hg, and bzr.

git would make 5 to support from an IT perspective.

But if that is the request here, we will pass this on to infrasec for review.

Comment 24

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #23)
> Because you are asking for yet another production level VCS system when we
> already have to support cvs, svn, hg, and bzr.
> 
> git would make 5 to support from an IT perspective.

I can understand IT's reluctance to set up yet-another-VCS, but Git should be judged on its own merits.

Git is an increasingly popular tool for Mozilla development, not only because of GitHub, but also because the tool itself provides useful development and collaboration features.  And a number of key Mozilla projects already use Git repositories, including Jetpack and the new version of AMO.

So it's worth the added assurance and integrability that comes with hosting Git repositories at Mozilla.

(And if anything, a limit to the number of VCSes IT can support would be more an argument for IT to stop supporting Bazaar, SVN, and/or CVS, given that Git is already more popular and certainly more functional than those VCSes).


> But if that is the request here, we will pass this on to infrasec for review.

That is the request here, so please do pass this on, and let us know what they say!

Comment 25

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #19)
> Really, all we need to get started is a Git server hosted by Mozilla on which
> repositories can be created and to which SSH key-based commit privileges can be
> extended just as they are for our other hosted RCSes.

Erm, one correction: we would also need Gitweb, which comes bundled with Git, to be installed on the server and configured to enable zip and tgz snapshots, since the mozilla-central buildslaves use those to download Add-on SDK versions to test.

Comment 26

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #24)
> (In reply to comment #23)
> > But if that is the request here, we will pass this on to infrasec for review.
> 
> That is the request here, so please do pass this on, and let us know what they
> say!

Ping!  What did they say?

Comment 27

[Myk Melez [:myk] [@mykmelez]]

Corey: can you give us a status update here?  The quarter is almost 2/3rds over, and a number of folks would really really like to see this set up by the end of it.  Were you able to talk to infrasec, and is there anything else holding us back from making this happen?

Comment 28

[Corey Shields [:cshields]]

Sorry Myk, this got mis-assigned somehow.  Picking this back up.

Comment 29

[Myk Melez [:myk] [@mykmelez]]

Awesome!  Looking forward to making forward progress here, as this is only growing in importance over time.

Comment 30

[Corey Shields [:cshields]]

Just as an update, he and I have met and there has been backchannel discussions about this.  We understand the need and will make it as high of a priority as we can, but with the understanding that IT is short staffed right now and highly preempted.

Myk - no progress on this for this week - our SAN/Tinderbox/MXR issues have taken precedence over everything.  Oh, and we lost Jake on Tuesday when his wife had a baby (yay).

Comment 31

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

What makes Git really awesome for me is Github. I would not know how to effectively use Git without the help that the Github site provides. 

Are we considering their 'enterprise' thing?

This is described at http://fi.github.com/

It would allow us to run Github on our own infrastructure.

Comment 32

[(not currently active) Ted Mielczarek]

That is a totally orthogonal issue. If you'd like that, please file a separate bug.

Comment 33

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

(In reply to Ted Mielczarek [:ted, :luser] from comment #32)
> That is a totally orthogonal issue. If you'd like that, please file a
> separate bug.

I don't see how that is a different issue. This bug is about setting up a git repository with gitweb. I am simply suggesting using a different git implementation that gives us way more value. If we think that is a good idea then that would replace the idea of using plain git+gitweb.

I don't really see the point of a separate bug and I would rather discuss that here.

Comment 34

[Gregory Szorc [:gps]]

I too would like to vouch for gitosis. I have used it in a corporate environment and thought it was a breeze to manage (I actually managed it a bit myself).

Comment #7 indicated that gitosis may not work since it needs to work with LDAP. For that point:

1) can you elaborate on what type of LDAP interaction is required?
2) gitosis is open source, so it might be possible to integrate LDAP support

FWIW, gitosis works by reading a master ini-like config file. If we wanted LDAP to be the source of truth for repository permissions (I'm guessing this is what LDAP would be used for), it shouldn't be too difficult to write a script which generates the gitosis config file from LDAP data.

If someone would chime in with requirements, I'd be more than happy to contribute my time to get git.mozilla.org up and running.

Comment 35

[Corey Shields [:cshields]]

(In reply to Gregory Szorc [:gps] from comment #34)
> 1) can you elaborate on what type of LDAP interaction is required?
> 2) gitosis is open source, so it might be possible to integrate LDAP support

We manage source code access levels via ldap groups (translating to POSIX groups).  In addition, pushes are done via ssh keys that are acquired through ldap.

We would like git to match this format as much as possible so we do not have to manage multiple sets of vcs access controls.  That said, we are working on decoupling ldap from these systems for other reasons, yet the accounts, their keys, and their groups will still be extracted from ldap.

> If someone would chime in with requirements, I'd be more than happy to
> contribute my time to get git.mozilla.org up and running.

Ideally we will be writing a puppet module soon to manage all of this soon.  Preempted too much on fixing the other 4 VCS systems we have to actively maintain though.

thanks

Comment 37

[Brian Warner [:warner :bwarner]]

FYI: https://github.com/blog/978-introducing-github-enterprise

They specifically mention LDAP and installed as an OVF vm image. Pricing starts at $5k/year for 20 users (hrm, that feels steep).

Comment 38

[[e-mail] (:bhearsum)]

(In reply to Brian Warner [:warner :bwarner] from comment #37)
> FYI: https://github.com/blog/978-introducing-github-enterprise
> 
> They specifically mention LDAP and installed as an OVF vm image. Pricing
> starts at $5k/year for 20 users (hrm, that feels steep).

I'm not really sure how to interpret this FAQ item:

Can I use GitHub Enterprise to host projects on the public internet?

    No. GitHub Enterprise is designed for collaboration within an organization only. Externally accessible installations must have the Private Mode option enabled. Note that github.com offers free hosting of unlimited public repositories.

Comment 39

[Brian Warner [:warner :bwarner]]

Nevermind, that product doesn't support publically-visible repositories (it's really intended for closed-source projects). Not so interesting.

Comment 40

[Jeff Muizelaar [:jrmuizel]]

Is there an eta for getting this bug fixed?

Comment 41

[Corey Shields [:cshields]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #40)
> Is there an eta for getting this bug fixed?

There was, but we've missed it thanks to other preemption, so I'm hesitant to make another date.  There have been a lot of hg requests and issues lately taking our time.

Status right now is that we've got our puppetized service mostly working..  Finishing up gitweb components right now, and will also need to finish creating a stage environment for all of this to test changes in.

Comment 42

[Ralph Giles (:rillian)]

Just a drive-bye, but the description recommends gitosis for shared repository management. Since then, Sitaram Chamarty has written Gitolite, which is generally considered better, and is actively maintained.

Comment 43

[Brian Warner [:warner :bwarner]]

tri-yearly ping.. any updates?

Comment 44

[Corey Shields [:cshields]]

(In reply to Brian Warner [:warner :bwarner] from comment #43)
> tri-yearly ping.. any updates?

yes, we've passed opsec review and are nearing production once a couple of tweaks are made and releng is ready to support git.

Comment 45

[Shyam Mani [:fox2mike]]

A short status update, this is still on track to be finished before Q2 is out.

Most of the setup is complete and releng is doing testing on repos etc. From the IT side, we have some issues with the http clone bits of things but will work on fixing it.

Overall, on track.

Comment 46

[Ben Kero [:bkero]]

This has been set up, and can be deployed as needed