Closed Bug 528360 Opened 11 years ago Closed 8 years ago

Set up a git server

Categories

(Developer Services :: General, task, P2)

All
Other

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: clouserw, Assigned: bkero)

References

Details

(Whiteboard: [2012q2])

We talked about this in the meeting today with IT and it didn't sound like it would be much work.  The AMO team is interested in trying out git for Zamboni (amo:v4).

jbalogh recommends using http://github.com/res0nat0r/gitosis for permissions and management but that's up to y'all.
Assignee: server-ops → mrz
Going to roll this into the whole hudson CI stuff.
Assignee: mrz → shyam
Assignee: shyam → jeremy.orem+bugs
As a side note, we'll probably be tracking Bouncer in git soon as well. I am merging the current trunk and production code into my git repository at http://github.com/fwenzel/tuxedo/ , but once we have a production git server at Mozilla, we can use that as the reference copy of the code.
If we don't want to use the ugliness that is gitweb, gitorious (http://gitorious.org/gitorious) seems to be a nice little open source alternative that provides some of the features github does. Maybe we want to use that so that our git.m.o upstream repositories aren't too sad-looking.
(In reply to comment #3)
> Maybe we want to use that so that our git.m.o upstream repositories aren't too sad-looking.

I think an ugly gitmo is good, it'll push people back to github. ;)
Aravind, will you create that netapp mount we talked about yesterday. Please assign back to me when it is ready.
Assignee: jeremy.orem+bugs → aravind
Assignee: aravind → jeremy.orem+bugs
Component: Server Operations → Server Operations: Projects
Priority: -- → P2
Whiteboard: [2010q2]
More Gitosis docs: http://progit.org/book/ch4-7.html

http://progit.org/book/ has a whole chapter of "Git on the Server."
Unfortunately, I don't think gitosis will work since it needs to work with LDAP. Aravind already wrote a bunch of scripts for Mercurial, so I should be able to mostly copy those.
FWIW, this would help IT stay independent on github, especially now that we set up vendor repositories per application (and thus shifted our dependence from pypi to github).
Duplicate of this bug: 618934
Can somebody in IT give us an update on this?
Summary: Setup a git repository → Set up a git repository
I haven't looked at this, because webdev has been happy with github. I think Aravind could get this up and running the fastest since he's done a bunch of work on mercurial.
Assignee: jeremy.orem+bugs → aravind
Summary: Set up a git repository → Setup a git repository
Github has been in pain this morning, underscoring that we shouldn't have dependencies on external resources. This would, to use the vernacular, "close the tree".

I have an instance running in labs, but this really should be on HA hardware with good backups.
I'm guessing this should be assigned to somebody other than aravind.
Assignee: aravind → nobody
Pete, I think we talked about this last week and you had mentioned it is something you could tackle? This might be a dupe though.
Assignee: nobody → petef
Component: Server Operations: Projects → Server Operations
Alternatively, we can mirror github repos automatically whenever there's an update (bug 649483).
cshields, you mentioned you guys have a machine somewhere I can run a local copy of gitorious on. Can you point me at that machine, and what kind of storage we should be using?
(In reply to comment #15)
> Alternatively, we can mirror github repos automatically whenever there's an
> update (bug 649483).

I don't think that's an alternative. In fact, I am quite sure this is exactly what's intended.
This bug might be switching to what jabba is talking about comment 14, which is a git server for internal git repos that aren't hosted on github.

I'm not sure if we should combine the internal git repos and the git mirror from bug 649483. I don't have a strong opinion either way, but prefer internal repos go on a completely separate server i.e., this bug.
For the Jetpack project, we'd like a Mozilla-maintained public Git server like git.mozilla.org that serves as the canonical host for the Add-on SDK and Add-on Builder repositories.

We'll continue to use GitHub as a hub for collaboration, but our mirroring should be from gitmo to GitHub, so the only way to land a change into one of the canonical repos is to check it into that repo using a Mozilla-authorized account, which provides greater assurance to both us and RelEng as to the origins of changes.

Thus, when a principal contributor has a change to make, he lands the change on gitmo, and a mirroring script then pushes it to GitHub.

And when a non-principal contributor issues a pull request on GitHub, a principal contributor pulls it into his local repo for testing, then lands it on gitmo, whereupon the mirroring script pushes it to GitHub (which will mark the pull request closed, even though the push happened via the mirroring script).

We have no interest in a web interface to gitmo like Gitorious, as we intend to use GitHub as that interface.  And we can even set up the mirroring script ourselves to begin with, although eventually we'd like to get that onto hosted hardware.

Really, all we need to get started is a Git server hosted by Mozilla on which repositories can be created and to which SSH key-based commit privileges can be extended just as they are for our other hosted RCSes.
Is there any reason you guys can think of that this git server would need to be accessible from the public internet (iow, requires vpn to push).  Pushing from there to github would still work as an outbound connection.
(In reply to comment #20)
> Is there any reason you guys can think of that this git server would need to be
> accessible from the public internet (iow, requires vpn to push). 

Because Mozilla projects have contributors outside MoCo, and (at least in Myk's model) our git repository is the master?

Why would you want to set it up as internal-only?

Gerv
What Gerv said!  In the Jetpack project, we want to use Git like we've previously used Hg and like other Mozilla projects have used Hg, SVN, and CVS: as the Mozilla-hosted RCS for our community-centric software development.

That means the RCS must be public, or principal contributors who aren't MoCo employees won't be able to push to it.  GitHub doesn't obviate the need to provide access to such contributors, it just enhances our ability to collaborate with the larger circle of peripheral contributors, i.e. folks who follow along and contribute a change here and there.
(In reply to comment #21)
> (In reply to comment #20)
> > Is there any reason you guys can think of that this git server would need to be
> > accessible from the public internet (iow, requires vpn to push). 
> 
> Because Mozilla projects have contributors outside MoCo, and (at least in Myk's
> model) our git repository is the master?
> 
> Why would you want to set it up as internal-only?

Because you are asking for yet another production level VCS system when we already have to support cvs, svn, hg, and bzr.

git would make 5 to support from an IT perspective.

But if that is the request here, we will pass this on to infrasec for review.
(In reply to comment #23)
> Because you are asking for yet another production level VCS system when we
> already have to support cvs, svn, hg, and bzr.
> 
> git would make 5 to support from an IT perspective.

I can understand IT's reluctance to set up yet-another-VCS, but Git should be judged on its own merits.

Git is an increasingly popular tool for Mozilla development, not only because of GitHub, but also because the tool itself provides useful development and collaboration features.  And a number of key Mozilla projects already use Git repositories, including Jetpack and the new version of AMO.

So it's worth the added assurance and integrability that comes with hosting Git repositories at Mozilla.

(And if anything, a limit to the number of VCSes IT can support would be more an argument for IT to stop supporting Bazaar, SVN, and/or CVS, given that Git is already more popular and certainly more functional than those VCSes).


> But if that is the request here, we will pass this on to infrasec for review.

That is the request here, so please do pass this on, and let us know what they say!
(In reply to comment #19)
> Really, all we need to get started is a Git server hosted by Mozilla on which
> repositories can be created and to which SSH key-based commit privileges can be
> extended just as they are for our other hosted RCSes.

Erm, one correction: we would also need Gitweb, which comes bundled with Git, to be installed on the server and configured to enable zip and tgz snapshots, since the mozilla-central buildslaves use those to download Add-on SDK versions to test.
(In reply to comment #24)
> (In reply to comment #23)
> > But if that is the request here, we will pass this on to infrasec for review.
> 
> That is the request here, so please do pass this on, and let us know what they
> say!

Ping!  What did they say?
Corey: can you give us a status update here?  The quarter is almost 2/3rds over, and a number of folks would really really like to see this set up by the end of it.  Were you able to talk to infrasec, and is there anything else holding us back from making this happen?
Sorry Myk, this got mis-assigned somehow.  Picking this back up.
Assignee: petef → cshields
Whiteboard: [2010q2]
Awesome!  Looking forward to making forward progress here, as this is only growing in importance over time.
Summary: Setup a git repository → Set up a git repository
Just as an update, he and I have met and there has been backchannel discussions about this.  We understand the need and will make it as high of a priority as we can, but with the understanding that IT is short staffed right now and highly preempted.

Myk - no progress on this for this week - our SAN/Tinderbox/MXR issues have taken precedence over everything.  Oh, and we lost Jake on Tuesday when his wife had a baby (yay).
What makes Git really awesome for me is Github. I would not know how to effectively use Git without the help that the Github site provides. 

Are we considering their 'enterprise' thing?

This is described at http://fi.github.com/

It would allow us to run Github on our own infrastructure.
That is a totally orthogonal issue. If you'd like that, please file a separate bug.
(In reply to Ted Mielczarek [:ted, :luser] from comment #32)
> That is a totally orthogonal issue. If you'd like that, please file a
> separate bug.

I don't see how that is a different issue. This bug is about setting up a git repository with gitweb. I am simply suggesting using a different git implementation that gives us way more value. If we think that is a good idea then that would replace the idea of using plain git+gitweb.

I don't really see the point of a separate bug and I would rather discuss that here.
I too would like to vouch for gitosis. I have used it in a corporate environment and thought it was a breeze to manage (I actually managed it a bit myself).

Comment #7 indicated that gitosis may not work since it needs to work with LDAP. For that point:

1) can you elaborate on what type of LDAP interaction is required?
2) gitosis is open source, so it might be possible to integrate LDAP support

FWIW, gitosis works by reading a master ini-like config file. If we wanted LDAP to be the source of truth for repository permissions (I'm guessing this is what LDAP would be used for), it shouldn't be too difficult to write a script which generates the gitosis config file from LDAP data.

If someone would chime in with requirements, I'd be more than happy to contribute my time to get git.mozilla.org up and running.
(In reply to Gregory Szorc [:gps] from comment #34)
> 1) can you elaborate on what type of LDAP interaction is required?
> 2) gitosis is open source, so it might be possible to integrate LDAP support

We manage source code access levels via ldap groups (translating to POSIX groups).  In addition, pushes are done via ssh keys that are acquired through ldap.

We would like git to match this format as much as possible so we do not have to manage multiple sets of vcs access controls.  That said, we are working on decoupling ldap from these systems for other reasons, yet the accounts, their keys, and their groups will still be extracted from ldap.

> If someone would chime in with requirements, I'd be more than happy to
> contribute my time to get git.mozilla.org up and running.

Ideally we will be writing a puppet module soon to manage all of this soon.  Preempted too much on fixing the other 4 VCS systems we have to actively maintain though.

thanks
Duplicate of this bug: 690306
FYI: https://github.com/blog/978-introducing-github-enterprise

They specifically mention LDAP and installed as an OVF vm image. Pricing starts at $5k/year for 20 users (hrm, that feels steep).
(In reply to Brian Warner [:warner :bwarner] from comment #37)
> FYI: https://github.com/blog/978-introducing-github-enterprise
> 
> They specifically mention LDAP and installed as an OVF vm image. Pricing
> starts at $5k/year for 20 users (hrm, that feels steep).

I'm not really sure how to interpret this FAQ item:

Can I use GitHub Enterprise to host projects on the public internet?

    No. GitHub Enterprise is designed for collaboration within an organization only. Externally accessible installations must have the Private Mode option enabled. Note that github.com offers free hosting of unlimited public repositories.
Nevermind, that product doesn't support publically-visible repositories (it's really intended for closed-source projects). Not so interesting.
Is there an eta for getting this bug fixed?
(In reply to Jeff Muizelaar [:jrmuizel] from comment #40)
> Is there an eta for getting this bug fixed?

There was, but we've missed it thanks to other preemption, so I'm hesitant to make another date.  There have been a lot of hg requests and issues lately taking our time.

Status right now is that we've got our puppetized service mostly working..  Finishing up gitweb components right now, and will also need to finish creating a stage environment for all of this to test changes in.
Just a drive-bye, but the description recommends gitosis for shared repository management. Since then, Sitaram Chamarty has written Gitolite, which is generally considered better, and is actively maintained.
tri-yearly ping.. any updates?
Blocks: 743626
Blocks: 713782
No longer depends on: 713782
Summary: Set up a git repository → Set up a git server
No longer blocks: 713782
Whiteboard: [2012q2]
(In reply to Brian Warner [:warner :bwarner] from comment #43)
> tri-yearly ping.. any updates?

yes, we've passed opsec review and are nearing production once a couple of tweaks are made and releng is ready to support git.
Assignee: cshields → bkero
Component: Server Operations → Server Operations: Developer Services
QA Contact: mrz → shyam
Blocks: 758003
A short status update, this is still on track to be finished before Q2 is out.

Most of the setup is complete and releng is doing testing on repos etc. From the IT side, we have some issues with the http clone bits of things but will work on fixing it.

Overall, on track.
This has been set up, and can be deployed as needed
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.