Closed Bug 388332 Opened 18 years ago Closed 18 years ago

Allow Hg committers the ability to clone new trees on hg.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: benjamin, Assigned: aravind)

References

Details

Attachments

(1 file)

create_new.pl
2.66 KB, text/plain
Details
In the next few months, a lot of Mozilla groups are going to want to start doing Mozilla2 experiments using feature branches. Rather than having each team file a bug for each new Hg clone, we should provide a way for users to fill in a simple webform (or issue an ssh command?) to make a clone. E.g. "ssh hg.mozilla.org hg-clone /mozilla-central /projects/2007-xptcall-tamarin" For the moment, I think these project trees should allow everyone with hg access to commit to them: fine-grained permission control shouldn't be necessary.
Severity: normal → enhancement
OS: Linux → All
Hardware: PC → All
I am okay with setting something like this on our Hg servers. However, we will need to figure out the specifics, like how much space is this expected to take. How much is this expected to grow. I will try to get a script together that will automate this.
Assignee: server-ops → aravind
Can we do this with just keeping multiple branches in a single repo? I have my local tree with a bunch of different heads (I haven't even been using the named branch support), and it works fine; using named branches it would be even easier. See http://blog.medallia.com/2007/02/ Otherwise everyone will need a duplicate of the base hg repo (need to see if it can use hardlinks for cloning as well), and people will end up with multiple heads/branches anyway in their local repos that they'll have to be careful to push correctly to avoid mixing the two upstream. (Unless they keep the repos entirely separate, which would require disk space on the user system, as well..)
From the dup, see http://db48x.net/.hglandfill/create_new.html. go ahead and try it out, it works.
Blocks: 389875
btw, hard links do make the repositories pretty cheap. 2.2M db48x.net/.hglandfill/repos/another_test 2.2M db48x.net/.hglandfill/repos/tamarinwithgewgaws 9.9M db48x.net/.hglandfill/repos/bobhope 48.4M repos/actionmonkey-tamarin 393.0M repos/mozilla-central 393.2M repos/actionmonkey 394.3M repos/2007-configure-rewrite 403.1M repos/cvs-trunk-mirror 1.61G total The first two are clones of actionmonkey-tamarin, the third is a clone of mozilla-central.
(In reply to comment #4) > From the dup, see http://db48x.net/.hglandfill/create_new.html. go ahead and > try it out, it works. Urk, are you really asking people to type a password into a form over http? :p I'm not sure what this form actually does -- is this just on a test machine and not on hg.mozilla.org?
(In reply to comment #6) > I'm not sure what this form actually does -- is this just on a test machine and > not on hg.mozilla.org? It's definitely not on hg.mozilla.org. Also, notice missing '>' at '<form action="create_new.pl" method="POST"'.
Yea, it's on my own server, and yes, it should be over ssl. It's just for testing, to see how things work, not for anything serious.
(In reply to comment #6) > I'm not sure what this form actually does As for that, it clones an Hg repository, and sets the username and password up so that you can push to it. Basically, it does what this bug requests, though it's not perfect. In the other bug I mentioned that pushing doesn't work, but I've since fixed that using a different configuration. You can see a list of available repositories at http://db48x.net/.hglandfill/, and you can see some new changes that I've pushed at http://db48x.net/.hglandfill/tamarinwithgewgaws.
Attached file create_new.pl —
the current script. could use some work, and I should have forced myself to write it in python.
Attachment #274314 - Attachment mime type: application/x-perl → text/plain
It's gotten more complicated than just a single script, so see http://db48x.net/hg/hg-for-landfill/
Daniel, I don't think the user/password behavior is what we want. Cloned project trees should be pushable by anyone with HG commit access. The clone form itself should also be protected by LDAP auth so that random non-committers cannot use our HG server.
Fair enough, I suppose. My thought had been that there are plenty of contributors that don't have CVS/Hg write access and yet might wish to avail themselves of this service. Perhaps it could be tied to bugzilla usernames and passwords?
I do not think we want to open up that can of worms at this time.
(In reply to comment #2) > Can we do this with just keeping multiple branches in a single repo? So, did we decide that we don't want to create branches to do this stuff? Are we going ahead with creating clones on the server side?
Clones are really the easiest to work with.
I keep seeing clone action, but IIRC vlad favored branches. Vlad, you there? /be
I was concerned about branches for several reasons: 1) branch support wasn't added until 0.9.3, which is not in many distros. 2) I don't understand whether branches in mozilla-central affect the default pull size... if they do, I worry about that 3) I don't know how branches interact with important Hg tools such as mq and transplant It is easy to retrofit branches onto mozilla-central later, if it turns out to be a good idea.
branches don't really matter for mq/transplant, they're just named heads, basically. But sure, I really have no preference here -- just wanted to make sure that we explored all the options. For 0.9.3 though, I think people may have to manually upgrade their hg for a while until it hits 1.0 -- e.g., 0.9.4 is out, and it has a number of bugfixes for win32 especially. We shouldn't rely on "what the distro gives me" -- there are enough contributed packages that it shouldn't be that big of a deal (and if there aren't, it's trivial to build on anything other than win32, for which there is an installer).
I would rather not open this up to the public through an automated system. For now, my strong inclination is to have folks that require repos on the server file bugs. I will still create shell scripts and such on the server side to automate the process for sysadmins, but it will not be available for regular users. My reason for restricting this is to prevent people from creating random repositories that wont really be used.
(In reply to comment #20) > I would rather not open this up to the public through an automated system. For > now, my strong inclination is to have folks that require repos on the server > file bugs. I will still create shell scripts and such on the server side to > automate the process for sysadmins, but it will not be available for regular > users. My reason for restricting this is to prevent people from creating > random repositories that wont really be used. > I'd like you to change your mind. The cost on the server of a fallow clone is minimal: a few megs of disk space (an unused clone of mozilla-central is 9.9 megs). Once it's created, there are essentially no ongoing costs. If disk space starts to get scarce, simply moving it bodily to a larger disk is all that is required. Even the index page of Hg's web interface can (and should) be replaced with a static version that doesn't need to be regenerated on each request. The benefit of being able to create new repositories without having to file a bug and wait an hour or a week for someone to run the script seems obvious. Two people talking on irc can decide to collaborate on some bug, create a repository for that work, and immediately begin pushing changes out to it. If you make them file a bug in order to ask a third party for approval, in all likelihood the moment will be lost. Their ardor will have cooled.
I agree with Daniel -- in addition, there are lots of repos that don't necessarily need a mozilla clone; e.g., I have a repo that i'm hosting myself with a (non-copyright-encumbered) pageset for testing, and other such smaller projects may want to create their own. The cost of an additional repo isn't very high, and it's trivial to remove them if someone creates something inappropriate.
If the concern is over cluttering up the hg.mozilla.org namespace, I think that ship may already have sailed. There are enough repositories on hg.m.o today that you can't just look at them and tell what they are, who's using them, and whether they're active. I think that's just how hg encourages you to work. Many of the developers who work on Mercurial itself have their own public repositories.
Yup, my concern was cluttering the hg.m.o namespace - and yeah looks like that point is moot now. But I'd still like you guys to re-consider. Maintainability becomes a hassle - on the server side that is. I realize that as the admin, hg works differently and encourages folks to create ad-hoc repos etc. Maybe hosting them on hg.m.o is not the right solution here? Or maybe creating a to level directory, like hg.m.o/adhoc/blah is better than hg.m.o/blah. I am trying to enforce some structure w.r.t whats created and used. I guess I am envisioning that the top level hg.m.o would look like cvs, and would like to hide all the adhoc stuff a level deeper. Ultimately, its the developers that will have to keep all this straight, so I am okay with allowing you guys to create ad-hoc top-level repos if you really, really want that :)
I think that ad-hoc repos should go in the projects/ subfolder. The only trees that should end up in the toplevel are the "central" trees that are of general use.
Also, take a look at http://db48x.net/.hglandfill/ again. About a week ago I added a way to categorize the repositories, so that things like mozilla-central and tamarin-central that are actually going to result in shipping products can be in one category, while everything else can be listed elsewhere. It works by simply adding "category = whatever" to each repository's hgrc file. Between that and putting newly created repositories in a subdirectory there shouldn't be any confusion.
Okay, so here is what I think we should do. Any ad-hoc user created stuff will go into hg.mozilla.org/users/user_domain.com/blah To create any top level repos, you will need to file an IT bug. Any user created repos will NOT be browsable in http://hg.mozilla.org. Folks will still be able to checkin, clone, etc into these repositories. They just won't be visible in http://hg.m.o. This involves editing files in the apache docroot, etc and I would rather not do it right now. If this is okay with folks, I will script away.
Assuming that anyone can push to the cloned repository, that sounds like a great plan.
I don't understand why they won't be browsable online. If you're using the [collections] section in the config file, it'll recurse down through any sub directories to find repositories to list. Also, while you're at it, could you flip the style to gitweb? Just put style = gitweb in /etc/hgrc or ~/.hgrc so that if affects the index as well.
(In reply to comment #29) > I don't understand why they won't be browsable online. If you're using the > [collections] section in the config file, it'll recurse down through any sub > directories to find repositories to list. The collections idea won't work for us, because our repo is not hosted off the root. Its hosted off /repo/hg/mozilla and I use some shell scripting hacks to make your ssh operations work seamlessly. There are probably other tweaks to it, but using the default collections puts the entire path as the repo path and our hg setup wont work with that. > Also, while you're at it, could you flip the style to gitweb? Just put style = > gitweb in /etc/hgrc or ~/.hgrc so that if affects the index as well. Done.
(In reply to comment #30) > (In reply to comment #29) > > Also, while you're at it, could you flip the style to gitweb? Just put style = > > gitweb in /etc/hgrc or ~/.hgrc so that if affects the index as well. > > Done. Sweet, I like it! Aravind changed the time format from being "x days/weeks/... ago" to be the timestamp on the checkin, which is more bonsai like, and more importantly, more granual and thus easier to find regression candidates etc. Thanks aravind!
I now have a scripted solution ready for this. Please try it out and let me know if you think I should change anything. To create a clone of any repository on the hg server, ssh to hg.mozilla.org with your hg account name (e-mail address) and run the "clone newrepo" command. You will need to run this command as a remote command and not as a shell command. What I mean by this is that you will not be able to get a shell on the hg server to run your command, instead you will have to run it as a part of your ssh statement. so in my case, the invocation will look like this. ssh hg.mozilla.org -l 'aravind@mozilla.com' clone myrepo1 myrepo1 in the name of the NEW repo that I want to create in /users/aravind_mozilla.com/myrepo1 The clone command above will prompt you through the necessary steps. I am including a sample run below. <pre> ~ $ssh hg.mozilla.org -l 'aravind@mozilla.com' clone myrepo1 Making repo myrepo1 for aravind@mozilla.com This repo will appear as hg.mozilla.org/users/aravind_mozilla.com/myrepo1 If you need a top level repo, please quit now and file a bug for IT to create one for you 1) yes 2) no Continue? 1 Do you want to clone an existing public repo or a users private repo? 1) Public 2) Private Public or Private? 1 Select the repo you want to clone 1) cvs-trunk-mirror 12) libqual 2) actionmonkey 13) libregion 3) mozilla-central 14) oink 4) cvs-trunk-mirror.old 15) oink-scripts 5) test/cvs-trunk-mirror 16) oink-stack 6) test/mozilla-central 17) platform-model 7) tamarin-central 18) smbase 8) actionmonkey-tamarin 19) stack-scripts 9) ast 20) projects/2007-configure-rewrite 10) elkhound 21) pork-barrel 11) elsa 22) None of the above Pick a source repo:12 About to clone /libqual to /users/aravind_mozilla.com/myrepo1 1) yes 2) no Proceed? 1 Please do not interrupt this operation. 55 files updated, 0 files merged, 0 files removed, 0 files unresolved Clone complete. Fixing permissions Done ~ $ </pre> I admit this is not quite web 2.0 and a bit old fashioned, but it should do the job nicely. I will close this bug out once you guys get a chance to try it and let me know if this fits your needs.
After a few discussions yesterday, I added an option to create an empty private repository. It will show up as one of the menu options. Let me know if you guys want me to change the way it works.
Marking this fixed. Re-open if necessary.
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Should this be documented somewhere beneath http://wiki.mozilla.org/WorkingWithMercurial?
I followed the instructions to create a repository here, but I'm getting a 404 on it. Does the script give out the wrong link? "This repo will appear as hg.mozilla.org/users/bhearsum_mozilla.com/my-moz-central". http://hg.mozilla.org/users/bhearsum_mozilla.com/my-moz-central is a 404.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
07:56 < Pike> bhearsum: that 404 is expected, it's a secret clone. You can hg clone it, but there's no hg serve for it
Status: REOPENED → RESOLVED
Closed: 18 years ago → 18 years ago
Resolution: --- → FIXED
I created a page on devmo documenting these steps: http://developer.mozilla.org/en/docs/Publishing_Mercurial_Clones
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: