626641 - better interface to hg.m.o user repos

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We make use of user repositories in doing test releases regularly. In order to do a valid test, we need copies of roughly 75 repositories in stage-ffxbld{,1,2,3}'s user section. Currently, we set this up through:
for r in repos; do
ssh hg.mozilla.org edit $r delete YES
done
sleep 600
for r in repos; do
ssh hg.mozilla.org clone $r $prefix$r
done
sleep 600

Two issues have prevented us from making this work better:
1) hgweb is not updated instantly, so it's impossible to be sure that a repository really exists or doesn't exist.
2) We can't reclone directly after deleting, we have to wait or sometimes we'll be told that the repository still exists.

What would be better for us, is to have an interface to user repository management that met the following requirements:
- Be able to test for repository existence accurately. Doesn't have to be through HTTP.
- Able to recreate a repository simply. Preferably, done through one command, eg, "ssh hg.mozilla.org reclone myrepo sourcerepo". That command would delete "myrepo" if exists, and then clone it from "sourcerepo"

Speeding up the time it takes to delete/clone a user repo would be really helpful, too.

Comment 1

[Chris AtLee [:catlee]]

> Speeding up the time it takes to delete/clone a user repo would be really
> helpful, too.

This is almost critical.  Re-cloning all these repos takes an hour, and recloning mozilla-central in particular takes > 20 minutes, if it doesn't time out.

Comment 2

[Noah Meyerhans [:noahm]]

Hey guys. This ticket was just assigned to me. I just want to let you know that hg performance is going to get a bunch of attention coming up. Yours isn't the only ticket open about it. No ETA yet, but it'll be a high priority for me...

Comment 3

[Noah Meyerhans [:noahm]]

(In reply to comment #0)
> 1) hgweb is not updated instantly, so it's impossible to be sure that a
> repository really exists or doesn't exist.

I'm testing a fix for Bug 665013, a request to clear the hgweb cache when a repo is deleted.  I'll also verify that creating a new repo updates the cache.

> 2) We can't reclone directly after deleting, we have to wait or sometimes
> we'll be told that the repository still exists.

Because deletion of a repository uses a call to 'rm -rf', which exits successfully even if the given path isn't deleting, it's possible for a deletion to silently fail.  I've seen at least one case where a repository is left with NFS artifacts (.nfsBLAHBLAHBLAH files).  The server-side hg wrapper should probably try a second time to delete a repo if it still exists after a deletion request, and exit unsuccessfully if the repo still exists after that.

Comment 4

[Noah Meyerhans [:noahm]]

hg will now clear the cache upon deletion of repositories and creation of new ones (via clones of existing repos).

After deleting a repository, you should be able to verify that it's gone by checking the HTTP status:

foo:~/src/svn/source_control$ HEAD http://hg.mozilla.org/users/nmeyerhans_mozilla.com/hghooks
200 Script output follows
Connection: close
Date: Wed, 22 Jun 2011 20:57:40 GMT
Via: 1.1 varnish
Age: 0
Server: Apache/2.2.3 (Red Hat)
Content-Length: 13127
Content-Type: text/html; charset=UTF-8
Client-Date: Wed, 22 Jun 2011 20:57:40 GMT
Client-Peer: 63.245.208.189:80
Client-Response-Num: 1
X-Varnish: 1080559740

foo:~/src/svn/source_control$ ssh [e-mail]@hg.mozilla.org edit hghooks delete YES
Deleting...
Finished deleting
Purging [/users/nmeyerhans_mozilla.com] from varnish cache
EXEC [/usr/bin/varnishadm -S /etc/varnish/secret -T dm-vcview04:6082 purge.url /users/nmeyerhans_mozilla.com]

Breaking after reading 0 lines from subprocess
foo:~/src/svn/source_control$ HEAD http://hg.mozilla.org/users/nmeyerhans_mozilla.com/hghooks
404 Not Found
Connection: close
Date: Wed, 22 Jun 2011 20:57:56 GMT
Via: 1.1 varnish
Age: 0
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1379
Content-Type: text/html; charset=UTF-8
Client-Date: Wed, 22 Jun 2011 20:57:56 GMT
Client-Peer: 63.245.208.189:80
Client-Response-Num: 1
X-Varnish: 1080559908

Comment 5

[Noah Meyerhans [:noahm]]

Closing this ticket. The immediate issues (items 1 and 2 from the original report) have been addressed.  The general performance concerns are being tracked elsewhere.