640439 - hg server's read-only file-system prevents mercurial caches from being persisted, making some requests extremely expensive

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Andrew Sutherland [:asuth] (he/him)]

This is a spin-off of Bug 635765 which was about queries for the pushlog taking many seconds (ex: 6) for comm-central.  While a small part of the problem may have been the SQLite schema, it appears that a greater part of the problem is that the pushlog populates the set of tags used for each revision and that this is very expensive to compute without caching.

Comments in the thread revealed that a read-only filesystem is used for security purposes and that this is likely the cause of the problem.  Hope is given because mercurial 1.8, which is now released, should allow a symlink or something so the caches could potentially be directed at a writable directory tree.

Excerpted below, but see Bug 635765 for the entire context:

(In reply to comment #17)
> (In reply to comment #16)
> > On the other hand, if
> > everything is configured correctly, there are on-disk caches that should help.
> > These only get written if the permissions are set correctly, but I think we
> > already went through setting that up right at some point. Aravind, can you
> > check the mtimes on mozilla-central/.hg/*.cache?
> 
> So.. this is probably why our queries take longer - they are served out of a
> nfs read-only mount point on the vcview* servers.  This was the outcome of a
> security bug to separate out the read and the read/write parts of the
> application.
> 
> The *.cache directories correspondingly have not been updated in a while.
> 
> -rw-rw-r-- Jul  9  2009 branch.cache
> -rw-rw-r-- Feb 23 16:28 branchheads.cache
> 
> Is it possible to override the locations of these cache directories into some
> sort of a (different) shared read-write location?

(In reply to comment #18)
> No. In the upcoming Mercurial 1.8 (to be released on March 1 if everything goes
> according to plan), there's a separate dir .hg/cache where these are saved, but
> I'm not sure if that will help this use case?

Comment 1

[matthew zeier [:mrz]]

No time to tackle this under post-Fx4.  It'll be related to bug 567488 and part of the "move VCS".

Comment 2

[Ben Bucksch (:BenB)]

mrz, the whole idea of bug 567488 was to separate the web frontends (where you do pushlog etc.) from the write access and repo.

Comment 3

[matthew zeier [:mrz]]

Yes I know, I'm saying it's all part of the "move this to Phoenix" where the whole thing will be rebuilt.

Comment 4

[Noah Meyerhans [:noahm]]

Quick update on this.  Bug 661882 is tracking work on getting hg mirrors for the build system installed on the build network.  These mirrors will use local disk, and will thus not have this problem.  That will benefit releng side of things greatly.  It'll also help us work on a fix for the main hg infrastructure by taking quite a bit of load off the current machines.

Unfortunately, a stright upgrade of mercurial to 1.8 doesn't appear to be entirely trivial.  There I haven't had time to investigate the problems fully, but some of our customizations cause 1.8 to crash.

Comment 5

[Dirkjan Ochtman (:djc)]

Would it be possible to give me accesss to bug 661882? And do we have a separate bug for the upgrade? That usually requires some work for the pushlog stuff.

We should also contemplate upgrading to 1.9 instead, which enables much faster push/pull (if 1.9+ is also used on the client).

Comment 6

[Noah Meyerhans [:noahm]]

(In reply to comment #5)
> Would it be possible to give me accesss to bug 661882? And do we have a
> separate bug for the upgrade? That usually requires some work for the
> pushlog stuff.

I've added you to the CC list for bug 661882, so you should be able to see it.  There is not currently a bug for an hg upgrade, but that will come once the mirrors are in place.

> We should also contemplate upgrading to 1.9 instead, which enables much
> faster push/pull (if 1.9+ is also used on the client).

I'm fine with going straight to 1.9.

Comment 7

[Jake Maul [:jakem]]

Setting aside the Hg 1.8/1.9 upgrade stuff (which rightly belongs in a separate bug anyway), this appears to have been taken care of at some point... either that, or things have changed such that these files are properly writable now.

[root@dm-svn02 .hg]# pwd
/repo/hg/mozilla/comm-central/.hg
[root@dm-svn02 .hg]# ls -al *.cache
-rw-rw-r-- 1 hg                          scm_level_3  290 Jul  9  2009 branch.cache
-rw-rw-r-- 1 [e-mail] scm_level_3 1825 Oct 13 10:29 branchheads.cache
-rw-rw-r-- 1 [e-mail]   scm_level_3 7304 Jun 20 07:23 tags.cache

[root@dm-svn02 .hg]# pwd
/repo/hg/mozilla/mozilla-central/.hg
[root@dm-svn02 .hg]# ls -al *.cache
lrwxrwxrwx 1 root                  scm_level_3    33 Apr 20 10:59 branch.cache -> /tmp/mozilla-central-branch.cache
-rwxrwxrwx 1 [e-mail] scm_level_3  2725 Oct 13 12:12 branchheads.cache
-rw-rw-r-- 1 [e-mail]     scm_level_3 11944 Oct 13 06:47 tags.cache

I don't know why mozilla-central/.hg/branch.cache is a symlink (and a broken one at that... the destination file doesn't exist), but it seems that's not regularly updated, so it could just be not used much. This file in comm-central hasn't been touched since 2009. The other cache files are obviously being updated.

Can we close this bug out?

Comment 8

[Phong Tran [:phong]]

reopen if you still need this.