557325 - reset try repo

Status: RESOLVED DUPLICATE of bug 561699

(Release Engineering :: General, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

time wget http://hg.mozilla.org/try/rev/1156304c2b12
--16:55:36--  http://hg.mozilla.org/try/rev/1156304c2b12
           => `1156304c2b12.1'
Resolving hg.mozilla.org... 63.245.208.188, 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.188|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: unspecified [text/html]

    [                                <=>  ] 7,196,033      1.18M/s             

16:56:41 (1003.13 KB/s) - `1156304c2b12.1' saved [7196033]


real	1m4.825s
user	0m0.042s
sys	0m0.234s

Most of the time is spent waiting for the transfer to begin.

Comment 1

[Aravind Gottipati [:aravind]]

its not hg.m.o thats the problem here.  The try repo usually goes crazy after folks have been using it for some time.  The rest of hg.m.o is plenty speedy for me.

The fix here is to trim the try repo.  That has to come from rel eng. after they announce it to the developers etc.

Comment 2

[Armen [:armenzg]]

John when could you get to your commits stats from the try repo so we can schedule another reboot of it?

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Assigning to John O'Duinn until stats are gathered.

Comment 4

[Chris AtLee [:catlee]]

So, it's been less than a month since we last reset the try repo.  Having it take more than a minute to get data for a revision points to performance issues with hg itself, or the machines.

Aravind, is there something you can do to speed up the response?  What kind of machines/disk are serving this data?  Can we cache responses?

Comment 5

[Aravind Gottipati [:aravind]]

Short of adding some super fast ssd type option just for this one repo,(and thats a fine way to solve this problem (imo)), there isn't much I can do here.  To give you an idea of whats going on with this repo..

[root@dm-svn01 mozilla]# du -hs try
947M    try
[root@dm-svn01 mozilla]#

I don't expect all of that data to be traversed when folks request the a specific rev or do a pull, but still.. with a repo with that size, I'd expect a significant portion of the time is spent with the server navigating that maze to find out what you need.

I guess that's the price we pay for having a distributed vcs, with all the history in there.  There might be a way for us to truncate the history in the repo and just start off with a fresh import of a specific tag, and folks use that to submit patches.  I am not sure how workable that is or how much it will help, but I am looking into those options.

copying our mercurial expert, he might have some insight into making this more bearable.

Comment 6

[Dirkjan Ochtman (:djc)]

Okay, AFAIK, the try repo is the one with the crazy number of heads, right? Large numbers of heads make hg slow because it has to do tag calculation, and we have to traverse all the heads. Bug 554656 might help with that.

Also, in hg 1.5 we got a tag cache that should alleviate this problem. So upgrading might help, although there might be an issue in that the tag cache files are often not easily writable by the web server process, so we'd need a solution for that.

Alternatively, I think the try repo is mostly used to push (series of) patches to let it run the tests. It might work to write a custom hg protocol implementation that uses the underlying repo to decide what patches to send, but that doesn't actually store everything (that's not in m-c), but instead just saves the patches off somewhere, and then invokes the tests.

Comment 7

[Dirkjan Ochtman (:djc)]

(Of course the first solution is really the same as the last one, except that it prunes even earlier.) Thought of another solution: get a patch into hg that makes tag-loading lazy for changeset pages, so that template writers can make a meaningful choice not to include tags on some pages (but this will not prevent some other pages from being slow).

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Sorry, just caught up on this, and tweaking summary to match what needs to be done. 

Conveniently, the end of the month is coming up, so maybe we schedule this for over the weekend, or on Monday 1st May?

Comment 9

[Jeff Muizelaar [:jrmuizel]]

This is getting really bad, it makes tinderboxpushlog on try nearly unusable. I frequently see pushlog fetching fail when trying to use tinderboxpushlog. I think we need a better solution than just waiting for resets.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Ping?

Comment 11

[Jeff Muizelaar [:jrmuizel]]

I've filed bug 563674 for the long term issue.