Last Comment Bug 554656 - Automatically prune try repo
: Automatically prune try repo
Status: RESOLVED DUPLICATE of bug 1055298
[kanban:engops:https://mozilla.kanban...
:
Product: Developer Services
Classification: Other
Component: Mercurial: hg.mozilla.org (show other bugs)
: other
: x86 Linux
: -- normal
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
: 529179 (view as bug list)
Depends on: 633161
Blocks: 564045 676420
  Show dependency treegraph
 
Reported: 2010-03-24 09:08 PDT by Chris AtLee [:catlee]
Modified: 2014-12-30 21:31 PST (History)
17 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Chris AtLee [:catlee] 2010-03-24 09:08:14 PDT
Our try repo accumulates a lot of heads over time, which slows down hg eventually.  It would be nice to be able to prune older heads.
Comment 1 Justin Wood (:Callek) 2010-03-28 01:17:09 PDT
Since we don't have any presumption of try being a stable-pullable repo, would it be possible to periodically just drop its current state on the floor, and re-push a copy from our then-current m-c to the same location.

Instead of trying to do some unsupported function of hg?

[Note I'm not sure if any of our infra needs the head information AFTER the fact]
Comment 2 Ted Mielczarek [:ted.mielczarek] 2010-03-28 09:29:20 PDT
That's what we currently do. We're looking for a solution that we can enact without having to enforce a downtime while we nuke the entire repo.
Comment 3 Chris AtLee [:catlee] 2010-03-29 06:49:34 PDT
(In reply to comment #1)
> Since we don't have any presumption of try being a stable-pullable repo, would
> it be possible to periodically just drop its current state on the floor, and
> re-push a copy from our then-current m-c to the same location.
> 
> Instead of trying to do some unsupported function of hg?
> 
> [Note I'm not sure if any of our infra needs the head information AFTER the
> fact]

Yeah, there's the risk of breaking builds if we drop the entire repo.  There's always *some* delay between a change being pushed to try, and the hg checkout being started.  This delay can be several minutes depending on if there's a free build slave available or not.  If we reset the entire repo in this period, then the subsequent hg clone will fail.
Comment 4 Dirkjan Ochtman (:djc) 2010-05-04 11:26:18 PDT
*** Bug 563340 has been marked as a duplicate of this bug. ***
Comment 5 Dirkjan Ochtman (:djc) 2010-05-04 11:31:02 PDT
Adding some people from the other bugs. See https://bugzilla.mozilla.org/show_bug.cgi?id=557325 (especially comment 6) for solutions here. Bug 563340 may also have useful information.
Comment 6 Chris AtLee [:catlee] 2010-05-04 11:46:24 PDT
As per bug 563340, if we can determine why try is so slow, and fix the underlying issue, then we wouldn't have to prune the repo automatically.  Pruning is just a workaround.
Comment 7 :Ehsan Akhgari 2010-05-06 17:17:32 PDT
Is the issue here only the number of heads?  If yes, can we backout every branch that people push and merge it against the base mozilla-central changeset, to get rid of the added head?
Comment 8 Ben Hearsum (:bhearsum) 2010-05-07 06:48:53 PDT
(In reply to comment #7)
> Is the issue here only the number of heads?  If yes, can we backout every
> branch that people push and merge it against the base mozilla-central
> changeset, to get rid of the added head?

Yeah, number of heads is the issue AFAIK. If we did this directly on an hg host we'd avoid getting these merges in pushlog, too, so we wouldn't get builds from them -- which is ideal.
Comment 9 Jeff Muizelaar [:jrmuizel] 2010-06-03 11:26:05 PDT
This is bad again:

time wget http://hg.mozilla.org/try/rev/e61a1699f49c
--14:21:12--  http://hg.mozilla.org/try/rev/e61a1699f49c
           => `e61a1699f49c'
Resolving hg.mozilla.org... 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.189|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: unspecified [text/html]

    [   <=>                               ] 45,685        47.67K/s             

14:21:29 (47.52 KB/s) - `e61a1699f49c' saved [45685]


real	0m17.285s
user	0m0.002s
sys	0m0.009s
Comment 10 Nick Thomas [:nthomas] 2010-06-03 13:48:16 PDT
There's two issues here I think:
1) accessing the try repo itself (eq comment #9) becomes slow; impacts people who're inspecting changesets and possibly build slaves pulling to build. Trimming heads may help
2) querying pushlog becomes slow, which impacts tbpl and buildbot

Ted, any thoughts on whether 2) is fixed by trimming heads ?
Comment 11 Ted Mielczarek [:ted.mielczarek] 2010-06-03 17:10:33 PDT
The pushlog queries that buildbot do should not touch the repo at all. (The feed simply hits the database.)

TBPL hits the HTML page, which does have to query the repo to get the changeset author and commit message.
Comment 12 Lukas Blakk [:lsblakk] use ?needinfo 2010-10-25 10:03:57 PDT
*** Bug 529179 has been marked as a duplicate of this bug. ***
Comment 13 Lukas Blakk [:lsblakk] use ?needinfo 2010-12-09 20:16:49 PST
I'm not actively working on this, putting it back in the pool.
Comment 14 Gregory Szorc [:gps] 2014-12-30 21:31:47 PST

*** This bug has been marked as a duplicate of bug 1055298 ***

Note You need to log in before you can comment on or make changes to this bug.