Automatically prune try repo

RESOLVED DUPLICATE of bug 1055298

Status

Developer Services
Mercurial: hg.mozilla.org
RESOLVED DUPLICATE of bug 1055298
7 years ago
3 years ago

People

(Reporter: catlee, Unassigned)

Tracking

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/819] )

(Reporter)

Description

7 years ago
Our try repo accumulates a lot of heads over time, which slows down hg eventually.  It would be nice to be able to prune older heads.
Since we don't have any presumption of try being a stable-pullable repo, would it be possible to periodically just drop its current state on the floor, and re-push a copy from our then-current m-c to the same location.

Instead of trying to do some unsupported function of hg?

[Note I'm not sure if any of our infra needs the head information AFTER the fact]
That's what we currently do. We're looking for a solution that we can enact without having to enforce a downtime while we nuke the entire repo.
(Reporter)

Comment 3

7 years ago
(In reply to comment #1)
> Since we don't have any presumption of try being a stable-pullable repo, would
> it be possible to periodically just drop its current state on the floor, and
> re-push a copy from our then-current m-c to the same location.
> 
> Instead of trying to do some unsupported function of hg?
> 
> [Note I'm not sure if any of our infra needs the head information AFTER the
> fact]

Yeah, there's the risk of breaking builds if we drop the entire repo.  There's always *some* delay between a change being pushed to try, and the hg checkout being started.  This delay can be several minutes depending on if there's a free build slave available or not.  If we reset the entire repo in this period, then the subsequent hg clone will fail.

Updated

7 years ago
Duplicate of this bug: 563340
Adding some people from the other bugs. See https://bugzilla.mozilla.org/show_bug.cgi?id=557325 (especially comment 6) for solutions here. Bug 563340 may also have useful information.
(Reporter)

Comment 6

7 years ago
As per bug 563340, if we can determine why try is so slow, and fix the underlying issue, then we wouldn't have to prune the repo automatically.  Pruning is just a workaround.
Blocks: 564045
Is the issue here only the number of heads?  If yes, can we backout every branch that people push and merge it against the base mozilla-central changeset, to get rid of the added head?
(In reply to comment #7)
> Is the issue here only the number of heads?  If yes, can we backout every
> branch that people push and merge it against the base mozilla-central
> changeset, to get rid of the added head?

Yeah, number of heads is the issue AFAIK. If we did this directly on an hg host we'd avoid getting these merges in pushlog, too, so we wouldn't get builds from them -- which is ideal.
This is bad again:

time wget http://hg.mozilla.org/try/rev/e61a1699f49c
--14:21:12--  http://hg.mozilla.org/try/rev/e61a1699f49c
           => `e61a1699f49c'
Resolving hg.mozilla.org... 63.245.208.189
Connecting to hg.mozilla.org|63.245.208.189|:80... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: unspecified [text/html]

    [   <=>                               ] 45,685        47.67K/s             

14:21:29 (47.52 KB/s) - `e61a1699f49c' saved [45685]


real	0m17.285s
user	0m0.002s
sys	0m0.009s
There's two issues here I think:
1) accessing the try repo itself (eq comment #9) becomes slow; impacts people who're inspecting changesets and possibly build slaves pulling to build. Trimming heads may help
2) querying pushlog becomes slow, which impacts tbpl and buildbot

Ted, any thoughts on whether 2) is fixed by trimming heads ?
The pushlog queries that buildbot do should not touch the repo at all. (The feed simply hits the database.)

TBPL hits the HTML page, which does have to query the repo to get the changeset author and commit message.
Duplicate of this bug: 529179
Assignee: nobody → lsblakk
I'm not actively working on this, putting it back in the pool.
Assignee: lsblakk → nobody
Depends on: 633161

Updated

6 years ago
Blocks: 676420
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

3 years ago
Product: Release Engineering → Developer Services

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/182]

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/182] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/819] [kanban:engops:https://kanbanize.com/ctrl_board/6/182]
(Assignee)

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/819] [kanban:engops:https://kanbanize.com/ctrl_board/6/182] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/819]

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1055298
You need to log in before you can comment on or make changes to this bug.