696682 - Please manually reset the Try repo

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Phil Ringnalda (:philor)]

Yeah, I know all about the multiple duplicate bugs to automate it, but they're stalled and People Are Dying.

I'd dismiss

slice:~/mc/mozilla philor$ time hg push -f -v try
pushing to ssh://hg.mozilla.org/try/
running ssh hg.mozilla.org "hg -R try/ serve --stdio"
searching for changes
1 changesets found
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 4 changes to 4 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://people.mozilla.org/~lsblakk/trychooser/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

real	66m50.535s
user	0m1.723s
sys	0m0.463s

as just my crappy connection, but I heard about someone else's 40 minute push the other day, and 5, 10, or 15 minute pushes (mine prior to that 66 minute one was 7 minutes) are constantly being complained-about.

Please do whatever stats gathering that was that you used to do when we used to trim it regularly, schedule a downtime, and just throw it away and start over with a nice new mozilla-central.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(philor: Unstated, but I assume you are only seeing this slow performance on try repo. If you are seeing this with other repos, we have a wider hg.m.o problem to address.)


Instead of deleting the try repo, lets do what worked earlier this year: 

schedule downtime to:
* rename try -> try.broken
* create new try repo from m-c

...and then I can pull stats data from try.broken when I get back from vacation.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Do we need to close anything other than Try?

Comment 3

[Phil Ringnalda (:philor)]

Yes, I'm only seeing it and only hearing about it for Try.

And I can't imagine why it would require closing anything but Try, but I didn't even know we had done a reset earlier this year - what did we do then?

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

according to our notes, we don't have to close anything other than try.

Comment 5

[Marco Bonardo [:mak]]

If the issue is due to high number of heads, I wonder if it would be possible to make a script that once a week hg strips all heads older than a certain amount of time (like a month). Going through heads would be easy, filtering them by date would be easy, but I wonder if there's a way to get the first changeset that inited a specified branch to strip it. Alternatively I think just closing heads may work similarly. This would allow to cleanup without the need to close the tree, while maintaining perf constant.

Comment 6

[Ed Morley [:emorley]]

(In reply to Marco Bonardo [:mak] from comment #5)
> If the issue is due to high number of heads, I wonder if it would be
> possible to make a script that...

Bug 554656 / bug 633161 are looking into this, but have been stalled for a while:

(In reply to Phil Ringnalda (:philor) from comment #0)
> Yeah, I know all about the multiple duplicate bugs to automate it, but
> they're stalled and People Are Dying.

Comment 7

[Chris Cooper [:coop] (he/him)]

Taking advantage of the tree closure due to bug 697374 this morning, Amy moved the old try repo aside (to address joduinn's comment #1), and then re-cloned the try repo from m-c.

We're currently investigating how this is going to affect using local mirrors with our build systems (bug 665021), but try *should* be usable again once the trees reopen.

Comment 8

[Amy Rich [:arr] [:arich]]

(In reply to John O'Duinn [:joduinn] from comment #1)
> (philor: Unstated, but I assume you are only seeing this slow performance on
> try repo. If you are seeing this with other repos, we have a wider hg.m.o
> problem to address.)
> 
> 
> Instead of deleting the try repo, lets do what worked earlier this year: 
> 
> schedule downtime to:
> * rename try -> try.broken
> * create new try repo from m-c

I've moved the existing repo to /repo/hg/mozilla/try-bug696682 and run /repo/hg/scripts/reset_try.sh.  Bug 697457 has been opened to fix the interaction between the (not yet in use) mirrors.

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Aside from the deletion of the bad repo when joduinn is done with it, is there any work left here?

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #9)
> Aside from the deletion of the bad repo when joduinn is done with it, is
> there any work left here?

Devs are reporting that new commits aren't appearing in hgweb, although I can confirm that the commits *are* making it into the try repo. 

I've pinged IT about tracking down what needs to be flushed in hgweb.

Comment 11

[Ed Morley [:emorley]]

Changing to blocker due to comment 10 + inability to use TBPL with try:
https://tbpl.mozilla.org/?tree=Try

Comment 12

[Daniel Holbert [:dholbert]]

(In reply to Chris Cooper [:coop] from comment #10)
> Devs are reporting that new commits aren't appearing in hgweb

In particular, http://hg.mozilla.org/try/pushloghtml is basically blank.
(compare to http://hg.mozilla.org/mozilla-central/pushloghtml )

Comment 13

[Chris Cooper [:coop] (he/him)]

OK, since try is still closed, I've asked IT to try re-cloning Try from m-c, making sure that the pushlog and all the hooks are in place from the get-go.

Apologies to devs that have pushed-to-try in the interim, you'll need to push again once we reopen the tree.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

we ended up cloning again due to a busted pushlog.  If you have pushed to try today before now, you might need to push again due to the repository being removed.

I just tested pushing to try results in an entry in /try/pushlog and that I was emailed on the start of the build jobs.  That email comes from buildbot, so I take that as a sign that things are working.

Comment 15

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #14)
> we ended up cloning again due to a busted pushlog.  If you have pushed to
> try today before now, you might need to push again due to the repository
> being removed.
> 
> I just tested pushing to try results in an entry in /try/pushlog and that I
> was emailed on the start of the build jobs.  That email comes from buildbot,
> so I take that as a sign that things are working.

Yes, second re-clone did the trick.