Last Comment Bug 696682 - Please manually reset the Try repo
: Please manually reset the Try repo
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: P2 blocker (vote)
: ---
Assigned To: Chris Cooper [:coop]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-23 16:32 PDT by Phil Ringnalda (:philor)
Modified: 2013-08-12 21:54 PDT (History)
11 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Phil Ringnalda (:philor) 2011-10-23 16:32:59 PDT
Yeah, I know all about the multiple duplicate bugs to automate it, but they're stalled and People Are Dying.

I'd dismiss

slice:~/mc/mozilla philor$ time hg push -f -v try
pushing to ssh://hg.mozilla.org/try/
running ssh hg.mozilla.org "hg -R try/ serve --stdio"
searching for changes
1 changesets found
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 4 changes to 4 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://people.mozilla.org/~lsblakk/trychooser/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

real	66m50.535s
user	0m1.723s
sys	0m0.463s

as just my crappy connection, but I heard about someone else's 40 minute push the other day, and 5, 10, or 15 minute pushes (mine prior to that 66 minute one was 7 minutes) are constantly being complained-about.

Please do whatever stats gathering that was that you used to do when we used to trim it regularly, schedule a downtime, and just throw it away and start over with a nice new mozilla-central.
Comment 1 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2011-10-24 01:58:42 PDT
(philor: Unstated, but I assume you are only seeing this slow performance on try repo. If you are seeing this with other repos, we have a wider hg.m.o problem to address.)


Instead of deleting the try repo, lets do what worked earlier this year: 

schedule downtime to:
* rename try -> try.broken
* create new try repo from m-c

...and then I can pull stats data from try.broken when I get back from vacation.
Comment 2 John Ford [:jhford] CET/CEST Berlin Time 2011-10-24 07:51:25 PDT
Do we need to close anything other than Try?
Comment 3 Phil Ringnalda (:philor) 2011-10-24 10:41:09 PDT
Yes, I'm only seeing it and only hearing about it for Try.

And I can't imagine why it would require closing anything but Try, but I didn't even know we had done a reset earlier this year - what did we do then?
Comment 4 John Ford [:jhford] CET/CEST Berlin Time 2011-10-24 10:53:49 PDT
according to our notes, we don't have to close anything other than try.
Comment 5 Marco Bonardo [::mak] 2011-10-26 06:37:16 PDT
If the issue is due to high number of heads, I wonder if it would be possible to make a script that once a week hg strips all heads older than a certain amount of time (like a month). Going through heads would be easy, filtering them by date would be easy, but I wonder if there's a way to get the first changeset that inited a specified branch to strip it. Alternatively I think just closing heads may work similarly. This would allow to cleanup without the need to close the tree, while maintaining perf constant.
Comment 6 Ed Morley [:emorley] 2011-10-26 07:39:39 PDT
(In reply to Marco Bonardo [:mak] from comment #5)
> If the issue is due to high number of heads, I wonder if it would be
> possible to make a script that...

Bug 554656 / bug 633161 are looking into this, but have been stalled for a while:

(In reply to Phil Ringnalda (:philor) from comment #0)
> Yeah, I know all about the multiple duplicate bugs to automate it, but
> they're stalled and People Are Dying.
Comment 7 Chris Cooper [:coop] 2011-10-26 09:13:04 PDT
Taking advantage of the tree closure due to bug 697374 this morning, Amy moved the old try repo aside (to address joduinn's comment #1), and then re-cloned the try repo from m-c.

We're currently investigating how this is going to affect using local mirrors with our build systems (bug 665021), but try *should* be usable again once the trees reopen.
Comment 8 Amy Rich [:arr] [:arich] 2011-10-26 09:38:00 PDT
(In reply to John O'Duinn [:joduinn] from comment #1)
> (philor: Unstated, but I assume you are only seeing this slow performance on
> try repo. If you are seeing this with other repos, we have a wider hg.m.o
> problem to address.)
> 
> 
> Instead of deleting the try repo, lets do what worked earlier this year: 
> 
> schedule downtime to:
> * rename try -> try.broken
> * create new try repo from m-c

I've moved the existing repo to /repo/hg/mozilla/try-bug696682 and run /repo/hg/scripts/reset_try.sh.  Bug 697457 has been opened to fix the interaction between the (not yet in use) mirrors.
Comment 9 John Ford [:jhford] CET/CEST Berlin Time 2011-10-26 10:35:23 PDT
Aside from the deletion of the bad repo when joduinn is done with it, is there any work left here?
Comment 10 Chris Cooper [:coop] 2011-10-26 10:52:30 PDT
(In reply to John Ford [:jhford] from comment #9)
> Aside from the deletion of the bad repo when joduinn is done with it, is
> there any work left here?

Devs are reporting that new commits aren't appearing in hgweb, although I can confirm that the commits *are* making it into the try repo. 

I've pinged IT about tracking down what needs to be flushed in hgweb.
Comment 11 Ed Morley [:emorley] 2011-10-26 12:15:44 PDT
Changing to blocker due to comment 10 + inability to use TBPL with try:
https://tbpl.mozilla.org/?tree=Try
Comment 12 Daniel Holbert [:dholbert] 2011-10-26 12:24:17 PDT
(In reply to Chris Cooper [:coop] from comment #10)
> Devs are reporting that new commits aren't appearing in hgweb

In particular, http://hg.mozilla.org/try/pushloghtml is basically blank.
(compare to http://hg.mozilla.org/mozilla-central/pushloghtml )
Comment 13 Chris Cooper [:coop] 2011-10-26 13:04:48 PDT
OK, since try is still closed, I've asked IT to try re-cloning Try from m-c, making sure that the pushlog and all the hooks are in place from the get-go.

Apologies to devs that have pushed-to-try in the interim, you'll need to push again once we reopen the tree.
Comment 14 John Ford [:jhford] CET/CEST Berlin Time 2011-10-26 13:27:15 PDT
we ended up cloning again due to a busted pushlog.  If you have pushed to try today before now, you might need to push again due to the repository being removed.

I just tested pushing to try results in an entry in /try/pushlog and that I was emailed on the start of the build jobs.  That email comes from buildbot, so I take that as a sign that things are working.
Comment 15 Chris Cooper [:coop] 2011-10-31 13:13:56 PDT
(In reply to John Ford [:jhford] from comment #14)
> we ended up cloning again due to a busted pushlog.  If you have pushed to
> try today before now, you might need to push again due to the repository
> being removed.
> 
> I just tested pushing to try results in an entry in /try/pushlog and that I
> was emailed on the start of the build jobs.  That email comes from buildbot,
> so I take that as a sign that things are working.

Yes, second re-clone did the trick.

Note You need to log in before you can comment on or make changes to this bug.