962275 - Please reset the Try repo

Status: RESOLVED FIXED

(Developer Services :: General, task, P1)

Description

[John Hopkins (:jhopkins)]

Performance against the Try repo has been deteriorating (eg. bug 959769) and very much needs a reset.
Please reset Try as soon as possible.

Comment 1

[John Hopkins (:jhopkins)]

I've been informed that we need to communicate the reset in advance to developers so please hold off for the moment.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Note that due to Ed's recent TBPL patch (https://hg.mozilla.org/webtools/tbpl/rev/4e3ed9dc8c66), the effect on developers is basically mitigated. He posted to dev-platform about it today.

Comment 3

[John Hopkins (:jhopkins)]

pir: would you be available for a Try reset early Thursday morning, Eastern time?

Comment 4

[Ashish Vijayaram [:ashish]]

Lowering priority to stop paging oncall.

Comment 5

[Shyam Mani [:fox2mike]]

Hal filed Bug 962291 for this...do you need it done before that?

Comment 6

[Hal Wine [:hwine] use NI!]

A slow try server has at least 2 impacts:
 a) pushlog issues, mitigated already per comment 2
 b) longer wait times for devs when pushing to try

In this case, I don't know if (b) was also being reported. If (b) is visible to devs, then yes, we should reset "now" (with time for reasonable notification & quiet time of day).

Regardless of what we do for this bug, bug 962291 is filed to reset at the next TCW (and then every other TCW) so we don't suffer from (b) just because they've become used to the poorer performance.

Comment 7

[Shyam Mani [:fox2mike]]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #6)

> Regardless of what we do for this bug, bug 962291 is filed to reset at the
> next TCW (and then every other TCW) so we don't suffer from (b) just because
> they've become used to the poorer performance.

Perfect. If we're planning on resets every TCW to make sure we never see this again, I'm okay with that. Just wanted to confirm that this bug isn't clashing with the TCW reset bug and if we do the reset now (before the next window), we'll probably skip the Feb window and do a reset again for the one after.

Comment 8

[Mike Hommey [:glandium]]

When is the next window? I have current on-going work that is blocked by (b), because pushing ~85 changesets in 2 hours apparently kills the mercurial server.

Comment 9

[Shyam Mani [:fox2mike]]

(In reply to Mike Hommey [:glandium] from comment #8)
> When is the next window? I have current on-going work that is blocked by
> (b), because pushing ~85 changesets in 2 hours apparently kills the
> mercurial server.

A month from now, so I guess we need to do this sooner than later.

Comment 10

[John Hopkins (:jhopkins)]

hwine: Ms2ger reported a 'somewhat' sluggish push to Try but not critical.

Comment 11

[Corey Shields [:cshields]]

cab-review+  (can't find the flag for some reason) - scheduled for Thursday 1/23 at 0500 PST.

Comment 12

[Shyam Mani [:fox2mike]]

(In reply to Corey Shields [:cshields] from comment #11)
> cab-review+  (can't find the flag for some reason) - scheduled for Thursday
> 1/23 at 0500 PST.

Because the bug isn't in the right component.

Comment 13

[Kendall Libby [:fubar] (he/him)]

Try reset; :jhopkins has kicked buildbot and reopened the tree.

hgssh1.dmz.scl3# /repo/hg/scripts/reset_try.sh 
Resetting try is a disruptive event to developer worflows and must be
coordinated with RelEng buildduty, and notifications sent to the CAB and
dev mailing lists.
Proceed? (y/N): y
Okay, here we go!
Moving current try repo to /repo/hg/nonlive/try-reset-2014-01-23-0501
Cloning mozilla-central into the try repo
requesting all changes
adding changesets
adding manifests
adding file changes
added 164827 changesets with 925666 changes to 147289 files
Trying to insert into pushlog.
Please do not interrupt...
Inserted into the pushlog db successfully.

real	15m44.117s
user	8m38.472s
sys	1m1.543s
Fixing try repo permissions
Cleaning up pushlog.db
All done

Comment 14

[Gregory Szorc [:gps]]

Are the old try repos publicly available via hg.mozilla.org?

FWIW, my unofficial mirror at http://hg.gregoryszorc.com/mirrors/try/ has both the old and new heads. I'm kinda curious how well it will scale with Mercurial 2.8.2...

Comment 15

[Hal Wine [:hwine] use NI!]

(In reply to Gregory Szorc [:gps] from comment #14)
> Are the old try repos publicly available via hg.mozilla.org?

No, they should not be.  If you spot them, that would be bug worthy. :)

> FWIW, my unofficial mirror at http://hg.gregoryszorc.com/mirrors/try/ has
> both the old and new heads. I'm kinda curious how well it will scale with
> Mercurial 2.8.2...

Can you put any findings in bug 945383 so we can find them, please?