994028 - Timeouts on try while searching for changes ("remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:NNNNN")

Status: RESOLVED WORKSFORME

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Carsten Book [:Tomcat]]

A coupe of devs complaint that to try result in  "searching for changes" for more then +5 minutes and then times out. 

Not sure what the issue is, but this seems to affect several people in different locations

Comment 1

[Hannes Verschore [:h4writer]]

> $ try
> pushing to ssh://hg.mozilla.org/try
> searching for changes
> ^Cinterrupted!
> remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:1850'

Comment 2

[Julian Seward [:jseward]]

$ hg -v push -f ssh://[e-mail]@hg.mozilla.org/try/
pushing to ssh://[e-mail]/try/
running ssh [e-mail]@hg.mozilla.org 'hg -R try/ serve --stdio'
searching for changes
1 changesets found
^Cinterrupted!
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:25220'
remote: Killed by signal 2.

Comment 3

[Pete Moore [:pmoore][:pete]]

Duplicate of: 994647

Comment 5

[Gabor Krizsanits (INACTIVE)]

Seems to be working again. At least for me.

Comment 6

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

It has been working on and off for a couple of days now but it down for a good couple of hours when it fails. Occasionally it does go through but is really slow (10-15 minutes).

I keep getting:
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:25220'

So the same server for everyone in this bug.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

this is happening consistently for me.  Is there a workaround we can do locally?

Comment 8

[Carsten Book [:Tomcat]]

raising severity since again devs complaint about this issue and that this is blocking them for using try

Comment 9

[Ryan Watson [:w0ts0n]]

Taking until I can find someone from webops to work on this.

Comment 10

[Kendall Libby [:fubar] (he/him)]

Try is currently up to 7700 heads, which is around when things start to get ugly. OTOH, we've managed to get up to over 21,000...

It would be helpful to know what the parent revs are to changesets that are failing. The older they are, the longer it takes for hg to process, which can cause things to timeout and fall over.

I'd like to also note that we had planned to do regular resets of try during the 6-week tree closing windows, but were told by "devs" that doing so was painful and to not reset try. And yet, here we are, all unhappy. :-(

Comment 11

[Hal Wine [:hwine] use NI!]

Note: 5 minutes (comment 1) is not considered excessive. The guidelines at https://wiki.mozilla.org/ReleaseEngineering/TryServer#Pushes_to_try_take_a_very_long_time suggests 15 minutes is about when things are "bad".

Please confirm you are requesting a reset of try -- that process is disruptive to other devs.

Comment 13

[(no longer active)]

In comment 1 and 2, it seems like people are being blocked by a lock which was not properly released.  That is a different issue than the traditional "push to try takes long because too many heads" (the distinction being the push taking long versus timing out).  Which problem do you see, Joel?

(FWIW I have not noticed pushing to try to take an unusually long amount of time as of Sunday.)

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

when I was trying to push last week, I was unable to push at all to try for about 4 hours, then it started working.  While I couldn't push, I did see new pushes to try.  My case was a timeout on april 16th.

Comment 15

[(no longer active)]

(In reply to comment #14)
> when I was trying to push last week, I was unable to push at all to try for
> about 4 hours, then it started working.  While I couldn't push, I did see new
> pushes to try.  My case was a timeout on april 16th.

Yeah, seems like this was not the usual problem of pushes to try being slow because of the number of heads then.  Thanks!

Comment 16

[(no longer active)]

This has striken again, today I got the following twice:

pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:15332'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:15332

Comment 17

[Paul Rouget [:paul]]

I'm unable to push. No mater what I do. Stuck on "searching for changes". After ~10 minutes, it times out.

Comment 18

[Paul Rouget [:paul]]

Same logs for the past 2 hours (I tried to push many times):

pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:28056'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:28056
abort: unexpected response: empty string

Comment 19

[David Burns :automatedtester]

I have just been hit by this

(marionette)☁  marionette  hg try
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:18064'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:28994
abort: unexpected response: empty string

Comment 20

[Mason Chang [Inactive] [:mchang]]

Attachment: hg try push with --debug

:hwine asked me to post this on IRC. This is a try push with the --debug option from hg. I actually only have 1 changeset, so I'm not sure where the other 38 came from.

Comment 21

[Daniel Holbert [:dholbert]]

(The other 38 are just csets that you've pulled from upstream [m-c or m-i or wheverer] & you're the first person to push those csets to try. That's normal.)

There have been no successful pushes to Try by anyone since 11:55 PDT today (so, approaching 4 hours). hwine, not sure if you're actively looking into this -- if you are, awesome. If not, do you know who should look at it & how we can escalate it?

Comment 22

[Phil Ringnalda (:philor)]

It unhorked itself, after four hours and ten minutes.

If you don't want to do this again, there are two things to do:

Right now, reply to the thread in https://groups.google.com/d/msg/mozilla.dev.platform/Hb2EKXZmY70/Ijzo3Jo2WxcJ saying that you disagree that it is better to let try get to the point of completely sucking and then do an emergency reset (and that both of those things are better than just sitting around for four hours and ten minutes).

Next time it happens, https://wiki.mozilla.org/ReleaseEngineering/TryServer#Pushes_to_try_take_a_very_long_time says that if you are unable to push for more than 15 minutes (and implies that if your timeouts keep coming up with the same PID), to file a bug. Judging by the outcome of bug 1001735, where the eventual response was to ask the reporter whether or not he personally was still unable to push, at a time five days later when it was known by all that nobody had been able to push for three hours, that means *everyone* file *their own* bug asking for a reset.

Comment 23

[Hal Wine [:hwine] use NI!]

(In reply to Phil Ringnalda (:philor) from comment #22)
> It unhorked itself, after four hours and ten minutes.
> 
> If you don't want to do this again, there are two things to do:
> 
> Right now, reply to the thread in
> https://groups.google.com/d/msg/mozilla.dev.platform/Hb2EKXZmY70/
> Ijzo3Jo2WxcJ 

Excellent suggestion. The major impact of resets is on developers. We need their help in coming up with criteria to make the call of "enough pain - reset!".


> Next time it happens, Judging by the outcome of bug 1001735,
> *everyone* file *their own* bug asking for a reset.

Please don't -- what we need is a process that works for all devs. The dev.platform thread is the right place to do this.

Comment 25

[Gregory Szorc [:gps]]

Don't see much value in keeping this bug open.