665077 - unable to push to try due to lock held by dm-svn02.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

oel@joel-laptop:~/mozilla/src$ hg push -f ssh://[e-mail]@hg.mozilla.org/try
pushing to ssh://[e-mail]@hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:15443'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by dm-svn02.mozilla.org:15443
abort: unexpected response: empty string
joel@joel-laptop:~/mozilla/src$

Comment 1

[Ed Morley [:emorley]]

Has happened to me four times in a row this evening, I'm not able to push to try at all.

% hg --cwd C:\mozilla\mozilla-central push try -f
pushing to ssh://hg.mozilla.org/try/
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'dm-svn02.mozilla.org:15443'
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by dm-svn02.mozilla.org:15443
abort: unexpected response: empty string
[command returned code 255 Fri Jun 17 19:11:38 2011]

% hg --repository C:\mozilla\mozilla-central pull --verbose http://hg.mozilla.org/mozilla-central/
pulling from http://hg.mozilla.org/mozilla-central/
searching for changes
no changes found
[command completed successfully Fri Jun 17 19:25:55 2011]

% hg --cwd C:\mozilla\mozilla-central push try -f
pushing to ssh://hg.mozilla.org/try/
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'dm-svn02.mozilla.org:15443'
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by dm-svn02.mozilla.org:15443
abort: unexpected response: empty string
[command returned code 255 Fri Jun 17 19:37:17 2011]

% hg --cwd C:\mozilla\mozilla-central push try -f
pushing to ssh://hg.mozilla.org/try/
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'dm-svn02.mozilla.org:15443'
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by dm-svn02.mozilla.org:15443
abort: unexpected response: empty string
[command returned code 255 Fri Jun 17 20:14:23 2011]

Comment 2

[Armen [:armenzg]]

It seems that this has also happened to edmorley and jdm yesterday.

Comment 3

[Josh Matthews [:jdm]]

See bug 664835 for yesterday's problems, which look identical.

Comment 4

[Armen [:armenzg]]

Other similar bugs:
* https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20prod%3Amozilla.org%20%22column%20rev%20is%20not%20unique%22&list_id=512724

Comment 5

[Benjamin Smedberg]

bsmedberg $ hg push
pushing to ssh://hg.mozilla.org/mozilla-central
searching for changes
  >> I hit control-C here
^Cinterrupted!
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3 changesets with 16 changes to 15 files
remote: Sending messages to pulse.mozilla.org
remote: Please do not interrupt...
remote:     Sending the push message...
remote: Killed by signal 2.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Note that comment 5 is mozilla-central rather than try.


This also happened to me pushing to a user repo:

dbaron@crum Linux (0) ~/builds/mozilla-central/mozilla/toolkit/xre $ mq push
pushing to ssh://hg.mozilla.org/users/dbaron_mozilla.com/patches/
searching for changes
^Cinterrupted!
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 9 changes to 9 files
remote: Sending messages to pulse.mozilla.org
remote: Please do not interrupt...
remote:     Sending the push message...
dbaron@crum Linux (255) ~/builds/mozilla-central/mozilla/toolkit/xre $ mq push
pushing to ssh://hg.mozilla.org/users/dbaron_mozilla.com/patches/
searching for changes
^Cinterrupted!
remote: waiting for lock on repository /repo/hg/mozilla/users/dbaron_mozilla.com/patches/ held by 'dm-svn02.mozilla.org:1852'
dbaron@crum Linux (255) ~/builds/mozilla-central/mozilla/toolkit/xre $ type mq
mq is aliased to `hg -R$(hg root)/.hg/patches'
dbaron@crum Linux (0) ~/builds/mozilla-central/mozilla/toolkit/xre $

Comment 7

[Justin Dow [:jabba]]

We've traced these issues down to hg waiting for pulse.mozilla.org, which appears to have had an incorrectly set timeout. Christian lowered the timeout and kicked whatever process was stuck and things seem to be flowing smoothly again. We've discussed in IRC that further action will be taken to ensure that pulse doesn't become a single point of failure for hg and will look into building it out into a more scalable infrastructure.

For now the issues seem to be resolved and I'm told that further problems with pulse shouldn't block an hg push anymore.

Comment 8

[Paul O'Shannessy [:zpao] (not bugmail, email directly)]

I think this is happening again. When I kill the push, it was in the "Sending messages to pulse.mozilla.org" stage.

Comment 9

[Dave Townsend [:mossop]]

I'm seeing this pushing to m-c too, here is the error if I let it finish (takes about 10 minutes):

pushing to ssh://hg.mozilla.org/mozilla-central
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/mozilla-central held by 'dm-svn02.mozilla.org:2432'
remote: abort: repository /repo/hg/mozilla/mozilla-central: timed out waiting for lock held by dm-svn02.mozilla.org:2432
abort: unexpected response: empty string

Comment 10

[LegNeato]

Bug 665602 is for turning to hook off for now.

Comment 11

[Justin Dow [:jabba]]

Is this still an issue? I was afk most of the day yesterday, so didn't see that this got re-opened until now. If a bug is a blocker and requires immediate attention when it gets re-opened, re-assigning to the default assignee will ensure that the oncall sysadmin gets paged and will receive immediate attention.

Judging by the last two comments, it appears that the problem was with the pulse hook, which has been disabled. I'll close this out as such. If there are still issues, please file a new bug for server-ops or re-open this one and re-assign to server-ops, as I'll be afk again most of today.