988665 - Pushing to try is hanging

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Originally reported by bholley in #releng:

bholley	try repo appears locked - known issue?
nthomas	fubar: ^^
bholley	gets hangs when he tries to push to it, at least
nthomas	pushing just a few changesets from recent m-c rev, presumably ?
bholley	nthomas: yes
nthomas: also, nobody's pushed since my last push half an hour ago
nthomas	ok, just wanted to rule out something big
nthomas	bkero: you there ?
bholley	nthomas: which, given the normal frequency, is suspicious 

I tried to verify. Updated by clone of mozilla-central, made a whitespace change to a file in a mq patch. hg out worked fine, < 10 seconds for it to detect push of recent m-c changes, and my patch.

Got this on push:

$ time hg push -f ssh://hg.mozilla.org/try
pushing to ssh://hg.mozilla.org/try
searching for changes
^Cinterrupted!
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:9505'
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 33 changesets with 142 changes to 29 files (+1 heads)
remote: Killed by signal 2.

real	2m17.554s
user	0m1.880s
sys	0m0.179s

It's possible that my ^C has put changesets in the repo but not pushlog. Please investigate ASAP.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> It's possible that my ^C has put changesets in the repo but not pushlog.
> Please investigate ASAP.

Probably not, they're still shown if I do 'hg out ssh://hg.mozilla.org/try' again. Here's a push with debug output:
   https://pastebin.mozilla.org/4695956

Looks like my side generates all the bundles it needs to send, then output stops.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

:fubar is looking at the backend, while my push from comment #1 is still running (19mins so far).

Comment 3

[Kendall Libby [:fubar] (he/him)]

hg was stuck waiting for a lock, which it already had. strace was silent. killed the proc, which freed everything up, but left the push without a pushlog entry.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We're back in business. I did a push in 22s, and a couple of others have also pushed.

bholley, could you provide a diff of what you were trying to push when you got the initial hang ? bkero says that (typically for these hangs, I'm paraphrasing slightly) 'hg spends a lot of time finding parent deltas to apply the new commit. Since our repo history is so long and parents can be so old this can take quite a bit of time.' Which I take to mean changing a file which hasn't sat for a long time can be problematic.

Comment 5

[Bobby Holley (:bholley)]

(In reply to Nick Thomas [:nthomas] from comment #4)
> bholley, could you provide a diff of what you were trying to push when you
> got the initial hang ?

What leads you to believe that my hang was the initial one?

I had several different try pushes hang. IIRC, the first one touched only content/base/public/nsIContent.h, which is touched pretty often.

Comment 6

[Hal Wine [:hwine] use NI!]

tracking, but this particular issue is past.