$ time hg push -f try pushing to ssh://hg.mozilla.org/try searching for changes remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:10522' remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 5 files (+1 heads) remote: Looks like you used try syntax, going ahead with the push. remote: If you don't get what you expected, check http://people.mozilla.org/~lsblakk/trychooser/ for help with building your trychooser request. remote: Thanks for helping save resources, you're the best! remote: Trying to insert into pushlog. remote: Please do not interrupt... remote: Inserted into the pushlog db successfully. real 2m24.139s user 0m0.875s sys 0m0.071s
Resetting the Talos benchmark hurt me today, so it would be really painful for me to see the stuff that I've been working on try disappear by resetting try. :(
I this still a problem?
I have noticed a push being slow today. I will time it next time I push something.
Last try was real 1m45.594s user 0m1.094s sys 0m0.185s
now it is real 2m58.934s user 0m0.941s sys 0m0.056s
Ben, any thoughts from your hg archeology?
I haven't been looking at try as much as fixing the underlying problems with varnish, which appear to be completed now, so I can focus on this. try pushes get sent from nginx on dm-hg02 directly to apache on dm-vcview04. dm-vcview04 apache is extremely busy, pegging all 8 cores with a loadavg hovering around 20. The load is entire CPU-bound, with very little iowait. I'm wondering if the load itself could explain slow push times.
looks like it just got really bad: remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:30578' remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by dm-svn02.mozilla.org:30578 abort: unexpected response: empty string
Bug 708632 is a more recent incarnation of this.