688240 - pushing to try is really slow again

Status: RESOLVED DUPLICATE of bug 708632

(mozilla.org Graveyard :: Server Operations, task)

Description

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

$ time hg push -f try
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:10522'
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 5 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://people.mozilla.org/~lsblakk/trychooser/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

real	2m24.139s
user	0m0.875s
sys	0m0.071s

Comment 1

[(no longer active)]

Resetting the Talos benchmark hurt me today, so it would be really painful for me to see the stuff that I've been working on try disappear by resetting try.  :(

Comment 2

[Corey Shields [:cshields]]

I this still a problem?

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I have noticed a push being slow today. I will time it next time I push something.

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Last try was

real	1m45.594s
user	0m1.094s
sys	0m0.185s

Comment 5

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

now it is

real	2m58.934s
user	0m0.941s
sys	0m0.056s

Comment 6

[Corey Shields [:cshields]]

Ben, any thoughts from your hg archeology?

Comment 7

[Ben Kero [:bkero]]

I haven't been looking at try as much as fixing the underlying problems with varnish, which appear to be completed now, so I can focus on this.

try pushes get sent from nginx on dm-hg02 directly to apache on dm-vcview04.  dm-vcview04 apache is extremely busy, pegging all 8 cores with a loadavg hovering around 20.  The load is entire CPU-bound, with very little iowait.  I'm wondering if the load itself could explain slow push times.

Comment 9

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

looks like it just got really bad:

remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:30578'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by dm-svn02.mozilla.org:30578
abort: unexpected response: empty string

Comment 10

[Chris Cooper [:coop] (he/him)]

Bug 708632 is a more recent incarnation of this.