pushing to try is really slow again



mozilla.org Graveyard
Server Operations
6 years ago
2 years ago


(Reporter: espindola, Assigned: cshields)



$ time hg push -f try
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:10522'
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 5 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://people.mozilla.org/~lsblakk/trychooser/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

real	2m24.139s
user	0m0.875s
sys	0m0.071s
Resetting the Talos benchmark hurt me today, so it would be really painful for me to see the stuff that I've been working on try disappear by resetting try.  :(

Comment 2

6 years ago
I this still a problem?
Assignee: server-ops → cshields
I have noticed a push being slow today. I will time it next time I push something.
Last try was

real	1m45.594s
user	0m1.094s
sys	0m0.185s
now it is

real	2m58.934s
user	0m0.941s
sys	0m0.056s

Comment 6

6 years ago
Ben, any thoughts from your hg archeology?

Comment 7

6 years ago
I haven't been looking at try as much as fixing the underlying problems with varnish, which appear to be completed now, so I can focus on this.

try pushes get sent from nginx on dm-hg02 directly to apache on dm-vcview04.  dm-vcview04 apache is extremely busy, pegging all 8 cores with a loadavg hovering around 20.  The load is entire CPU-bound, with very little iowait.  I'm wondering if the load itself could explain slow push times.


6 years ago
Duplicate of this bug: 695454
looks like it just got really bad:

remote: waiting for lock on repository /repo/hg/mozilla/try held by 'dm-svn02.mozilla.org:30578'
remote: abort: repository /repo/hg/mozilla/try: timed out waiting for lock held by dm-svn02.mozilla.org:30578
abort: unexpected response: empty string
Bug 708632 is a more recent incarnation of this.
Last Resolved: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 708632
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.