529488 - Try-server build failing due to "transaction abort!" / "abort: premature EOF reading chunk" during hg clone

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Daniel Holbert [:dholbert]]

This just happened on both the Linux and the Windows box, when generating a try-server build for me:
{
/tools/python/bin/hg clone --verbose --noupdate --rev b1748380ef8c0da4857ef64b77b0bf72b8a77c74 http://hg.mozilla.org/try /builds/slave/sendchange-linux-hg/build
 in dir /builds/slave/sendchange-linux-hg (timeout 3600 secs)
[SNIP]
requesting all changes
adding changesets
transaction abort!
rollback completed
abort: premature EOF reading chunk (got 102 bytes, expected 286)
elapsedTime=371.049690
program finished with exit code 255
}

The above is taken from the Windows log. (linked below)  The linux log looks basically the same.

I've just tried running the "hg clone" command locally (exactly as it is given in the snippet above), and it worked fine on my machine. Not sure why it failed on both of these builders...

Linux log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258526669.1258527588.25905.gz
Windows log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258526669.1258527588.25905.gz

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Aravind, could this be fallout from the new hg head ?

Comment 2

[Daniel Holbert [:dholbert]]

FWIW, I re-pushed-to-try, and it was fine that time.

I then hit this again a little while later, with a different series of builds:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258538663.1258539639.554.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258538549.1258539078.26289.gz
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258538543.1258539468.30563.gz

Comment 3

[Daniel Holbert [:dholbert]]

Note that the logs in comment 2 are for Windows, Linux, & OS X.  So this is very cross-platform, at least on the build-slave side.

Perhaps it's something wonky on the server side? (i.e. maybe whenever it receives a push, it interrupts in-progress clones of itself?)

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

Happened to me too:
WINNT 5.2 try hg build on 2009/11/18 06:14:22
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1258553662.1258554146.10007.gz

Comment 5

[[e-mail] (:bhearsum)]

> Aravind, could this be fallout from the new hg head ?

Can someone from IT have a look at the state of the hg webheads? Given the timing, it seems pretty likely the new one is having issues.

Comment 6

[Axel Hecht [:Pike]]

bug 529515 is about a different error, started at 2009-11-18 00:57:20+0000 according to my logs.

Comment 7

[(not currently active) Ted Mielczarek]

We shouldn't be marking infra problems as [orange]. We should just fix them!

Comment 8

[Aravind Gottipati [:aravind]]

This should be fixed now, I upgraded the load balancer and redistributed the weights on the backed servers.