551663 - cn-sea-qm-win2k3-01 has been timing out on 'make upload' for days

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect)

Description

[Serge Gautherie (:sgautherie)]

At least on trunk...

Comment 1

[Robert Kaiser]

I see two potential causes possibly plying together:
1) This machine is hosted in Amsterderdam
2) Windows machines somehow don't print their progress while uploading

Still, the timeout is set quite high, IIRC, I wonder what's really up there, as this hasn't always been a problem with that box.

Comment 2

[Robert Kaiser]

OK, I currently am running such an upload that failed before manually for testing, and I see how the rate can slow down enough that we can run out of the 20 minutes timeout on stdout - I guess the actual output is running to stderr there, which buildbot doesn't apply this timeout to :(

Comment 3

[Robert Kaiser]

Due to the crashreporter-symbols.zip files, which are 60-70MB, we're usually uploading almost 90MB per build, with rates on this machine in Amsterdam going down to 60-70KB/s at times, we easily go over the 1200 second default limit of build steps (90MB at 65KB/s takes slightly over 1400s).


I see 4 possible solutions:
1) Get that VM back to the US side of the Atlantic (same for the Linux box in .nl)
2) Care that the upload rate from .nl to .sj doesn't drop that low
3) Get "make upload" to actually send output to stdout on Windows in trunk and 1.9.1 trees
4) Bump the timeout on the upload build step high enough that we still have a comfortable failure margin even with the bandwidth in Amsterdam, i.e. to at least 30min, better closer to an hour.


1) and 2) are IT stuff, so CCing mrz for that.
3) and 4) are RelEng, CCing bhearsum for those.

Matthew, Ben, what do you think? What solutions seem to best or most reasonable way to go with? What is doable at all?

Comment 4

[Robert Kaiser]

Attachment: bump the timeout to 1h

As nobody seems to care about what I was asking, and we have red builds out there, I'm going for the simplest way and bump the upload timeout locally on the SeaMonkey master. Requesting review to make this a general thing.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 433072 [details] [diff] [review]
bump the timeout to 1h

Yeah, this seems necessary, sadly. Windows builds don't seem to output progress on the uploads, so you get very little output until the step is fully done.

Comment 6

[Serge Gautherie (:sgautherie)]

Ftr, it looks like (MozillaBuild 1.3) scp should display a progress meter by default:
http://www.mkssoftware.com/docs/man1/scp.1.asp
"-q    Disables the progress meter."
But, when I try "scp originalname copyname" locally, I don't get any output :-(

Comment 7

[Robert Kaiser]

Pushed as http://hg.mozilla.org/build/buildbotcustom/rev/af4475e73c25 with permission from coop, who's on build duty this week.

Comment 8

[Serge Gautherie (:sgautherie)]

Seemed fixed but happened again:

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1269889473.1269895261.28441.gzWINNT 5.2 comm-central-trunk leak test build on 2010/03/29 12:04:33

We'll see how often...