398460 - Intermittent slave failures on qm-pxp0*

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rob Campbell [:rc] (:robcee)]

Seen on the qm-pxp0n "blades", occasionally a talos run will turn red due to download or zip file failures. Download failures could probably be fixed by repolling the download site a few times. Zip failures are not really "fixable" and should probably continue to fail.

Comment 1

[alice nodelman [:alice] [:anode]]

If you are being served corrupted zips is there someone in build who could look at it?

Comment 2

[Rob Campbell [:rc] (:robcee)]

yes, absolutely. And they should!

Comment 3

[[e-mail] (:bhearsum)]

I've been watching for these lately and I'm 99% sure the packages get corrupted during the transfer. On a few occasions I've manually tested a package that the Talos machine had trouble with and each time I had no issues with it.

Comment 4

[alice nodelman [:alice] [:anode]]

So, we could try something simple like adding a step to test the zip and looping to re-download, with some failsafe to break out if it looks like there really is something wrong with the copy on the server.

unzip comes with a test feature, you can check if a zip is okay with unzip -tq.  I'd assume that other unzippers have similar options for us to work with.

Comment 5

[[e-mail] (:bhearsum)]

It's going to be a bit more complicated than that -- Buildbot doesn't have a "looping" concept for BuildSteps.

We may be able to string some shell commands together to accomplish this. "wget ... && unzip -tq firefox.zip"..something like that.

Another idea that popped into my head is simply doing a TinderboxPrint when unpacking the zip file fails. Showing "bad build" on the main page may mitigate the red tree.

Comment 6

[alice nodelman [:alice] [:anode]]

Can't buildbot have a buildstep which is a script that can loop?  It's got to be running the Talos code somehow...

Comment 7

[[e-mail] (:bhearsum)]

Yeah, I think that's what I was trying to say in my second paragraph (but did a poor job of it).

Comment 8

[alice nodelman [:alice] [:anode]]

This was resolved with having talos pull build zips from dated directories.  The redness was due to talos attempting to download a build while a given build machine was dropping a new build with the same name in the same directory.  Having builds go into unique, dated directories means that we no longer get any collisions.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #8)
> This was resolved with having talos pull build zips from dated directories. 
> The redness was due to talos attempting to download a build while a given build
> machine was dropping a new build with the same name in the same directory. 
> Having builds go into unique, dated directories means that we no longer get any
> collisions.
 
...and we really like that! :-D

Comment 10

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.