546939 - unit test/talos jobs fail when symbols are very slow to download

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Mostly this happens on staging, but could happen on production as well. Basically, because we set 'wget -nv' when downloading symbols there is no output at all while wget downloads. So if the download takes more than 20 minutes, Buildbot kills the build. We could either bump the timeout or use 'wget --progress=dot:mega' to generate some output and avoid being "idle" in the eyes of Buildbot.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Is staging-stage that slow to dish out a couple of hundred meg of file ? Yikes!

Comment 2

[Aki Sasaki (not active)]

I have a patch to make the -nv overridable with other options (or empty list) since I was running into this a lot with the n810s.
I'll dig that up and attach.

Comment 3

[Aki Sasaki (not active)]

Attachment: add wget_args to DownloadFile

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 427642 [details] [diff] [review]
add wget_args to DownloadFile

This is fine, but I'd rather the default be --progress=dot:mega or :giga rather than no output at all. r=me with that change.

Comment 5

[[e-mail] (:bhearsum)]

Apparently Alice has hit this on Talos, too

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 427642 [details] [diff] [review]
add wget_args to DownloadFile

>+        else:
>+            self.wget_args = ['-nv']

As checked in
  http://hg.mozilla.org/build/buildbotcustom/rev/fcc6ac419906
using
              self.wget_args = ['--progress=dot:mega']

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Updated pm01, pm02, talos-pool, talos-r3, talos-try. What else needs doing ?