794467 - Use '--no-verbose' with wget in the download steps

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Background to bug 765224: Many of the buildbot logs are extremely verbose even in the case of success, causing larger log sizes, thereby meaning slower TBPL parsing (and of late, timeouts) - as well as making it harder for someone to look at the log and ascertain what actually failed without trying a bunch of searches.

Another place I think we can cut back is by not outputting the non-error messages from wget, ie:

{
========= Started download (results: 0, elapsed: 13 secs) (at 2012-09-26 05:55:07.463120) =========
...
--2012-09-26 05:55:07--  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1348662559/firefox-18.0a1.en-US.linux-x86_64.tar.bz2
Resolving ftp.mozilla.org... 63.245.215.46
Connecting to ftp.mozilla.org|63.245.215.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26832572 (26M) [application/x-bzip2]
Saving to: `firefox-18.0a1.en-US.linux-x86_64.tar.bz2'

     0K ........ ........ ........ ........ ........ ........ 11% 10.5M 2s
  3072K ........ ........ ........ ........ ........ ........ 23% 6.17M 3s
  6144K ........ ........ ........ ........ ........ ........ 35% 10.3M 2s
  9216K ........ ........ ........ ........ ........ ........ 46% 7.96M 2s
 12288K ........ ........ ........ ........ ........ ........ 58% 9.54M 1s
 15360K ........ ........ ........ ........ ........ ........ 70% 9.59M 1s
 18432K ........ ........ ........ ........ ........ ........ 82% 9.30M 1s
 21504K ........ ........ ........ ........ ........ ........ 93% 9.01M 0s
 24576K ........ ........ ........ .                         100% 5.63M=3.0s

2012-09-26 05:55:10 (8.51 MB/s) - `firefox-18.0a1.en-US.linux-x86_64.tar.bz2' saved [26832572/26832572]
}

Using --no-verbose cuts this out but stills allows errors to be shown.

Comment 1

[Chris AtLee [:catlee]]

Does --no-verbose still show the overall transfer rate, or at least bytes received and time?

Comment 2

[Ed Morley [:emorley]]

Attachment: Use wget --no-verbose

Comment 3

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #1)
> Does --no-verbose still show the overall transfer rate, or at least bytes
> received and time?

No, but the buildbot log shows the length of time for each download step, so do we still need those? (The main reason for log bloat is due to wget's progress bar, but sadly there isn't any way to disable just that but keep the final transfer time/size).

Comment 4

[Ed Morley [:emorley]]

(This obviously won't gain us as much as say bug 794453 or bug 794458, so I won't be too fussed if you'd rather not sacrifice the transfer rate info.)

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 664953 [details] [diff] [review]
Use wget --no-verbose

http://hg.mozilla.org/build/buildbotcustom/rev/625c327617d8

Comment 6

[Ed Morley [:emorley]]

I've backed this out, to help us diagnose bug 811063:
https://hg.mozilla.org/build/buildbotcustom/rev/7c1d5441001c