843787 - download_builds.sh wget retries don't write to the proper file

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P3)

Description

[John Hopkins (:jhopkins)]

When a wget call in download_builds.sh* fails, the original filename is not written to on subsequent retries due to wget not wanting to overwrite an existing file by default.

Log snippet of the problem:

--------------------
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x100511aa0> with args: (['wget', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox 20.0b1.dmg'], 300, None, None, False, True), kwargs: {}, attempt #2
Executing: ['wget', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox 20.0b1.dmg']
--21:21:55--  http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox%2020.0b1.dmg
           => `Firefox 20.0b1.dmg.1'
--------------------

Note the ".1" suffix.

* http://hg.mozilla.org/build/tools/file/d9d28485bdd4/release/common/download_builds.sh#l27

Comment 1

[[e-mail] (:bhearsum)]

Huh, I thought we killed this script. It's dead code AFAIK.

Comment 2

[John Hopkins (:jhopkins)]

Ok, I may have misidentified the source script.  We can rename this bug once it's found.

Comment 3

[[e-mail] (:bhearsum)]

I was confused, this script *is* used by update verify.

Comment 4

[John Hopkins (:jhopkins)]

The log I snipped from above is:

 ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/20.0b1-candidates/build2/logs/release-mozilla-beta-macosx64_update_verify_1-bm30-build1-build10.txt.gz

Comment 5

[[e-mail] (:bhearsum)]

There doesn't seem to be a way to prevent wget from appending the filename without specifying -O. I _think_ we could do that, because we use 'basename' to detect filenames: https://github.com/mozilla/build-tools/blob/master/release/updates/verify.sh#L143

Needs testing. I'm worried about things breaking on Windows if they don't have the right file extension.

Comment 6

[John Hopkins (:jhopkins)]

Attachment: [tools] give wget an explicit output filename

Local command-line testing passes, but we may want to do a staging run.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 746609 [details] [diff] [review]
[tools] give wget an explicit output filename

We could also nuke the $source_file, no?

Comment 8

[John Hopkins (:jhopkins)]

Local |wget -O| testing passed for Aki and me, so I think we're ok to land this when it's convenient.

Comment 9

[John Hopkins (:jhopkins)]

Comment on attachment 746609 [details] [diff] [review]
[tools] give wget an explicit output filename

Landed in https://hg.mozilla.org/build/tools/rev/5abb1bbf8239