download_builds.sh wget retries don't write to the proper file

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
P3
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: jhopkins, Assigned: jhopkins)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

5 years ago
When a wget call in download_builds.sh* fails, the original filename is not written to on subsequent retries due to wget not wanting to overwrite an existing file by default.

Log snippet of the problem:

--------------------
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x100511aa0> with args: (['wget', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox 20.0b1.dmg'], 300, None, None, False, True), kwargs: {}, attempt #2
Executing: ['wget', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox 20.0b1.dmg']
--21:21:55--  http://stage.mozilla.org/pub/mozilla.org//firefox/nightly/20.0b1-candidates/build2/mac/ca/Firefox%2020.0b1.dmg
           => `Firefox 20.0b1.dmg.1'
--------------------

Note the ".1" suffix.

* http://hg.mozilla.org/build/tools/file/d9d28485bdd4/release/common/download_builds.sh#l27
Huh, I thought we killed this script. It's dead code AFAIK.
(Assignee)

Comment 2

5 years ago
Ok, I may have misidentified the source script.  We can rename this bug once it's found.
Priority: -- → P3
I was confused, this script *is* used by update verify.
Priority: P3 → --
There doesn't seem to be a way to prevent wget from appending the filename without specifying -O. I _think_ we could do that, because we use 'basename' to detect filenames: https://github.com/mozilla/build-tools/blob/master/release/updates/verify.sh#L143

Needs testing. I'm worried about things breaking on Windows if they don't have the right file extension.
Priority: -- → P3
(Assignee)

Comment 6

5 years ago
Created attachment 746609 [details] [diff] [review]
[tools] give wget an explicit output filename

Local command-line testing passes, but we may want to do a staging run.
Attachment #746609 - Flags: review?(aki)

Comment 7

5 years ago
Comment on attachment 746609 [details] [diff] [review]
[tools] give wget an explicit output filename

We could also nuke the $source_file, no?
Attachment #746609 - Flags: review?(aki) → review+
(Assignee)

Comment 8

5 years ago
Local |wget -O| testing passed for Aki and me, so I think we're ok to land this when it's convenient.
(Assignee)

Updated

5 years ago
Assignee: nobody → jhopkins
(Assignee)

Comment 9

5 years ago
Comment on attachment 746609 [details] [diff] [review]
[tools] give wget an explicit output filename

Landed in https://hg.mozilla.org/build/tools/rev/5abb1bbf8239
Attachment #746609 - Flags: checked-in+
(Assignee)

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.