Incomplete download of Firefox binaries without HTTP error cause installation bustage

RESOLVED FIXED

Status

Cloud Services
Operations
P1
normal
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: whimboo, Assigned: oremj)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
This problem started some days ago and is visible on all platforms. Somehow the installer of Firefox is invalid.

12:18:29 Traceback (most recent call last):
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\firefox_ui_harness\runtests.py", line 66, in cli
12:18:29     runner = startTestRunner(runner_class, args, args.tests)
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\firefox_ui_harness\runtests.py", line 40, in startTestRunner
12:18:29     install_folder = mozinstall.install(installer, dest_folder)
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\mozinstall\mozinstall.py", line 106, in install
12:18:29     raise InvalidSource(src + ' is not valid installer file.')
12:18:29 
12:18:29 InvalidSource: c:\jenkins\workspace\mozilla-aurora_functional\firefox-44.0a2.en-US.win32.zip is not valid installer file.

Linux:

13:15:57 InstallError: Failed to install "/home/mozauto/jenkins/workspace/mozilla-aurora_update/firefox-43.0a2.en-US.linux-i686.tar.bz2 (compressed file ended before the logical end-of-stream was detected)"

OS X:

12:36:08 hdiutil: attach failed - not recognized


This breaks  lots of our tests. See the following for staging:

https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=bc4551debe17&filter-job_group_symbol=Ff&filter-job_group_symbol=Fr&filter-job_group_symbol=Fu
(Reporter)

Comment 1

3 years ago
I retriggered some of those jobs and it works now. Checking manually a workspace of a broken build indeed shows a Firefox DMG installer file of only 30MB. mozdownload ended normally without any failure message. So somehow we got only a part of the remote file.

Looks like we should get https://github.com/mozilla/mozdownload/issues/241 implemented to be warned by mozdownload about download corruption.
(Reporter)

Comment 2

3 years ago
Here the output from mozdownload:

 INFO | Saving as: /Users/mozauto/jenkins/workspace/mozilla-central_update/firefox-44.0a1.en-US.mac.dmg
  0% |                                              | ETA:  --:--:--   0.00 B/s
  1% |                                              | ETA:  01:36:18  18.60 K/s
  2% |                                              | ETA:  01:22:05  21.60 K/s
  3% |#                                             | ETA:  01:24:49  20.70 K/s
100% |##############################################| Time: 00:03:15 555.17 K/s
(Reporter)

Comment 3

3 years ago
Jeremy, can you please have a look at the comments above? Our jobs are downloading builds from archive.mozilla.org and earlier today all jobs failed due to invalid installer files. Are you aware of any download corruption with the new archive.mozilla.org host?
Flags: needinfo?(oremj)
(Reporter)

Updated

3 years ago
Component: Firefox UI Tests → Infrastructure
(Assignee)

Comment 4

3 years ago
The flow for this is: you -> cloudfront -> s3. To file a meaningful support case with AWS, I'll need the "X-Amz-Cf-Id" header for the failed request. Any curl output or traceroutes would probably also be helpful.
Flags: needinfo?(oremj)
(Assignee)

Comment 5

3 years ago
This is being tracked in bug 1170832.
(Reporter)

Updated

3 years ago
Depends on: 1170832
(Reporter)

Comment 6

3 years ago
So this is not only for Linux but all platforms. Adjusting summary and flags.

The situation got way better. I haven't seen a broken download yesterday and today so far. Jeremy, was there any update on routing?
OS: Unspecified → All
Hardware: Unspecified → All
Summary: Installing Firefox binary fails with "Invalid source: [..] not a valid installer file → Incomplete download of Firefox binaries without HTTP error cause installation bustage
(Assignee)

Comment 7

3 years ago
Looks like we are getting a different endpoint, but this isn't clear yet. I'll update when AWS tells us it is fixed.
(Reporter)

Comment 8

3 years ago
Sounds good. Thanks. Btw. I got another failure today for a Windows build. So it's indeed not fixed yet.

Comment 9

3 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 5

Platform breakdown:
* windows7-32: 1
* osx-10-9: 1
* osx-10-6: 1
* osx-10-11: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1219934&startday=2015-11-16&endday=2015-11-22&tree=all
(Reporter)

Updated

2 years ago
Component: Infrastructure → Operations
Product: Mozilla QA → Cloud Services
QA Contact: hskupin

Updated

2 years ago
Assignee: hskupin → oremj
(Reporter)

Updated

2 years ago
See Also: → bug 1231938
(Reporter)

Updated

2 years ago
See Also: bug 1231938
(Assignee)

Comment 10

2 years ago
How are things looking lately?
(Reporter)

Comment 11

2 years ago
I think the last time we have seen it was about 2 weeks ago. What has been changed meanwhile if you wanna share this information with us?
(Assignee)

Comment 12

2 years ago
Nothing on our side. This issue is all on AWS.
(Reporter)

Comment 13

2 years ago
So lets call it fixed then. I will reopen if it encounters again. Thanks for the feedback Jeremy.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.