Closed Bug 1219934 Opened 9 years ago Closed 8 years ago

Incomplete download of Firefox binaries without HTTP error cause installation bustage

Categories

(Cloud Services :: Operations: Miscellaneous, task, P1)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: whimboo, Assigned: oremj)

References

Details

This problem started some days ago and is visible on all platforms. Somehow the installer of Firefox is invalid.

12:18:29 Traceback (most recent call last):
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\firefox_ui_harness\runtests.py", line 66, in cli
12:18:29     runner = startTestRunner(runner_class, args, args.tests)
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\firefox_ui_harness\runtests.py", line 40, in startTestRunner
12:18:29     install_folder = mozinstall.install(installer, dest_folder)
12:18:29 
12:18:29   File "c:\jenkins\workspace\mozilla-aurora_functional\tests_venv\lib\site-packages\mozinstall\mozinstall.py", line 106, in install
12:18:29     raise InvalidSource(src + ' is not valid installer file.')
12:18:29 
12:18:29 InvalidSource: c:\jenkins\workspace\mozilla-aurora_functional\firefox-44.0a2.en-US.win32.zip is not valid installer file.

Linux:

13:15:57 InstallError: Failed to install "/home/mozauto/jenkins/workspace/mozilla-aurora_update/firefox-43.0a2.en-US.linux-i686.tar.bz2 (compressed file ended before the logical end-of-stream was detected)"

OS X:

12:36:08 hdiutil: attach failed - not recognized


This breaks  lots of our tests. See the following for staging:

https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=bc4551debe17&filter-job_group_symbol=Ff&filter-job_group_symbol=Fr&filter-job_group_symbol=Fu
I retriggered some of those jobs and it works now. Checking manually a workspace of a broken build indeed shows a Firefox DMG installer file of only 30MB. mozdownload ended normally without any failure message. So somehow we got only a part of the remote file.

Looks like we should get https://github.com/mozilla/mozdownload/issues/241 implemented to be warned by mozdownload about download corruption.
Here the output from mozdownload:

 INFO | Saving as: /Users/mozauto/jenkins/workspace/mozilla-central_update/firefox-44.0a1.en-US.mac.dmg
  0% |                                              | ETA:  --:--:--   0.00 B/s
  1% |                                              | ETA:  01:36:18  18.60 K/s
  2% |                                              | ETA:  01:22:05  21.60 K/s
  3% |#                                             | ETA:  01:24:49  20.70 K/s
100% |##############################################| Time: 00:03:15 555.17 K/s
Jeremy, can you please have a look at the comments above? Our jobs are downloading builds from archive.mozilla.org and earlier today all jobs failed due to invalid installer files. Are you aware of any download corruption with the new archive.mozilla.org host?
Flags: needinfo?(oremj)
Component: Firefox UI Tests → Infrastructure
The flow for this is: you -> cloudfront -> s3. To file a meaningful support case with AWS, I'll need the "X-Amz-Cf-Id" header for the failed request. Any curl output or traceroutes would probably also be helpful.
Flags: needinfo?(oremj)
This is being tracked in bug 1170832.
Depends on: 1170832
So this is not only for Linux but all platforms. Adjusting summary and flags.

The situation got way better. I haven't seen a broken download yesterday and today so far. Jeremy, was there any update on routing?
OS: Unspecified → All
Hardware: Unspecified → All
Summary: Installing Firefox binary fails with "Invalid source: [..] not a valid installer file → Incomplete download of Firefox binaries without HTTP error cause installation bustage
Looks like we are getting a different endpoint, but this isn't clear yet. I'll update when AWS tells us it is fixed.
Sounds good. Thanks. Btw. I got another failure today for a Windows build. So it's indeed not fixed yet.
Component: Infrastructure → Operations
Product: Mozilla QA → Cloud Services
QA Contact: hskupin
Assignee: hskupin → oremj
See Also: → 1231938
See Also: 1231938
How are things looking lately?
I think the last time we have seen it was about 2 weeks ago. What has been changed meanwhile if you wanna share this information with us?
Nothing on our side. This issue is all on AWS.
So lets call it fixed then. I will reopen if it encounters again. Thanks for the feedback Jeremy.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.