1443036 - Deved 60.0b1/build2 update verify failing because of slow downloads

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Andreea Pavel [:apavel]]

+++ This bug was initially created as a clone of Bug #1430535 +++


Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=165582526&repo=mozilla-beta&lineNumber=873

FINISH REMOVEFILE api-ms-win-core-console-l1-1-0.dll
FINISH REMOVEFILE IA2Marshal.dll
FINISH REMOVEFILE AccessibleMarshal.dll
FINISH REMOVEFILE AccessibleHandler.dll
FINISH REMOVEFILE Accessible.tlb
FINISH REMOVEDIR uninstall/
non-fatal error removing directory: uninstall/, rv: 0, err: 39
FINISH REMOVEDIR gmp-clearkey/0.1/
non-fatal error removing directory: gmp-clearkey/0.1/, rv: 0, err: 39
FINISH REMOVEDIR gmp-clearkey/
non-fatal error removing directory: gmp-clearkey/, rv: 0, err: 39
FINISH REMOVEDIR fonts/
non-fatal error removing directory: fonts/, rv: 0, err: 39
FINISH REMOVEDIR dictionaries/
non-fatal error removing directory: dictionaries/, rv: 0, err: 39
FINISH REMOVEDIR defaults/pref/
non-fatal error removing directory: defaults/pref/, rv: 0, err: 39
FINISH REMOVEDIR defaults/
non-fatal error removing directory: defaults/, rv: 0, err: 39
FINISH REMOVEDIR browser/features/
non-fatal error removing directory: browser/features/, rv: 0, err: 39
FINISH REMOVEDIR browser/extensions/
non-fatal error removing directory: browser/extensions/, rv: 0, err: 39
FINISH REMOVEDIR browser/VisualElements/
non-fatal error removing directory: browser/VisualElements/, rv: 0, err: 39
FINISH REMOVEDIR browser/
non-fatal error removing directory: browser/, rv: 0, err: 39
FINISH ADD xul.dll.sig
FINISH ADD xul.dll

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This looks unrelated to bug 1430535, and treeherder hasn't picked out the actual error:

Executing: ['wget', '-O', 'update/complete.mar', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://archive.mozilla.org/pub/devedition/candidates/60.0b1-candidates/build2/update/win64/hsb/firefox-60.0b1.complete.mar']
--2018-03-02 18:38:21--  http://archive.mozilla.org/pub/devedition/candidates/60.0b1-candidates/build2/update/win64/hsb/firefox-60.0b1.complete.mar
Resolving archive.mozilla.org (archive.mozilla.org)... 52.84.18.253
Connecting to archive.mozilla.org (archive.mozilla.org)|52.84.18.253|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Content-Type: application/octet-stream
  Content-Length: 42043644
  Connection: keep-alive
  Date: Fri, 02 Mar 2018 18:28:11 GMT
  x-amz-replication-status: COMPLETED
  Last-Modified: Fri, 02 Mar 2018 17:12:49 GMT
  ETag: "eb7663dd91960311257054a1d898e852"
  Cache-Control: public, max-age=14400
  x-amz-version-id: IabH3xG4Z9VqDgiLjB_Clmcp.4YQNzuP
  Accept-Ranges: bytes
  Server: AmazonS3
  Age: 612
  X-Cache: Hit from cloudfront
  Via: 1.1 97c5dcf9c391508cdebbbfdcf304912f.cloudfront.net (CloudFront)
  X-Amz-Cf-Id: zpNiBuL5YL4IwKPIqA4F03VpvBQVw37ndxTzpejCknbMNdKODMpiKQ==
Length: 42043644 (40M) [application/octet-stream]
Saving to: ‘update/complete.mar’

     0K ........ ........ ........ ........ ........ ........  7% 48.6M 1s
  3072K ........ ........ ........ ........ ........ ........ 14% 49.9M 1s
  6144K ........ ........ ........ ........ ........ ........ 22% 52.3M 1s
  9216K ........ ........ ........ ........ ........ ........ 29% 90.6K 80s
 12288K ........ ........ ........ ........ .......WARNING: Timeout (300) exceeded, killing process 6296
retry: Giving up on <function run_with_timeout at 0x7f7344d8aa28>
Unable to successfully run ['wget', '-O', 'update/complete.mar', '--progress=dot:mega', '--server-response', '--no-check-certificate', 'http://archive.mozilla.org/pub/devedition/candidates/60.0b1-candidates/build2/update/win64/hsb/firefox-60.0b1.complete.mar'] after 3 attempts
Could not download complete!
from: http://archive.mozilla.org/pub/devedition/candidates/60.0b1-candidates/build2/update/win64/hsb/firefox-60.0b1.complete.mar
FAIL: complete from https://aus5.mozilla.org/update/3/Firefox/56.0/20170914024831/WINNT_x86_64-msvc-x64/hsb/aurora-localtest/default/default/default/update.xml?force=1 wrong size
FAIL: update.xml size: 42043644
FAIL: actual size: 15148856
FAIL: [56.0 hsb complete] download_mars returned non-zero exit code: 1

aka - slow network throughput resulting in hitting a 5 minute timeout. The jobs were already rerun and were green, and build2 was replaced by build3.