Closed Bug 845563 Opened 12 years ago Closed 11 years ago

Halting on failure while running ['unzip', '-q', '-o', '/builds/slave/test/build/b2g-22.0a1.en-US.android-arm.tests.zip']

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ahal, Unassigned)

Details

(Keywords: intermittent-failure, Whiteboard: [mozharness]) 

See https://tbpl.mozilla.org/php/getParsedLog.php?id=20118553&tree=Mozilla-Inbound

Another chunk in the same push had a different problem downloading the tests.zip, so it may be network problems or something.

So far it's unclear whether other pushes are affected.
We can:

* do a checksum, either for a) FYI and postmortem debugging, or b) to compare against http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-ics_armv7a_gecko/1361914508/b2g-22.0a1.en-US.android-arm.checksums to make sure we think we've downloaded the right thing

* add download retries

* something else i haven't thought of

* a combination of things
Aki, Andrew: any updates?
We were prioritizing this as if it were a rare-occurrence bug.
From the comments here, looks like 2x every 2 weeks?  I wish it were rarer, but I'm also not sure it's higher priority than other bugs.
https://tbpl.mozilla.org/php/getParsedLog.php?id=20936503&tree=Firefox

Third time starring it today.
https://tbpl.mozilla.org/php/getParsedLog.php?id=21029260&tree=Mozilla-Inbound
OS: Gonk (Firefox OS) → All
Hardware: ARM → All
Product: mozilla.org → Release Engineering
https://tbpl.mozilla.org/php/getParsedLog.php?id=26758958&tree=Mozilla-Central#error1

{
03:07:58     INFO - #####
03:07:58     INFO - ##### Running download-and-extract step.
03:07:58     INFO - #####
03:07:58     INFO - Running pre-action listener: _resource_record_pre_action
03:07:58     INFO - Running main action method: download_and_extract
03:07:58     INFO - mkdir: /builds/slave/test/build
03:07:58     INFO - Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-pgo/1376982003/firefox-26.0a1.en-US.linux-x86_64.tests.zip to /builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip
03:07:58     INFO - retry: Calling <bound method DesktopUnittest._download_file of <__main__.DesktopUnittest object at 0x2d1bf10>> with args: ('http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-pgo/1376982003/firefox-26.0a1.en-US.linux-x86_64.tests.zip', '/builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip'), kwargs: {}, attempt #1
03:10:24     INFO - Downloaded 8396361 bytes.
03:10:24     INFO - mkdir: /builds/slave/test/build/tests
03:10:24     INFO - Running command: ['unzip', '-q', '-o', '/builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip', 'bin/*', 'certs/*', 'modules/*', 'mozbase/*', 'config/*', 'mochitest/*'] in /builds/slave/test/build/tests
03:10:24     INFO - Copy/paste: unzip -q -o /builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip bin/* certs/* modules/* mozbase/* config/* mochitest/*
03:10:24     INFO -  [/builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip]
03:10:24     INFO -    End-of-central-directory signature not found.  Either this file is not
03:10:24     INFO -    a zipfile, or it constitutes one disk of a multi-part archive.  In the
03:10:24     INFO -    latter case the central directory and zipfile comment will be found on
03:10:24     INFO -    the last disk(s) of this archive.
03:10:24     INFO -  unzip:  cannot find zipfile directory in one of /builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip or
03:10:24     INFO -          /builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip.zip, and cannot find /builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip.ZIP, period.
03:10:24    ERROR - Return code: 9
03:10:24    FATAL - Halting on failure while running ['unzip', '-q', '-o', '/builds/slave/test/build/firefox-26.0a1.en-US.linux-x86_64.tests.zip', 'bin/*', 'certs/*', 'modules/*', 'mozbase/*', 'config/*', 'mochitest/*']
03:10:24    FATAL - Running post_fatal callback...
03:10:24    FATAL - Exiting 9
03:10:24     INFO - Running post-action listener: _resource_record_post_action
03:10:24     INFO - Running post-run listener: _resource_record_post_run
program finished with exit code 9
elapsedTime=147.716420
========= Finished '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' failed (results: 2, elapsed: 2 mins, 57 secs) (at 2013-08-20 03:10:53.678460) =========
}
We now check content length before and after download, and error out if it doesn't match: http://hg.mozilla.org/build/mozharness/file/1ac44c9974e7/mozharness/base/script.py#l181

No comments in here since last August, and this landed in December.
Resolving with hope.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.