The default bug view has changed. See this FAQ.

Unzip on developer's Mac do not like test.zip files produced in releng infra

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: armenzg, Assigned: armenzg, Mentored)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [easier-mozharness][good first bug] workaround in comment 4)

Attachments

(3 attachments, 1 obsolete attachment)

This bug prevents developers to work with mozharness for b2g desktop jobs.

mdas has UnZip 5.52 of 28 February 2005.
I have the same version in my Mac.
I can reproduce the issue.

It seems that we use 6.0:
http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/unzip.pp#18

Perhaps related to this:
http://spbutterworth.com/2014/07/07/zipper-frustration/

    (venv)[mdas@Malinis-MacBook-Pro mozharness]$ python scripts/marionette.py --cfg configs/marionette/gaia_ui_test_prod_config.py --blob-upload-branch b2g-inbound --download-symbols true --cfg configs/developer_config.py --installer-url ftp://ftp.mozilla.org/pub/b2g/nightly/latest-mozilla-central/b2g-35.0a1.multi.mac64.dmg --installer-path installer_path/
    11:47:47     INFO - MultiFileLogger online at 20140912 11:47:47 in /Users/mdas/Code/mozharness
    11:47:47     INFO - Run as scripts/marionette.py --cfg configs/marionette/gaia_ui_test_prod_config.py --blob-upload-branch b2g-inbound --download-symbols true --cfg configs/developer_config.py --installer-url ftp://ftp.mozilla.org/pub/b2g/nightly/latest-mozilla-central/b2g-35.0a1.multi.mac64.dmg --installer-path installer_path/
    11:47:47     INFO - Dumping config to /Users/mdas/Code/mozharness/logs/localconfig.json.
    11:47:47     INFO - {'append_to_log': False,
    11:47:47     INFO -  'application': 'b2g',
    11:47:47     INFO -  'base_work_dir': '/Users/mdas/Code/mozharness',
    11:47:47     INFO -  'blob_upload_branch': 'b2g-inbound',
    11:47:47     INFO -  'blob_uploader_auth_file': '/Users/mdas/Code/mozharness/oauth.txt',
    11:47:47     INFO -  'buildbot_json_path': 'buildprops.json',
    11:47:47     INFO -  'config_files': ('configs/marionette/gaia_ui_test_prod_config.py',
    11:47:47     INFO -                   'configs/developer_config.py'),
    11:47:47     INFO -  'default_actions': ('clobber',
    11:47:47     INFO -                      'read-buildbot-config',
    11:47:47     INFO -                      'pull',
    11:47:47     INFO -                      'download-and-extract',
    11:47:47     INFO -                      'create-virtualenv',
    11:47:47     INFO -                      'install',
    11:47:47     INFO -                      'run-marionette'),
    11:47:47     INFO -  'default_blob_upload_servers': ('https://blobupload.elasticbeanstalk.com',),
    11:47:47     INFO -  'download_minidump_stackwalk': True,
    11:47:47     INFO -  'download_symbols': 'true',
    11:47:47     INFO -  'exes': {},
    11:47:47     INFO -  'find_links': ('http://pypi.pub.build.mozilla.org/pub',),
    11:47:47     INFO -  'gaia_branch': 'default',
    11:47:47     INFO -  'gaia_repo': 'https://hg.mozilla.org/integration/gaia-central',
    11:47:47     INFO -  'gaiatest': True,
    11:47:47     INFO -  'in_tree_config': 'config/mozharness/marionette.py',
    11:47:47     INFO -  'installer_path': 'installer_path/',
    11:47:47     INFO -  'installer_url': 'ftp://ftp.mozilla.org/pub/b2g/nightly/latest-mozilla-central/b2g-35.0a1.multi.mac64.dmg',
    11:47:47     INFO -  'log_level': 'info',
    11:47:47     INFO -  'log_to_console': True,
    11:47:47     INFO -  'marionette_address': 'localhost:2828',
    11:47:47     INFO -  'opt_config_files': (),
    11:47:47     INFO -  'pip_index': False,
    11:47:47     INFO -  'replace_urls': (('http://pvtbuilds.pvt.build', 'https://pvtbuilds'),
    11:47:47     INFO -                   ('http://runtime-binaries.pvt.build.mozilla.org/tooltool',
    11:47:47     INFO -                    'https://secure.pub.build.mozilla.org/tooltool/pvt/build')),
    11:47:47     INFO -  'require_test_zip': True,
    11:47:47     INFO -  'structured_output': False,
    11:47:47     INFO -  'test_manifest': 'unit-tests.ini',
    11:47:47     INFO -  'test_type': 'b2g',
    11:47:47     INFO -  'tooltool_servers': ('https://secure.pub.build.mozilla.org/tooltool/pvt/build',),
    11:47:47     INFO -  'vcs_output_timeout': 1760,
    11:47:47     INFO -  'vcs_share_base': '/builds/hg-shared',
    11:47:47     INFO -  'virtualenv_path': 'venv',
    11:47:47     INFO -  'volatile_config': {'actions': None, 'add_actions': None, 'no_actions': None},
    11:47:47     INFO -  'work_dir': 'build',
    11:47:47     INFO -  'xre_path': 'xulrunner-sdk',
    11:47:47     INFO -  'xre_url': 'http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/ce54c2f5abb8b8bae67b129e1218e3c3fe46ed965992987b991d5b401e47db07a894da203978a710ea5498321c3b3d39538db2bca7db230c326af6ff060d7c98'}
    11:47:47     INFO - #####
    11:47:47     INFO - ##### Running clobber step.
    11:47:47     INFO - #####
    11:47:47     INFO - Running pre-action listener: _resource_record_pre_action
    11:47:47     INFO - Running main action method: clobber
    11:47:47     INFO - rmtree: /Users/mdas/Code/mozharness/build
    11:47:47     INFO - retry: Calling <function rmtree at 0x10482cb18> with args: ('/Users/mdas/Code/mozharness/build',), kwargs: {}, attempt #1
    11:47:51     INFO - Running post-action listener: _resource_record_post_action
    11:47:51     INFO - #####
    11:47:51     INFO - ##### Running read-buildbot-config step.
    11:47:51     INFO - #####
    11:47:51     INFO - Running pre-action listener: _resource_record_pre_action
    11:47:51     INFO - Running main action method: read_buildbot_config
    11:47:51     INFO - Using buildbot properties:
    11:47:51     INFO - {
    11:47:51     INFO -     "properties": {
    11:47:51     INFO -         "project": "",
    11:47:51     INFO -         "product": "b2g",
    11:47:51     INFO -         "script_repo_revision": "production",
    11:47:51     INFO -         "scheduler": "tests-mozilla-inbound-mountainlion-b2gdt-opt-unittest",
    11:47:51     INFO -         "repository": "",
    11:47:51     INFO -         "buildername": "b2g_macosx64 mozilla-inbound opt test gaia-ui-test",
    11:47:51     INFO -         "buildid": "20140912071351",
    11:47:51     INFO -         "pgo_build": "False",
    11:47:51     INFO -         "basedir": "/builds/slave/talos-slave/test",
    11:47:51     INFO -         "buildnumber": 2660,
    11:47:51     INFO -         "slavename": "talos-mtnlion-r5-012",
    11:47:51     INFO -         "master": "http://buildbot-master106.srv.releng.scl3.mozilla.com:8201/",
    11:47:51     INFO -         "platform": "macosx64_gecko",
    11:47:51     INFO -         "branch": "mozilla-inbound",
    11:47:51     INFO -         "repo_path": "integration/mozilla-inbound",
    11:47:51     INFO -         "revision": "c599b5ffe376",
    11:47:51     INFO -         "stage_platform": "macosx64_gecko",
    11:47:51     INFO -         "builduid": "215141479f1e483dbd65d3e6befea359",
    11:47:51     INFO -         "slavebuilddir": "test"
    11:47:51     INFO -     },
    11:47:51     INFO -     "sourcestamp": {
    11:47:51     INFO -         "repository": "",
    11:47:51     INFO -         "hasPatch": false,
    11:47:51     INFO -         "project": "",
    11:47:51     INFO -         "branch": "mozilla-inbound-macosx64_gecko-opt-unittest",
    11:47:51     INFO -         "changes": [
    11:47:51     INFO -             {
    11:47:51     INFO -                 "category": null,
    11:47:51     INFO -                 "files": [
    11:47:51     INFO -                     {
    11:47:51     INFO -                         "url": null,
    11:47:51     INFO -                         "name": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410530571/en-US/b2g-35.0a1.en-US.mac64.dmg"
    11:47:51     INFO -                     },
    11:47:51     INFO -                     {
    11:47:51     INFO -                         "url": null,
    11:47:51     INFO -                         "name": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410530571/en-US/b2g-35.0a1.en-US.mac64.tests.zip"
    11:47:51     INFO -                     }
    11:47:51     INFO -                 ],
    11:47:51     INFO -                 "repository": "",
    11:47:51     INFO -                 "rev": "55b1104e04f4",
    11:47:51     INFO -                 "who": "sendchange-unittest",
    11:47:51     INFO -                 "when": 1410532403,
    11:47:51     INFO -                 "number": 4501679,
    11:47:51     INFO -                 "comments": "Bug 1020039 - Intermittent relations/test_embeds.xul",
    11:47:51     INFO -                 "project": "",
    11:47:51     INFO -                 "at": "Fri 12 Sep 2014 07:33:23",
    11:47:51     INFO -                 "branch": "mozilla-inbound-macosx64_gecko-opt-unittest",
    11:47:51     INFO -                 "revlink": "",
    11:47:51     INFO -                 "properties": [
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "buildid",
    11:47:51     INFO -                         "20140912070251",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ],
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "builduid",
    11:47:51     INFO -                         "3be1ed857ead47b7a533ff2919fd7c5d",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ],
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "pgo_build",
    11:47:51     INFO -                         "False",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ]
    11:47:51     INFO -                 ],
    11:47:51     INFO -                 "revision": "55b1104e04f4"
    11:47:51     INFO -             },
    11:47:51     INFO -             {
    11:47:51     INFO -                 "category": null,
    11:47:51     INFO -                 "files": [
    11:47:51     INFO -                     {
    11:47:51     INFO -                         "url": null,
    11:47:51     INFO -                         "name": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.dmg"
    11:47:51     INFO -                     },
    11:47:51     INFO -                     {
    11:47:51     INFO -                         "url": null,
    11:47:51     INFO -                         "name": "http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip"
    11:47:51     INFO -                     }
    11:47:51     INFO -                 ],
    11:47:51     INFO -                 "repository": "",
    11:47:51     INFO -                 "rev": "c599b5ffe376",
    11:47:51     INFO -                 "who": "sendchange-unittest",
    11:47:51     INFO -                 "when": 1410532742,
    11:47:51     INFO -                 "number": 4501715,
    11:47:51     INFO -                 "comments": "Bug 1055402 - Remove RegExp.test cache, add a more robust match only mode to irregexp, r=jandem.",
    11:47:51     INFO -                 "project": "",
    11:47:51     INFO -                 "at": "Fri 12 Sep 2014 07:39:02",
    11:47:51     INFO -                 "branch": "mozilla-inbound-macosx64_gecko-opt-unittest",
    11:47:51     INFO -                 "revlink": "",
    11:47:51     INFO -                 "properties": [
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "buildid",
    11:47:51     INFO -                         "20140912071351",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ],
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "builduid",
    11:47:51     INFO -                         "215141479f1e483dbd65d3e6befea359",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ],
    11:47:51     INFO -                     [
    11:47:51     INFO -                         "pgo_build",
    11:47:51     INFO -                         "False",
    11:47:51     INFO -                         "Change"
    11:47:51     INFO -                     ]
    11:47:51     INFO -                 ],
    11:47:51     INFO -                 "revision": "c599b5ffe376"
    11:47:51     INFO -             }
    11:47:51     INFO -         ],
    11:47:51     INFO -         "revision": "c599b5ffe376"
    11:47:51     INFO -     }
    11:47:51     INFO - }
    11:47:51     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip.
    11:47:51     INFO - Running post-action listener: _resource_record_post_action
    11:47:51     INFO - #####
    11:47:51     INFO - ##### Running pull step.
    11:47:51     INFO - #####
    11:47:51     INFO - Running pre-action listener: _resource_record_pre_action
    11:47:51     INFO - Running main action method: pull
    11:47:51     INFO - dest: /Users/mdas/Code/mozharness/gaia
    11:47:51     INFO - retry: Calling <bound method MarionetteTest.load_json_from_url of <__main__.MarionetteTest object at 0x104aa93d0>> with args: ('https://hg.mozilla.org/integration/mozilla-inbound/raw-file/c599b5ffe376/b2g/config/gaia.json',), kwargs: {}, attempt #1
    11:47:52     INFO - Running command: ['hg', '--config', 'extensions.purge=', 'purge'] in /Users/mdas/Code/mozharness/gaia
    11:47:52     INFO - Copy/paste: hg --config extensions.purge= purge
    11:47:53     INFO - Return code: 0
    11:47:53     INFO - Changing directory to /Users/mdas/Code/mozharness.
    11:47:53     INFO - retry: Calling <bound method MarionetteTest._get_revision of <__main__.MarionetteTest object at 0x104aa93d0>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x104ac20d0>, '/Users/mdas/Code/mozharness/gaia'), kwargs: {}, attempt #1
    11:47:53     INFO - Setting /Users/mdas/Code/mozharness/gaia to https://hg.mozilla.org//integration/gaia-central revision 51bb0dde2b9800784dc6b4688eb8108aa18de765 using shared directory /builds/hg-shared.
    11:47:53     INFO - Checking if share extension works.
    11:47:53     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'help', 'share']
    11:47:53     INFO - Copy/paste: hg --config ui.merge=internal:merge help share
    11:47:53     INFO - Reading from file tmpfile_stdout
    11:47:53  WARNING - Disabling sharing since share extension doesn't seem to work (2)
    11:47:53     INFO - Pulling https://hg.mozilla.org//integration/gaia-central to /Users/mdas/Code/mozharness/gaia and updating.
    11:47:53     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'pull', u'https://hg.mozilla.org//integration/gaia-central'] in /Users/mdas/Code/mozharness/gaia
    11:47:53     INFO - Copy/paste: hg --config ui.merge=internal:merge pull https://hg.mozilla.org//integration/gaia-central
    11:47:53     INFO - Calling ['hg', '--config', 'ui.merge=internal:merge', 'pull', u'https://hg.mozilla.org//integration/gaia-central'] with output_timeout 1760
    11:47:53     INFO -  warning: hg.mozilla.org certificate with fingerprint af:27:b9:34:47:4e:e5:98:01:f6:83:2b:51:c9:aa:d8:df:fb:1a:27 not verified (check hostfingerprints or web.cacerts config setting)
    11:47:55     INFO -  pulling from https://hg.mozilla.org//integration/gaia-central
    11:47:55     INFO -  searching for changes
    11:47:55     INFO -  adding changesets
    11:47:55     INFO -  adding manifests
    11:47:55     INFO -  adding file changes
    11:47:55     INFO -  added 5 changesets with 9 changes to 8 files
    11:47:55     INFO -  (run 'hg update' to get a working copy)
    11:47:55     INFO - Return code: 0
    11:47:55     INFO - Updating /Users/mdas/Code/mozharness/gaia revision 51bb0dde2b9800784dc6b4688eb8108aa18de765.
    11:47:55     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C', '-r', u'51bb0dde2b9800784dc6b4688eb8108aa18de765'] in /Users/mdas/Code/mozharness/gaia
    11:47:55     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C -r 51bb0dde2b9800784dc6b4688eb8108aa18de765
    11:47:56     INFO -  206 files updated, 0 files merged, 23 files removed, 0 files unresolved
    11:47:56     INFO - Return code: 0
    11:47:56     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node|short}'] in /Users/mdas/Code/mozharness/gaia
    11:47:56     INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node|short}
    11:47:56     INFO - Reading from file tmpfile_stdout
    11:47:56     INFO - Output received:
    11:47:56     INFO -  51bb0dde2b98
    11:47:56     INFO - Updating /Users/mdas/Code/mozharness/gaia revision 51bb0dde2b9800784dc6b4688eb8108aa18de765.
    11:47:56     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C', '-r', u'51bb0dde2b9800784dc6b4688eb8108aa18de765'] in /Users/mdas/Code/mozharness/gaia
    11:47:56     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C -r 51bb0dde2b9800784dc6b4688eb8108aa18de765
    11:47:56     INFO -  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
    11:47:56     INFO - Return code: 0
    11:47:56     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'parent', '--template', '{node|short}'] in /Users/mdas/Code/mozharness/gaia
    11:47:56     INFO - Copy/paste: hg --config ui.merge=internal:merge parent --template {node|short}
    11:47:57     INFO - Reading from file tmpfile_stdout
    11:47:57     INFO - Output received:
    11:47:57     INFO -  51bb0dde2b98
    11:47:57     INFO - Changing directory to /Users/mdas/Code/mozharness.
    11:47:57     INFO - Pull has nothing to do!
    11:47:57     INFO - Running post-action listener: _resource_record_post_action
    11:47:57     INFO - #####
    11:47:57     INFO - ##### Running download-and-extract step.
    11:47:57     INFO - #####
    11:47:57     INFO - Running pre-action listener: _resource_record_pre_action
    11:47:57     INFO - Running main action method: download_and_extract
    11:47:57     INFO - Replacing url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip -> https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip
    11:47:57     INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy.srv.releng.use1.mozilla.com', 'proxxy.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pvtbuilds.pvt.build.mozilla.org', 'pvtbuilds.mozilla.org'), ('http://runtime-binaries.pvt.build.mozilla.org', 'runtime-binaries.pvt.build.mozilla.org')]}
    11:47:57     INFO - https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip matches https://ftp-ssl.mozilla.org
    11:47:57     INFO - trying https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip
    11:47:57     INFO - mkdir: /Users/mdas/Code/mozharness/build
    11:47:57     INFO - Downloading https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip to /Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip
    11:47:57     INFO - retry: Calling <bound method MarionetteTest._download_file of <__main__.MarionetteTest object at 0x104aa93d0>> with args: ('https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-macosx64_gecko/1410531231/en-US/b2g-35.0a1.en-US.mac64.tests.zip', '/Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip'), kwargs: {}, attempt #1
    11:49:06     INFO - Downloaded 114939629 bytes.
    11:49:06     INFO - mkdir: /Users/mdas/Code/mozharness/build/tests
    11:49:06     INFO - Running command: ['unzip', '-q', '-o', '/Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip'] in /Users/mdas/Code/mozharness/build/tests
    11:49:06     INFO - Copy/paste: unzip -q -o /Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip
    11:49:06     INFO -  warning [/Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip]:  76 extra bytes at beginning or within zipfile
    11:49:06     INFO -    (attempting to process anyway)
    11:49:06     INFO -  error [/Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip]:  reported length of central directory is
    11:49:06     INFO -    -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
    11:49:06     INFO -    zipfile?).  Compensating...
    11:49:18     INFO -  error:  expected central file header signature not found (file #66983).
    11:49:18     INFO -    (please check that you have transferred or created the zipfile in the
    11:49:18     INFO -    appropriate BINARY mode and that you have compiled UnZip properly)
    11:49:18    ERROR - Return code: 3
    11:49:18    ERROR - 3 not in success codes: [0, 11]
    11:49:18    FATAL - Halting on failure while running ['unzip', '-q', '-o', '/Users/mdas/Code/mozharness/build/b2g-35.0a1.en-US.mac64.tests.zip']
    11:49:18    FATAL - Running post_fatal callback...
    11:49:18    FATAL - Exiting 3
    11:49:18     INFO - Running post-action listener: _resource_record_post_action
    11:49:18     INFO - Running post-run listener: _resource_record_post_run
    11:49:18     INFO - Running post-run listener: _upload_blobber_files
    11:49:18     INFO - Blob upload gear active.
    11:49:18  WARNING - Blob upload directory does not exist!
This is probably to a code change that requires updating the requirements on the developers' machine:
bug 1032391
At the moment, this is preventing mdas to work locally on https://bugzilla.mozilla.org/show_bug.cgi?id=1066187
See Also: → bug 1066187

Comment 3

3 years ago
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #1)
> This is probably to a code change that requires updating the requirements on
> the developers' machine:
> bug 1032391

yup, IIRC our test.zip was getting too big with web-platform-tests so we needed to use lib64 for osx like all our other platforms have. the shipped version on osx is old
<jlund> armenzg: this is the script I used to create the dmg in automation: http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/unzip-dmg.sh that dmg is here: http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/unzip-6.0.dmg it w*should* work for all osx versions
<jlund> warning: although beware this will replace /usr/bin/unzip (currently 5.52). as an alternative, you can do it manually via: http://www.info-zip.org/UnZip.html or http://sourceforge.net/projects/infozip/files/UnZip%206.x%20(latest)/UnZip%206.0/
Another user hit this issue when trying to reproduce mozharness locally.
Mentor: armenzg@mozilla.com
Whiteboard: [good first bug]
(Assignee)

Updated

3 years ago
Whiteboard: [good first bug] → [good first bug] workaround in comment 4
Created attachment 8495993 [details]
output of mozharness mac test job
Created attachment 8496057 [details] [diff] [review]
mozharness_unzip_version.diff

The dmg did not work for me.
I would double click and nothing would happen.

I ended up upgrading to it with:
http://www.macupdate.com/app/mac/35967/unzip

I tested this patch with both versions.
Attachment #8496057 - Flags: review?(jlund)

Comment 8

3 years ago
Comment on attachment 8496057 [details] [diff] [review]
mozharness_unzip_version.diff

Review of attachment 8496057 [details] [diff] [review]:
-----------------------------------------------------------------

awesome! lgtm. see my usual nitpicking personality come out in lines below.

my main concern is if you meant to check for 'Darin' is os.uname as I think os.uname will pretty much hold a value for all platforms.

::: mozharness/mozilla/testing/testbase.py
@@ +96,5 @@
>      jsshell_url = None
>      minidump_stackwalk_path = None
>      default_tools_repo = 'https://hg.mozilla.org/build/tools'
>      proxxy = None
> +    developer_mode = False

I think this would be better if it was configured.

i.e it's in developer_config.py so you just need to:
if c.get('developer_mode')

that way it can be used everywhere without any added script/mixin lines like the ones in this patch

@@ +291,5 @@
> +    def _query_unzip_version(self):
> +        cmd = [self.query_exe('unzip'), '-v']
> +        output = self.get_output_from_command(cmd, silent=True)
> +        try:
> +            version = output.splitlines()[0].split()[1]

seems fragile. is the output the same across platform, platform versions, and unzip versions?

I know we don't have a better way ATM and since this is only for local dev stuff where this is better than nothing, I'm OK with this.

@@ +292,5 @@
> +        cmd = [self.query_exe('unzip'), '-v']
> +        output = self.get_output_from_command(cmd, silent=True)
> +        try:
> +            version = output.splitlines()[0].split()[1]
> +            float(version)

maybe combine the above two lines so:

version = float(output.splitlines()[0].split()[1])
then... if version > 6.0

as it is now, you are checking if str > str, which I'm not sure how well comparing strings will fair in odd scenarios. Feel free to tell me it's fine the way it is. :)

@@ +299,5 @@
> +            self.fatal("Unzip returns a float number on the second word of its first line of output.\n" \
> +                    "However, we were unable to determine the version\n" \
> +                    "Please review this output and/or try this command: %s\n%s" % (cmd, output))
> +        except:
> +            self.fatal("_query_unzip_version() was unable to determine the version of unzip." \

maybe error in both these cases and say we can't verify if your unzip version is greater than 6.0? see argument both ways.

@@ +325,5 @@
>  """
>          if message:
>              self.fatal(message + "Can't run download-and-extract... exiting")
>  
> +        if self.developer_mode and os.uname()[0]:

os.uname()[0] == 'Darwin' ?
Created attachment 8496159 [details] [diff] [review]
mozharness_unzip_version.diff

This should deal with the issues of the previous patch.
Attachment #8496057 - Attachment is obsolete: true
Attachment #8496057 - Flags: review?(jlund)
Attachment #8496159 - Flags: review?(jlund)
Comment on attachment 8496159 [details] [diff] [review]
mozharness_unzip_version.diff

Review of attachment 8496159 [details] [diff] [review]:
-----------------------------------------------------------------

awesome. if it works for you, it works for me :)

::: mozharness/mozilla/testing/testbase.py
@@ +316,5 @@
>  
> +        if self.config.get("developer_mode") and os.uname()[0] == 'Darwin':
> +            # Bug 1066700 only affects Mac users that try to run mozharness locally
> +            version = self._query_binary_version(
> +                    regex=re.compile("UnZip\ (\d+\.\d+)\ .*",re.MULTILINE),

fancy
Attachment #8496159 - Flags: review?(jlund) → review+
Comment on attachment 8496159 [details] [diff] [review]
mozharness_unzip_version.diff

https://hg.mozilla.org/build/mozharness/rev/143f996273bb
Attachment #8496159 - Flags: checked-in+
This only affects developers.
Closing since we don't have to wait until it merges to production.
Assignee: nobody → armenzg
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Bob is having trouble running this code on Windows.
It seems that uname is not shared across platforms [1]

http://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/testing/testbase.py#l324
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 8497626 [details] [diff] [review]
mozharness_darwin.diff

I haven't tested it yet but it should work.

On Windows it returns "win32", on Linux it return "linux2" and on Mac it returns "darwin".

I'm using startswith since I don't know what other versions of Mac might return.
Comment on attachment 8497626 [details] [diff] [review]
mozharness_darwin.diff

This seems to work for me on my Mac.
Bob tested that it works on Windows.
Attachment #8497626 - Flags: review?(jlund)
<jlund> armenzg: based off http://stackoverflow.com/questions/4553129/when-to-use-os-name-sys-platform-or-platform-system I can see why BaseScript.is_windows() uses multiple conditions.
<jlund> armenzg: seems like the best pattern is to use a combination of platform.system() (which is runtime and probably best for CI) and fallsback on os.name (build configure time)
Merged to production, and deployed.
Comment on attachment 8497626 [details] [diff] [review]
mozharness_darwin.diff

Review of attachment 8497626 [details] [diff] [review]:
-----------------------------------------------------------------

I'm ok with this if we later make this more robust like mentioned in comment 16
Attachment #8497626 - Flags: review?(jlund) → review+
(Assignee)

Updated

3 years ago
Whiteboard: [good first bug] workaround in comment 4 → [easier-mozharness][good first bug] workaround in comment 4
Comment on attachment 8497626 [details] [diff] [review]
mozharness_darwin.diff

http://hg.mozilla.org/build/mozharness/rev/d572e654d6f6
Attachment #8497626 - Flags: checked-in+
(In reply to Jordan Lund (:jlund) from comment #18)
> Comment on attachment 8497626 [details] [diff] [review]
> mozharness_darwin.diff
> 
> Review of attachment 8497626 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I'm ok with this if we later make this more robust like mentioned in comment
> 16

Filed as bug 1078380.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.