894360 - Intermittent "KeyError: 'completeMarSize'" whilst running scripts/updates/balrog-submitter.py (caused by earlier "zipfile.BadZipfile: File is not a zip file")

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Android X86 mozilla-central nightly on 2013-07-15 21:31:19 PDT for push 5976b9c673f8

slave: bld-linux64-ec2-023

https://tbpl.mozilla.org/php/getParsedLog.php?id=25314069&tree=Mozilla-Central

{
========= Started 'python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py ...' failed (results: 2, elapsed: 16 secs) (at 2013-07-15 23:57:02.938854) =========
python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py --build-properties buildprops_balrog.json --api-root https://aus4-admin-dev.allizom.org --verbose --credentials-file BuildSlaves.py
 in dir /builds/slave/m-cen-and-x86-ntly-00000000000/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py', '-s', '1', '-r', '5', '-t', '1260', 'python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-ec2-023.build.releng.use1.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/slave/m-cen-and-x86-ntly-00000000000
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #2
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 2 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #3
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 4 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #4
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 8 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #5
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Giving up on <function run_with_timeout at 0x7f1a4f5ce848>
Unable to successfully run ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'] after 5 attempts
program finished with exit code 1
elapsedTime=16.587755
========= Finished 'python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py ...' failed (results: 2, elapsed: 16 secs) (at 2013-07-15 23:57:19.541643) =========
}

Comment 1

[[e-mail] (:bhearsum)]

The actual failure on this build was:
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 266, in <module>
    main()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 247, in main
    options.download_subdir)
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 180, in getPreviousBuildID
    return parseApk('previous.apk')[0]
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 96, in parseApk
    appini = StringIO(zipfile.ZipFile(apk_filename).read('application.ini'))
  File "/usr/lib64/python2.6/zipfile.py", line 696, in __init__
    self._GetContents()
  File "/usr/lib64/python2.6/zipfile.py", line 716, in _GetContents
    self._RealGetContents()
  File "/usr/lib64/python2.6/zipfile.py", line 728, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

Which actually caused the error in balrog-submitter.py. The create snippet step maybe should be haltOnFailure...

Comment 2

[Ed Morley [:emorley]]

We should also catch the exception in android_snippet.py and print a parseable error message :-)

Comment 3

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=25600715&tree=Mozilla-Aurora
Android X86 mozilla-aurora nightly on 2013-07-23 01:00:55
revision: e18d6a0ab17c
slave: bld-linux64-ec2-062

CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/home/cltbld/.ssh/auspush', 'aus3-staging.mozilla.org', 'mkdir', '-p', '/opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_x86-gcc3/20130723004004/en-US']' returned non-zero exit status 255
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'

Comment 4

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=25658488&tree=Mozilla-Central
Android Armv6 mozilla-central nightly on 2013-07-24 03:14:12
revision: 2983ca6d4d1a
slave: bld-linux64-ix-034

Automation Error: Unable to parse application.ini from previous.apk!
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'

Comment 5

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=25814305&tree=Mozilla-Central
Android mozilla-central nightly on 2013-07-27 03:22:25
revision: fb48c7d58b8b
slave: bld-linux64-ix-029

Automation Error: Unable to parse application.ini from previous.apk!
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'

Comment 6

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=25854147&tree=Mozilla-Central
Android mozilla-central nightly on 2013-07-29 03:02:17
revision: 73b69c146ca6
slave: bld-linux64-ix-023

Automation Error: Unable to parse application.ini from previous.apk!
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'
KeyError: 'completeMarSize'

Comment 7

[Ed Morley [:emorley]]

Closing inactive keywords:intermittent-failure bugs where the TBPLbot has previously commented and the test isn't marked as disabled; filter on orange-cleanup-201401.

Comment 8

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1150721
repository: mozilla-central
start_time: 2015-03-11T16:03:13
who: wkocher[at]mozilla[dot]com
machine: bld-lion-r5-074
buildname: OS X Mulet mozilla-central nightly
revision: 30916c9ca768

KeyError: 'completeMarSize'
Return code: 1