Intermittent "KeyError: 'completeMarSize'" whilst running scripts/updates/balrog-submitter.py (caused by earlier "zipfile.BadZipfile: File is not a zip file")

RESOLVED WORKSFORME

Status

Release Engineering
General Automation
RESOLVED WORKSFORME
5 years ago
3 years ago

People

(Reporter: emorley, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Android X86 mozilla-central nightly on 2013-07-15 21:31:19 PDT for push 5976b9c673f8

slave: bld-linux64-ec2-023

https://tbpl.mozilla.org/php/getParsedLog.php?id=25314069&tree=Mozilla-Central

{
========= Started 'python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py ...' failed (results: 2, elapsed: 16 secs) (at 2013-07-15 23:57:02.938854) =========
python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py --build-properties buildprops_balrog.json --api-root https://aus4-admin-dev.allizom.org --verbose --credentials-file BuildSlaves.py
 in dir /builds/slave/m-cen-and-x86-ntly-00000000000/. (timeout 1320 secs)
 watching logfiles {}
 argv: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py', '-s', '1', '-r', '5', '-t', '1260', 'python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-ec2-023.build.releng.use1.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/slave/m-cen-and-x86-ntly-00000000000
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #2
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 2 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #3
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 4 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #4
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Failed, sleeping 8 seconds before retrying
retry: Calling <function run_with_timeout at 0x7f1a4f5ce848> with args: (['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #5
Executing: ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 76, in run
    data = self.generate_data()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/lib/python/balrog/submitter/cli.py", line 59, in generate_data
    'filesize': props['completeMarSize'],
KeyError: 'completeMarSize'
retry: Giving up on <function run_with_timeout at 0x7f1a4f5ce848>
Unable to successfully run ['python', '/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'] after 5 attempts
program finished with exit code 1
elapsedTime=16.587755
========= Finished 'python /builds/slave/m-cen-and-x86-ntly-00000000000/tools/buildfarm/utils/retry.py ...' failed (results: 2, elapsed: 16 secs) (at 2013-07-15 23:57:19.541643) =========
}
The actual failure on this build was:
Traceback (most recent call last):
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 266, in <module>
    main()
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 247, in main
    options.download_subdir)
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 180, in getPreviousBuildID
    return parseApk('previous.apk')[0]
  File "/builds/slave/m-cen-and-x86-ntly-00000000000/tools/scripts/android/android_snippet.py", line 96, in parseApk
    appini = StringIO(zipfile.ZipFile(apk_filename).read('application.ini'))
  File "/usr/lib64/python2.6/zipfile.py", line 696, in __init__
    self._GetContents()
  File "/usr/lib64/python2.6/zipfile.py", line 716, in _GetContents
    self._RealGetContents()
  File "/usr/lib64/python2.6/zipfile.py", line 728, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

Which actually caused the error in balrog-submitter.py. The create snippet step maybe should be haltOnFailure...
(Reporter)

Comment 2

5 years ago
We should also catch the exception in android_snippet.py and print a parseable error message :-)
(Reporter)

Updated

5 years ago
Depends on: 894365
(Reporter)

Updated

5 years ago
Summary: Intermittent "KeyError: 'completeMarSize'" whilst running scripts/updates/balrog-submitter.py → Intermittent "KeyError: 'completeMarSize'" whilst running scripts/updates/balrog-submitter.py (caused by earlier "zipfile.BadZipfile: File is not a zip file")
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Reporter)

Comment 7

4 years ago
Closing inactive keywords:intermittent-failure bugs where the TBPLbot has previously commented and the test isn't marked as disabled; filter on orange-cleanup-201401.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME
Comment hidden (Treeherder Robot)
You need to log in before you can comment on or make changes to this bug.