1011041 - inari eng nightlies failing because stage-update.py can't find a mar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/?jobname=b2g_mozilla-central_inari_eng_nightly

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=39693142&tree=Mozilla-Central
{
17:31:06     INFO - mkdir: /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/testdata
17:31:06     INFO - Writing to file /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/testdata/smoketest-config.json
17:31:06     INFO - Contents:
17:31:06     INFO -  {"public_key": "/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/target/product/security/testkey.x509.pem", "private_key": "/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/target/product/security/testkey.pk8", "devices": {"otoro": {"serials": ["full_otoro"], "system_fs_type": "ext4", "data_fs_type": "ext4", "sdcard_recovery": "/sdcard", "system_location": "/dev/block/mmcblk0p19", "sdcard": "/mnt/sdcard", "data_location": "/dev/block/mmcblk0p22"}, "inari": {"serials": ["full_inari"], "system_fs_type": "ext4", "data_fs_type": "ext4", "sdcard_recovery": "/sdcard", "system_location": "/dev/block/mmcblk0p19", "sdcard": "/mnt/sdcard", "data_location": "/dev/block/mmcblk0p22"}}}
17:31:06     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build', '--unpriv', '--shell', u'/usr/bin/env MOZ_TELEMETRY_REPORTING=1 VARIANT=eng GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/tools/python27/bin:/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld "WGET_OPTS=-c -q" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140514160245 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build HOSTNAME=bld-linux64-spot-380.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-cen_inari_eng_ntly-00000 PROPERTIES_FILE=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=inari/2.0.0/default /tools/python27/bin/python2.7 /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py inari /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/testdata'] in /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build
17:31:06     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build --unpriv --shell "/usr/bin/env MOZ_TELEMETRY_REPORTING=1 VARIANT=eng GAIA_OPTIMIZE=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/tools/python27/bin:/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld \"WGET_OPTS=-c -q\" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140514160245 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko-l10n MOZILLA_OFFICIAL=1 \"MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build HOSTNAME=bld-linux64-spot-380.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-cen_inari_eng_ntly-00000 PROPERTIES_FILE=/builds/slave/b2g_m-cen_inari_eng_ntly-00000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=inari/2.0.0/default /tools/python27/bin/python2.7 /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py inari /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/testdata"
17:31:47     INFO -  FOTA Flash ZIP generated: /builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/testdata/inari/20140514160245/flash.zip
17:31:47     INFO -  Building flash zip for device inari, version 32.0a1, build 20140514160245...
17:31:47     INFO -  Traceback (most recent call last):
17:31:47     INFO -    File "/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 95, in <module>
17:31:47     INFO -      main()
17:31:47     INFO -    File "/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 90, in main
17:31:47     INFO -      stage_update(args.device, args.stage_dir)
17:31:47     INFO -    File "/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 62, in stage_update
17:31:47     INFO -      shutil.copy(gecko_mar, complete_mar)
17:31:47     INFO -    File "/tools/python27/lib/python2.7/shutil.py", line 117, in copy
17:31:47     INFO -      copyfile(src, dst)
17:31:47     INFO -    File "/tools/python27/lib/python2.7/shutil.py", line 82, in copyfile
17:31:47     INFO -      with open(src, 'rb') as fsrc:
17:31:47     INFO -  IOError: [Errno 2] No such file or directory: '/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/objdir-gecko/dist/b2g-update/b2g-gecko-update.mar'
17:31:47    ERROR - Return code: 1
17:31:47    FATAL - failed to stage b2g update testdata
17:31:47    FATAL - Running post_fatal callback...
17:31:47    FATAL - Exiting 2
program finished with exit code 2
elapsedTime=5141.881514
========= Finished 'scripts/scripts/b2g_build.py --target ...' failed (results: 2, elapsed: 1 hrs, 25 mins, 42 secs) (at 2014-05-14 17:31:48.441397) =========
}

Last good:
https://tbpl.mozilla.org/?jobname=b2g_mozilla-central_inari_eng_nightly&rev=e5f321740d10

First bad:
https://tbpl.mozilla.org/?rev=5c99d5136ad7&jobname=b2g_mozilla-central_inari_eng_nightly

Comment 1

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=39738355&tree=Mozilla-Central
b2g_mozilla-central_inari_eng_nightly on 2014-05-15 06:57:01
revision: e4843f4f08a7
slave: bld-linux64-spot-091

IOError: [Errno 2] No such file or directory: '/builds/slave/b2g_m-cen_inari_eng_ntly-00000/build/objdir-gecko/dist/b2g-update/b2g-gecko-update.mar'
Return code: 1
failed to stage b2g update testdata
Running post_fatal callback...
Exiting 2

Comment 2

[[e-mail] (:bhearsum)]

I'm pretty sure this is going to be permafail, not intermittent. In any case, it's probably a result of me changing the device build MAR filenames.

Comment 3

[Ed Morley [:emorley]]

Yeah it is permafail, the keyword was to get this to show in TBPL, rather than to indicate it's intermittent (helpful I know) :-)

Comment 4

[[e-mail] (:bhearsum)]

Attachment: kill smoketest

The failing code is part of the smoketest. It started failing when I landed a patch to change MAR filenames. jgriffin says we don't need to run it anymore though. I also killed the dead code.

Comment 5

[Aki Sasaki [:aki] (he/him) (UTC-6)]

Comment on attachment 8423213 [details] [diff] [review]
kill smoketest

Awesome. You probably can nuke the testdata_dir from query_abs_dirs.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 8423213 [details] [diff] [review]
kill smoketest

I killed that upon landing. I transplanted the patch to production - you should be able to safely retrigger the job now.

Comment 7

[Ed Morley [:emorley]]

Thank you :-)