858047 - b2g18 unagi_eng_nightly builds failing

Status: RESOLVED WORKSFORME

(Testing :: Mozbase, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Possible fallout from bug 810096?

https://tbpl.mozilla.org/php/getParsedLog.php?id=21420420&tree=Mozilla-B2g18

00:04:23     INFO - Writing to file /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/testdata/smoketest-config.json
00:04:23     INFO - Contents:
00:04:23     INFO -  {"public_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.x509.pem", "private_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.pk8", "devices": {"unagi": {"serials": ["full_unagi"], "system_fs_type": "ext4", "sdcard_recovery": "/sdcard", "sdcard": "/mnt/sdcard", "data_fs_type": "ext4", "system_location": "/dev/block/mmcblk0p19", "data_location": "/dev/block/mmcblk0p22"}}}
00:04:23     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build', '--unpriv', '--shell', u'/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130403230205 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build B2GUPDATER=1 HOSTNAME=bld-linux64-ec2-419.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000 PROPERTIES_FILE=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 /tools/python27/bin/python2.7 /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/testdata'] in /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build
00:04:23     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build --unpriv --shell "/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130403230205 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko-l10n MOZILLA_OFFICIAL=1 \"MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build B2GUPDATER=1 HOSTNAME=bld-linux64-ec2-419.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000 PROPERTIES_FILE=/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 /tools/python27/bin/python2.7 /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/testdata"
00:04:24     INFO -  Traceback (most recent call last):
00:04:24     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 95, in <module>
00:04:24     INFO -      main()
00:04:24     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 84, in main
00:04:24     INFO -      b2g = find_b2g()
00:04:24     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/smoketest.py", line 21, in find_b2g
00:04:24     INFO -      from b2ginstance import B2GInstance
00:04:24     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/client/marionette/b2ginstance.py", line 15, in <module>
00:04:24     INFO -      from mozdevice import DeviceManagerADB
00:04:24     INFO -  ImportError: No module named mozdevice
00:04:24    ERROR - Return code: 1
00:04:24    FATAL - failed to stage b2g update testdata
00:04:24    FATAL - Exiting 2

Comment 1

[Andrew Halberstadt [:ahal]]

This is from:
https://hg.mozilla.org/releases/mozilla-b2g18/diff/5c187b17c37c/testing/marionette/client/marionette/b2ginstance.py

I'm not sure why the build is using b2ginstance.py... It definitely shouldn't be using the crashreporting stuff so a quick and dirty fix would be to just except ImportError those. Though I wouldn't mind figuring out exactly why the build is doing this and solve it properly.

Comment 2

[Jonathan Griffin (:jgriffin)]

It looks like it is running update smoketests on the build, which do use b2ginstance.py.  I didn't realize we were running those as part of the build.

Comment 3

[Andrew Halberstadt [:ahal]]

I checked in a hacky work around to stop the nightly unagi build from failing:
https://hg.mozilla.org/releases/mozilla-b2g18/rev/6ad03d528939

I think we should figure out how to solve this properly though.. in order of preference:
1) ensure in-tree virtualenv (containing mozdevice/mozcrash) is activated
2) remove dependency on b2ginstance.py from the update smoketests (it seems to mostly only be used for finding the homedir)
3) leave as is (with the import hack)
4) refactor crashreporting out of b2ginstance.py

CC'ing Marshall as I believe he owns these tests.

Comment 4

[Jonathan Griffin (:jgriffin)]

My vote would be for #1, if it's not too difficult.

Comment 5

[Andrew Halberstadt [:ahal]]

I'm just not sure what exactly would be involved there. In the build log I see "populating virtualenv", so it looks like maybe it just isn't being activated? Is the build itself supposed to activate it? Or is that the responsibility of the update tests?

Comment 6

[Jonathan Griffin (:jgriffin)]

It looks like all we'd need to do is to change the invocation, which is currently:

B2G_UPDATER=1 /tools/python27/bin/python2.7 /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/testdata"

to use the virtualenv's python, instead of /tools/python27/bin/python2.7

:catlee or :bhearsum, any ideas on how to do this?

Comment 7

[Andrew Halberstadt [:ahal]]

We'll also need to add mozcrash to http://mxr.mozilla.org/mozilla-b2g18/source/build/virtualenv/packages.txt

Comment 8

[Jonathan Griffin (:jgriffin)]

Oh good point.  In m-c we load http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/packages.txt, which contains mozcrash, but we don't do that in mozilla-b2g18.

Comment 9

[Andrew Halberstadt [:ahal]]

Don't think we'll get around to fixing this the proper way. The workaround seems to work fine and will go away along with b2g18.