1242771 - echo-variable-UPLOAD_FILES via mozharness fails when stderr is found

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Jordan Lund (:jlund)]

this is bad because there is stderr that we should ignore. 

example:

http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-date_firefox_linux64_l10n_repack/builds/102/steps/run_script/logs/stdio

22:31:30     INFO - Getting output from command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '-q', '--cwd', '/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/browser/locales', '--unpriv', '--shell', u'/usr/bin/env TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache MOZ_CRASHREPORTER_NO_REPORT=1 DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/ UPLOAD_USER=ffxbld HG_SHARE_BASE_DIR=/builds/hg-shared EN_US_BINARY_URL=https://queue.taskcluster.net/v1/task/aNhgH6bgQWyyePFObQ9GPA/artifacts/public/build L10NBASEDIR=../../l10n TOOLTOOL_HOME=/builds UPLOAD_TO_TEMP=1 MAR=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/dist/host/bin/mar HOSTNAME=bld-linux64-spot-317.build.releng.usw2.mozilla.com MAIL=/var/spool/mail/cltbld PROPERTIES_FILE=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 DIST=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n MOZ_AUTOMATION=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin SYMBOL_SERVER_USER=ffxbld CCACHE_HASHDIR= TERM=linux IS_NIGHTLY=yes LATEST_MAR_DIR=fake_kill_me SHLVL=1 MOZ_MAKE_COMPLETE_MAR=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org UPLOAD_HOST=upload.ffxbld.productdelivery.prod.mozaws.net UPLOAD_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_rsa TINDERBOX_OUTPUT=1 "MOZ_SIGN_CMD=python /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/tools/release/signing/signtool.py --cachedir /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/signing_cache -t /builds/slave/rel-date_fx_l64_l10n_rpk-00000/token -n /builds/slave/rel-date_fx_l64_l10n_rpk-00000/nonce -c /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/tools/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110" LC_ALL=C _=/tools/buildbot/bin/python LOCALE_MERGEDIR=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/merged/ MBSDIFF=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/dist/host/bin/mbsdiff MOZ_SYMBOLS_EXTRA_BUILDID=linux64-date MOZ_UPDATE_CHANNEL=date SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_rsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/rel-date_fx_l64_l10n_rpk-00000 "POST_UPLOAD_CMD=post_upload.py -b date-l10n -p firefox -i 20160113124930 --release-to-latest --release-to-dated" make echo-variable-UPLOAD_FILES AB_CD=gl'] in /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/browser/locales
22:31:30     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 -q --cwd /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/browser/locales --unpriv --shell "/usr/bin/env TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache MOZ_CRASHREPORTER_NO_REPORT=1 DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110,gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110,dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/ UPLOAD_USER=ffxbld HG_SHARE_BASE_DIR=/builds/hg-shared EN_US_BINARY_URL=https://queue.taskcluster.net/v1/task/aNhgH6bgQWyyePFObQ9GPA/artifacts/public/build L10NBASEDIR=../../l10n TOOLTOOL_HOME=/builds UPLOAD_TO_TEMP=1 MAR=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/dist/host/bin/mar HOSTNAME=bld-linux64-spot-317.build.releng.usw2.mozilla.com MAIL=/var/spool/mail/cltbld PROPERTIES_FILE=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 DIST=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n MOZ_AUTOMATION=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin SYMBOL_SERVER_USER=ffxbld CCACHE_HASHDIR= TERM=linux IS_NIGHTLY=yes LATEST_MAR_DIR=fake_kill_me SHLVL=1 MOZ_MAKE_COMPLETE_MAR=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org UPLOAD_HOST=upload.ffxbld.productdelivery.prod.mozaws.net UPLOAD_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_rsa TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/tools/release/signing/signtool.py --cachedir /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/signing_cache -t /builds/slave/rel-date_fx_l64_l10n_rpk-00000/token -n /builds/slave/rel-date_fx_l64_l10n_rpk-00000/nonce -c /builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/tools/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9110 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9110 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9110\" LC_ALL=C _=/tools/buildbot/bin/python LOCALE_MERGEDIR=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/merged/ MBSDIFF=/builds/slave/rel-date_fx_l64_l10n_rpk-00000/build/date/obj-l10n/dist/host/bin/mbsdiff MOZ_SYMBOLS_EXTRA_BUILDID=linux64-date MOZ_UPDATE_CHANNEL=date SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_rsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/rel-date_fx_l64_l10n_rpk-00000 \"POST_UPLOAD_CMD=post_upload.py -b date-l10n -p firefox -i 20160113124930 --release-to-latest --release-to-dated\" make echo-variable-UPLOAD_FILES AB_CD=gl"
22:31:30     INFO - Reading from file tmpfile_stdout
22:31:30    ERROR - Errors received:
22:31:30     INFO - Reading from file tmpfile_stderr
22:31:30    ERROR -  Makefile:82: Missing searchplugin: creativecommons.xml

this looks like https://bugzilla.mozilla.org/show_bug.cgi?id=1233314 but since we use mozharness for l10n vs tools repo script in production, it seems we fatal out once we get stderr.

mshal proposed we try ignoring stderr, comment very clearly why we are ignoring it and see if that helps.

Comment 1

[Jordan Lund (:jlund)]

Attachment: 250116_echo-variable-UPLOAD_FILES_stderr-mh-l10n.patch

going to try this out on date ..

Comment 2

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3ecb01f315a4

Comment 3

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3ecb01f315a4