Closed Bug 1232466 Opened 9 years ago Closed 6 years ago

beta CI builds fail trying to use printconfigsetting.py and running prettyname targets for package and update when GTK+3 is enabled

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox45+ wontfix)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox45 + wontfix

People

(Reporter: rail, Assigned: jlund)

References

Details

Attachments

(3 files)

151214_bug_1232466_printconfigsettings-buildbotcustom.patch
8.59 KB, patch
rail
: review+
markco
: feedback+
rail
: checked-in-
Details | Diff | Splinter Review
151215_bug_1232466_printconfigsettings-buildbotcustom-v2.patch
10.12 KB, patch
rail
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
use_mock.diff
7.34 KB, patch
mshal
: review+
rail
: checked-in-
Details | Diff | Splinter Review 
See https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=16ce8562a975

buildid: Traceback (most recent call last):
  File "build/config/printconfigsetting.py", line 5, in <module>
    import configobj
ImportError: No module named configobj

or 

ImportError: No module named configobj --revision 16ce8562a975 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds
Traceback (most recent call last):
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 328, in <module>
    package=options.package, verbose=True)
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 232, in UploadFiles
    output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key)
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 96, in DoSSHCommand
    output = f(cmdline, stderr=STDOUT).strip()
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/redo/redo/__init__.py", line 185, in _retriable_wrapper
    **retry_kwargs)
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/redo/redo/__init__.py", line 140, in retry
    return action(*args, **kwargs)
  File "/tools/python/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_rsa', 'ffxbld@upload.ffxbld.productdelivery.prod.mozaws.net', 'post_upload.py --tinderbox-builds-dir mozilla-beta-macosx64-debug -p firefox -i Traceback (most recent call last):\n  File "build/config/printconfigsetting.py", line 5, in <module>\n    import configobj\nImportError: No module named configobj --revision 16ce8562a975 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.dmg" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/mac64/xpi/firefox-44.0.en-US.langpack.xpi" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//mozharness.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.common.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.cppunittest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.xpcshell.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.mochitest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.talos.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.reftest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.web-platform.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.crashreporter-symbols.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.txt" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.mozinfo.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//test_packages.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//jsshell-mac64.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/host/bin/mar" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/host/bin/mbsdiff" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.checksums" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.checksums.asc"']' returned non-zero exit status 1
make[1]: *** [upload] Error 1
make: *** [upload] Error 2
retry: Giving up on <function run_with_timeout at 0x1004edc08>
Unable to successfully run ['make', 'upload'] after 5 attempts
program finished with exit code 1
elapsedTime=907.006275


mshal pointed me to bug 957911, bug 1213839, bug 957951

FTR, m-a uses mozharness, beta uses old-style buildbot factories
Assignee: nobody → jlund
The signature for re.sub changed between 2.6 and 2.7: see https://docs.python.org/2.6/library/re.html and https://docs.python.org/2/library/re.html.  The very beginning of the log at http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-beta-android-api-9/1450134357/mozilla-beta-android-api-9-bm94-build1-build3.txt.gz is clear:

builder: mozilla-beta-android-api-9
slave: bld-linux64-spot-256
starttime: 1450128869.6
results: failure (2)
buildid: Traceback (most recent call last):
  File "build/config/printconfigsetting.py", line 17, in <module>
    content = re.sub('^\s*;', '#', fh.read(), flags=re.M)
TypeError: sub() got an unexpected keyword argument 'flags'
builduid: 82d1647865324b138fa85a47f4ba97a5
revision: 3f02f1340ad2

We'll need to bump the Python version, special case the version, or write equivalent code that doesn't use the new signature.
this is going to be fun..

okay so I'm making 3 wild assumptions:

1) we can call 'python2.7' on windows aws builders and it will be found in path <- markco?
2) we can use the new style of printconfigsettings.py on m-r/esr <- glandium?
3) the only reason TB used comms printconfigsettings.py was a workaround for not having this patch

dump master diff: http://people.mozilla.org/~jlund/printconfigsettings_fix_bbotcustom-dump-master.diff
Attachment #8698310 - Flags: review?(rail)
Attachment #8698310 - Flags: feedback?(mh+mozilla)
Attachment #8698310 - Flags: feedback?(mcornmesser)
I pushed a hack to see if we can just avoid this issue:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b

Hopefully we're green by tomorrow AM!
That's giving us green on Linux and Android, but not on Mac or Windows.
Comment on attachment 8698310 [details] [diff] [review]
151214_bug_1232466_printconfigsettings-buildbotcustom.patch

Review of attachment 8698310 [details] [diff] [review]:
-----------------------------------------------------------------

In overall the patch looks good to me. Given the urgency, let's land it and see what else breaks. :)
Attachment #8698310 - Flags: review?(rail) → review+
Comment on attachment 8698310 [details] [diff] [review]
151214_bug_1232466_printconfigsettings-buildbotcustom.patch

https://hg.mozilla.org/build/buildbotcustom/rev/694085d310b7
Attachment #8698310 - Flags: checked-in+
> 1) we can call 'python2.7' on windows aws builders and it will be found in
> path <- markco?
Yes, "C:\mozilla-build\python27" is within the default path.
Attachment #8698310 - Flags: feedback?(mcornmesser) → feedback+
I'm going to back it out in a bit. It didn't work. :(

I was watching http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%2064-bit%20mozilla-beta%20leak%20test%20build/builds/106 and it failed.

python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID
 in dir /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox (timeout 1200 secs)
...
/builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox/_virtualenv/bin/python: can't open file 'build/config/printconfigsetting.py': [Errno 2] No such file or directory

I tried to run it from /builds/slave/m-beta-m64-d-00000000000000000 on the same host and got this first time:
New python executable in /builds/slave/m-beta-m64-d-00000000000000000/build/obj-x86_64-apple-darwin11.2.0/_virtualenv/bin/python
Installing setuptools, pip...done.
Python 2.7.3 or greater (but not Python 3) is required to build. You are running Python 2.7.1.
Run |mach bootstrap| to ensure your system is up to date.

If you still receive this error, your shell environment is likely detecting
another Python version. Ensure a modern Python can be found in the paths
defined by the $PATH environment variable and try again.
Error running mach:

    ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Error populating virtualenv.

  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mach_commands.py", line 76, in python
    self._activate_virtualenv()
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/base.py", line 665, in _activate_virtualenv
    self.virtualenv_manager.ensure()
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/virtualenv.py", line 128, in ensure
    return self.build()
  File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/virtualenv.py", line 393, in build
    raise Exception('Error populating virtualenv.')


second run was less chatty:

[cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com m-beta-m64-d-00000000000000000]$ python2.7 build/config/printconfigsetting.py build/obj-firefox/dist/bin/
application.ini App BuildID                                                                                                                                    
Traceback (most recent call last):
  File "build/config/printconfigsetting.py", line 5, in <module>
    import configobj
ImportError: No module named configobj
[cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com m-beta-m64-d-00000000000000000]$ python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID                                                                     
Traceback (most recent call last):
  File "build/config/printconfigsetting.py", line 5, in <module>
    import configobj
ImportError: No module named configobj
Attachment #8698310 - Flags: checked-in+ → checked-in-
> I tried to run it from /builds/slave/m-beta-m64-d-00000000000000000 on the
> same host and got this first time:
> New python executable in
> /builds/slave/m-beta-m64-d-00000000000000000/build/obj-x86_64-apple-darwin11.
> 2.0/_virtualenv/bin/python
> Installing setuptools, pip...done.
> Python 2.7.3 or greater (but not Python 3) is required to build. You are
> running Python 2.7.1.
> Run |mach bootstrap| to ensure your system is up to date.

the osx one is using the wrong python2.7. I guess mach needs 2.7.3 or greater but we are using 2.7.1 :(

[cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ python --version
Python 2.7.1
[cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ which python
/usr/bin/python

for mozharness builds, we use python 2.7.3 interpreter
[cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ /tools/buildbot/bin/python --version
Python 2.7.3

this is defined from 'mozharness_python': https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#930
> second run was less chatty:
> 
> [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com
> m-beta-m64-d-00000000000000000]$ python2.7
> build/config/printconfigsetting.py build/obj-firefox/dist/bin/
> application.ini App BuildID                                                 
> 
> Traceback (most recent call last):
>   File "build/config/printconfigsetting.py", line 5, in <module>
>     import configobj
> ImportError: No module named configobj
> [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com
> m-beta-m64-d-00000000000000000]$ python2.7
> /builds/slave/m-beta-m64-d-00000000000000000/build/mach python
> build/config/printconfigsetting.py
> build/obj-firefox/dist/bin/application.ini App BuildID                      
> 
> Traceback (most recent call last):
>   File "build/config/printconfigsetting.py", line 5, in <module>
>     import configobj
> ImportError: No module named configobj

this osx one I suspect I screwed up the objdir workdir to run this because ... osx

this is what mozharness runs..
['/tools/buildbot/bin/python',
'/builds/slave/m-aurora-m64-00000000000000000/build/src/mach', 'python',
'/builds/slave/m-aurora-m64-00000000000000000/build/src/config/printconfigsetting.py',
'/builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64/dist/bin/application.ini',
'App', 'SourceStamp'] in
/builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64

this is what my buildbot patch ran:
python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python
build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini
App BuildID
 in dir /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox


good cwd: /builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64
bad cwd: /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox
(In reply to Jordan Lund (:jlund) from comment #11)
> > second run was less chatty:
> > 
> > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com
> > m-beta-m64-d-00000000000000000]$ python2.7
> > build/config/printconfigsetting.py build/obj-firefox/dist/bin/
> > application.ini App BuildID                                                 
> > 
> > Traceback (most recent call last):
> >   File "build/config/printconfigsetting.py", line 5, in <module>
> >     import configobj
> > ImportError: No module named configobj
> > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com
> > m-beta-m64-d-00000000000000000]$ python2.7
> > /builds/slave/m-beta-m64-d-00000000000000000/build/mach python
> > build/config/printconfigsetting.py
> > build/obj-firefox/dist/bin/application.ini App BuildID                      
> > 
> > Traceback (most recent call last):
> >   File "build/config/printconfigsetting.py", line 5, in <module>
> >     import configobj
> > ImportError: No module named configobj
> 
> this osx one I suspect I screwed up the objdir workdir to run this because
> ... osx

maybe not. the buildbot patched job in question failed doing a leak (debug) run. in the mozharness world, they both use the same objdir:

06:39:11     INFO - Copy/paste: /tools/buildbot/bin/python /builds/slave/m-aurora-m64-d-000000000000000/build/src/mach python /builds/slave/m-aurora-m64-d-000000000000000/build/src/config/printconfigsetting.py /builds/slave/m-aurora-m64-d-000000000000000/build/src/obj-firefox/dist/bin/application.ini App BuildID
Blocks: 1184181
interdiff: http://people.mozilla.org/~jlund/151215_bug_1232466_printconfigsettings-buildbotcustom-v2-interdiff.patch

full dump_master diff: http://people.mozilla.org/~jlund/printconfigsettings_fix_bbotcustom-dump-master-v2.diff

so, I think we had two issues based on logs

1) we were using a lesser version of python to invoke mach
2) we were using broken relative paths for app.ini and printconfigsetting (because we are now calling this from the objdir)

I did not test this on a master but I ran a test cmd locally that I think was mirrors what the patch will do based on dump_master and filling in the build prop vars:

[cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com obj-firefox]$ cd /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox[cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com obj-firefox]$ Apple_PubSub_Socket_Render=/tmp/launch-FL7jGa/Render CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 DISPLAY=/tmp/launch-7n2SR1/org.x:0 GIT_SHARE_BASE_DIR=/builds/git-shared HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld IDLEIZER_DISABLE_SHUTDOWN=true LC_ALL=C LOGNAME=cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-92ZnER/Listeners TINDERBOX_OUTPUT=1 TMPDIR=/var/folders/1t/62sfn2j55sx4w7c3jzgg8_kw00000w/T/ TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds TWISTD_LOG_PATH=/builds/slave/twistd.log USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1C:0:0 /tools/buildbot/bin/python /builds/slave/m-beta-m64-d-00000000000000000/build/mach python /builds/slave/m-beta-m64-d-00000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox/dist/bin/application.ini App BuildID
20151214205110
Attachment #8698562 - Flags: review?(rail)
Attachment #8698562 - Flags: review?(rail) → review+
Comment on attachment 8698562 [details] [diff] [review]
151215_bug_1232466_printconfigsettings-buildbotcustom-v2.patch

landed and reconfig'd. reconfig finished at about 230pm PT. after that I re-triggered some windows and osx builders. you can see those still running (grey) here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b

you need vpn to see the logs running live but I am watching them and they have been able to make it farther than before.

so far one has finished and it is looking good: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b&filter-searchStr=os%20x%20debug%20build
Attachment #8698562 - Flags: checked-in+
I think this patch is working so far. we have one failure (orange) on osx opt build[1]. But I think that has just shown itself now that we have made it past the printconfigsetting.py issue.

tracking the known failure

[1] https://treeherder.mozilla.org/logviewer.html#?job_id=684965&repo=mozilla-beta
Summary: beta CI builds fail trying to use printconfigsetting.py → beta CI builds fail trying to use printconfigsetting.py and running prettyname targets for package and update
1) failing to run `make package MOZ_PKG_PRETTYNAMES=1`

# Package mozharness
/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/_virtualenv/bin/python -m mozbuild.action.test_archive  mozharness /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/mac/en-US/mozharness.zip
Traceback (most recent call last):
  File "/tools/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tools/python/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/builds/slave/m-beta-m64-0000000000000000000/build/python/mozbuild/mozbuild/action/test_archive.py", line 332, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/builds/slave/m-beta-m64-0000000000000000000/build/python/mozbuild/mozbuild/action/test_archive.py", line 313, in main
    with open(args.outputfile, 'wb') as fh:
IOError: [Errno 2] No such file or directory: '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/mac/en-US/mozharness.zip'
make[3]: *** [stage-package] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2

iirc - taskcluster started packaging mozharness.zip. not sure if it's related.

2) failing to run `make -C obj-firefox/x86_64/tools/update-packaging MOZ_PKG_PRETTYNAMES=1`

log:
mkdir -p '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/update/mac/en-US/'
MAR=/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/host/bin/mar \
	  /builds/slave/m-beta-m64-0000000000000000000/build/tools/update-packaging/make_full_update.sh \
	  '../../dist/update/mac/en-US/nightly-44.0.complete.mar' \
	  '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/universal/firefox/Nightly.app'
/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/universal/firefox/Nightly.app /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/tools/update-packaging
precomplete file is missing!
make: *** [complete-patch] Error 1
I guess linux has been hitting this already.. https://treeherder.mozilla.org/logviewer.html#?job_id=684419&repo=mozilla-beta

windows just hit this too.

anything that runs pretty names (opt builds)
Rail filed bug 1232887 for the mozharness packaging error. Hopefully update packaging will work once normal packaging can create the precomplete file.
Depends on: 1232887
Comment on attachment 8698310 [details] [diff] [review]
151214_bug_1232466_printconfigsettings-buildbotcustom.patch

Looks like you figured it out.
Attachment #8698310 - Flags: feedback?(mh+mozilla)
This is still busted, actually:

========= Started set props: buildid (results: 2, elapsed: 0 secs) (at 2015-12-15 22:22:50.417902) =========
/tools/buildbot/bin/python /builds/slave/m-beta-lx-d-000000000000000000/build/mach python /builds/slave/m-beta-lx-d-000000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox/dist/bin/application.ini App BuildID
 in dir /builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox (timeout 1200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', '/builds/slave/m-beta-lx-d-000000000000000000/build/mach', 'python', '/builds/slave/m-beta-lx-d-000000000000000000/build/config/printconfigsetting.py', '/builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID']
 environment:
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_HASHDIR=
  CCACHE_UMASK=002
  DISPLAY=:2
  G_BROKEN_FILENAMES=1
  HG_SHARE_BASE_DIR=/builds/hg-shared
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-spot-036.build.releng.use1.mozilla.com
  LANG=en_US.UTF-8
  LC_ALL=C
  LD_LIBRARY_PATH=obj-firefox/dist/bin
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  MOZ_AUTOMATION=1
  MOZ_CRASHREPORTER_NO_REPORT=1
  PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
  PWD=/builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TINDERBOX_OUTPUT=1
  TMOUT=86400
  TOOLTOOL_CACHE=/builds/tooltool_cache
  TOOLTOOL_HOME=/builds
  USER=cltbld
  XPCOM_DEBUG_BREAK=stack-and-abort
  _=/tools/buildbot/bin/python
 using PTY: False
Error loading mozconfig: /builds/slave/m-beta-lx-d-000000000000000000/build/.mozconfig

Evaluation of your mozconfig exited with an error. This could be triggered
by a command inside your mozconfig failing. Please change your mozconfig
to not error and/or to catch errors in executed commands.

mozconfig output:

------BEGIN_AC_OPTION
--enable-debug
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-dmd
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-verify-mar
------END_AC_OPTION
------BEGIN_AC_OPTION
--with-google-oauth-api-keyfile=/builds/google-oauth-api.key
------END_AC_OPTION
------BEGIN_MK_OPTION
AUTOCLOBBER=1
------END_MK_OPTION
------BEGIN_AC_OPTION
--enable-crashreporter
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-release
------END_AC_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_BUILD_SYMBOLS=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_L10N_CHECK=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_PACKAGE=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_PACKAGE_TESTS=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_INSTALLER=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPDATE_PACKAGING=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPLOAD=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_SDK=0
------END_MK_OPTION
------BEGIN_MK_OPTION
PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
------END_MK_OPTION
------BEGIN_AC_OPTION
--enable-elf-hack
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-stdcxx-compat
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-default-toolkit=cairo-gtk3
------END_AC_OPTION
------BEGIN_MK_OPTION
export FONTCONFIG_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc/fonts
------END_MK_OPTION
------BEGIN_MK_OPTION
export PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc
------END_MK_OPTION
------BEGIN_MK_OPTION
export PANGO_LIBDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib
------END_MK_OPTION
------BEGIN_MK_OPTION
export GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
------END_MK_OPTION
------BEGIN_MK_OPTION
export GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders
------END_MK_OPTION
------BEGIN_MK_OPTION
export LD_LIBRARY_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib
------END_MK_OPTION
/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
program finished with exit code 1
elapsedTime=0.316526
buildid: 'Error loading mozconfig: /builds/slave/m-beta-lx-d-000000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not error and/or to catch errors in executed commands.\n\nmozconfig output:\n\n------BEGIN_AC_OPTION\n--enable-debug\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-dmd\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-verify-mar\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--with-google-oauth-api-keyfile=/builds/google-oauth-api.key\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nAUTOCLOBBER=1\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-crashreporter\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-release\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_BUILD_SYMBOLS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_L10N_CHECK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE_TESTS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_INSTALLER=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPDATE_PACKAGING=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD_SYMBOLS=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_SDK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nPATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-elf-hack\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-stdcxx-compat\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-default-toolkit=cairo-gtk3\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport FONTCONFIG_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc/fonts\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_LIBDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport LD_LIBRARY_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib\n------END_MK_OPTION\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory'
========= master_lag: 0.01 =========
========= Finished set props: buildid (results: 2, elapsed: 0 secs) (at 2015-12-15 22:22:50.748150) =========

This subsequently fails in post-upload because POST_UPLOAD_CMD has a busted value.
Or https://treeherder.mozilla.org/logviewer.html#?job_id=685601&repo=mozilla-beta if you like it opt, where instead of trying to upload with an error message in the commandline, we try to post to graphserver with an error message in the commandline.
sounds like we need to run those commands inside mock environments or install 32-bit libraries on builders.
Attached patch use_mock.diff — — Splinter Review 
still needs to be tested
Attachment #8698957 - Flags: review?(mshal)
Attachment #8698957 - Flags: review?(mshal) → review+
Whee, no longer blocks 44beta thanks to disabling GTK+3, but those last little bits, making Linux32 work for it, block 45beta since GTK+3 is planning on riding the next train, so six weeks to get it fixed and somehow tested.

[Tracking Requested - why for this release]: Because if it isn't fixed, you can't even ship the first beta of 45.
No longer blocks: 1214686
Severity: blocker → normal
tracking-firefox45: --- → ?
Summary: beta CI builds fail trying to use printconfigsetting.py and running prettyname targets for package and update → beta CI builds fail trying to use printconfigsetting.py and running prettyname targets for package and update when GTK+3 is enabled
Blocks: 1214686
Tracking to make sure it is fixed before the aurora => beta merge.
status-firefox45: --- → affected
tracking-firefox45: ? → +
Depends on: 1234220
See Also: → 1242641
No longer blocks: 1184181
No longer blocks: bld-lion-r5-068
Component: General Automation → General
I think this bug outlived its usefulness.
Closing this, feel free to reopen if I'm wrong.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: