TB Linux build failure: Exception: graph server did not add results successfully [gtk3]

RESOLVED FIXED in Thunderbird 47.0

Status

defect
--
blocker
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: aleth, Assigned: aleth)

Tracking

unspecified
Thunderbird 47.0
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Assignee

Description

4 years ago
Executing: ['python', '/builds/slave/tb-c-cen-l64-00000000000000000/tools/buildfarm/utils/graph_server_post.py', '--server', 'graphs.mozilla.org', '--selector', '/server/collect.cgi', '--branch', 'Thunderbird', '--buildid', 'Error loading mozconfig: /builds/slave/tb-c-cen-l64-00000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not error and/or to catch errors in executed commands.\n\nmozconfig output:\n\n------BEGIN_MK_OPTION\nAUTOCLOBBER=1\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-crashreporter\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-release\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_BUILD_SYMBOLS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_L10N_CHECK=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE_TESTS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_INSTALLER=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPDATE_PACKAGING=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD_SYMBOLS=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_SDK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nPATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-elf-hack\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-stdcxx-compat\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-default-toolkit=cairo-gtk3\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport FONTCONFIG_PATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/etc/fonts\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_SYSCONFDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/etc\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_LIBDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULE_FILE=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULEDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport LD_LIBRARY_PATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory', '--sourcestamp', 'Error loading mozconfig: /builds/slave/tb-c-cen-l64-00000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not error and/or to catch errors in executed commands.\n\nmozconfig output:\n\n------BEGIN_MK_OPTION\nAUTOCLOBBER=1\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-crashreporter\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-release\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_BUILD_SYMBOLS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_L10N_CHECK=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE_TESTS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_INSTALLER=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPDATE_PACKAGING=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD_SYMBOLS=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_SDK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nPATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-elf-hack\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-stdcxx-compat\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-default-toolkit=cairo-gtk3\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport FONTCONFIG_PATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/etc/fonts\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_SYSCONFDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/etc\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_LIBDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULE_FILE=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULEDIR=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport LD_LIBRARY_PATH=/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory', '--resultsname', 'TB_Linux_x86-64_comm-central', '--properties-file', 'properties.json', '--timestamp', '1450686078']
 results not added, response: 
 Bad format for metadata - 4 instead of 6 fields
   File "/var/www/html/graphs/server/pyfomatic/collect.py", line 293, in handleRequest
     metadata = MetaDataFromTalos(databaseCursor, databaseModule, inputStream)
   File "/var/www/html/graphs/server/pyfomatic/collect.py", line 62, in __init__
     self.readFromSource(dataSource)
   File "/var/www/html/graphs/server/pyfomatic/collect.py", line 77, in readFromSource
     raise ImproperFormatException("Bad format for metadata - %d instead of %d fields" % (numberOfInputFields, numberOfRequiredFields))
 
 
 Traceback (most recent call last):
   File "/builds/slave/tb-c-cen-l64-00000000000000000/tools/buildfarm/utils/graph_server_post.py", line 89, in <module>
     main()
   File "/builds/slave/tb-c-cen-l64-00000000000000000/tools/buildfarm/utils/graph_server_post.py", line 86, in main
     gp.postResult()
   File "/builds/slave/tb-c-cen-l64-00000000000000000/tools/buildfarm/utils/graph_server_post.py", line 58, in postResult
     self.doTinderboxPrint(content, testlongname, testname, prettyval)
   File "/builds/slave/tb-c-cen-l64-00000000000000000/tools/buildfarm/utils/graph_server_post.py", line 38, in doTinderboxPrint
     raise Exception("graph server did not add results successfully")
 Exception: graph server did not add results successfully
Assignee

Updated

4 years ago
Duplicate of this bug: 1234221
Assignee

Comment 2

4 years ago
build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory

appears to be the key part.
Assignee

Comment 3

4 years ago
I assume the path should be build/gtk3/setup.sh? (iirc tooltool fetches this)
Assignee

Comment 4

4 years ago
Is this similar to bug 1232466 comment 20 ? 

Looks like TOOLTOOL_DIR is suddenly pointing in the wrong place for c-c builds.
Flags: needinfo?(rail)
Assignee

Comment 5

4 years ago
Or maybe the problem is a little earlier in the log?

/tools/buildbot/bin/python /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/mach python /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/config/printconfigsetting.py /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/bin/application.ini App BuildID
 in dir /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb (timeout 1200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', '/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/mach', 'python', '/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/config/printconfigsetting.py', '/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/bin/application.ini', 'App', 'BuildID']

According to https://hg.mozilla.org/build/buildbotcustom/rev/b99e35634c32#l1.13 it should probably be calling build/config/printconfigsetting.py.
(In reply to aleth [:aleth] from comment #4)
> Is this similar to bug 1232466 comment 20 ? 

In a way similar, but not the same. It's related to GTK3 integration, but here we cannot even include a file, which is a in-tree file, not a tooltool file. I'm not sure what's the status of GTK3 in Thunderbird. If it's not implemented, disabling would help. If GTK3 integration is enabled, I'd play around mozconfig includes. :glandium is an expert here.
 
> Looks like TOOLTOOL_DIR is suddenly pointing in the wrong place for c-c
> builds.

Hmm, I'm not sure if it's related to tooltool.
Flags: needinfo?(rail)
Assignee

Comment 7

4 years ago
(In reply to Rail Aliiev [:rail] from comment #6)
> (In reply to aleth [:aleth] from comment #4)
> > Is this similar to bug 1232466 comment 20 ? 
> 
> In a way similar, but not the same. It's related to GTK3 integration, but
> here we cannot even include a file, which is a in-tree file, not a tooltool
> file. I'm not sure what's the status of GTK3 in Thunderbird. If it's not
> implemented, disabling would help. If GTK3 integration is enabled, I'd play
> around mozconfig includes. :glandium is an expert here.

GTK3 integration was working perfectly fine on comm-* until it very recently regressed.

Updated

4 years ago
Summary: TB Linux build failure: Exception: graph server did not add results successfully → TB Linux build failure: Exception: graph server did not add results successfully [gtk3]
(In reply to Rail Aliiev [:rail] from comment #6)
> (In reply to aleth [:aleth] from comment #4)
> > Is this similar to bug 1232466 comment 20 ? 
> 
> In a way similar, but not the same. It's related to GTK3 integration, but
> here we cannot even include a file, which is a in-tree file, not a tooltool
> file. I'm not sure what's the status of GTK3 in Thunderbird. If it's not
> implemented, disabling would help. 

rkent disabled gtk3 for beta 43 on 2015-12-21 some time after 10am PST (because AIUI Firefox had done so).

> If GTK3 integration is enabled, I'd play
> around mozconfig includes. :glandium is an expert here.
Flags: needinfo?(mh+mozilla)
Assignee

Comment 9

4 years ago
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #8)
> rkent disabled gtk3 for beta 43 on 2015-12-21 some time after 10am PST
> (because AIUI Firefox had done so).

This bug is about comm-central (and probably aurora). Beta 44 and 43 is fine (and has gtk3 disabled).
(In reply to aleth [:aleth] from comment #2)
> build/mozilla/build/unix/mozconfig.gtk: line 29:
> /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/gtk3/setup.sh: No
> such file or directory
> 
> appears to be the key part.

Note the /mozilla/ in that path. Your gtk3/setup.sh is probably under build/ directly, not build/mozilla/. Which is what TOOLTOOL_DIR would help you get right.
Flags: needinfo?(mh+mozilla)
Assignee

Comment 11

4 years ago
(In reply to Mike Hommey [:glandium] (VAC: 31 Dec - 11 Jan) from comment #10)
> Note the /mozilla/ in that path. Your gtk3/setup.sh is probably under build/
> directly, not build/mozilla/. Which is what TOOLTOOL_DIR would help you get
> right.

Yes, the problem here is it *was* set right until it broke ;)
Duplicate of this bug: 1235885

Comment 13

3 years ago
Realize it's been holiday season for many, but I wonder if there a way to get the builds going again even if the fix is not yet ideal (disable gtk3 or something)? 

Thanks!
Assignee

Updated

3 years ago
Assignee: nobody → aleth
Status: NEW → ASSIGNED
Assignee

Comment 15

3 years ago
(In reply to aleth [:aleth] from comment #14)
> Created attachment 8705387 [details] [diff] [review]
> TB builds should use the config/printconfigsetting.py from comm-* not
> mozilla-*

This is a patch based on comment 5 (i.e. restoring the previous printconfigsetting paths). It's not clear to me whether this on its own will fix this bug.

(Bug 1221502 originally fixed TOOLTOOL_DIR being set incorrectly for c-c builds (which at the time led to the same mozconfig error as the one which is arising here). Somehow those changes are now either no longer working or insufficient.)
Assignee

Updated

3 years ago
Attachment #8705387 - Attachment is obsolete: true
Attachment #8705387 - Flags: review?(rail)
Assignee

Comment 17

3 years ago
Comment on attachment 8705387 [details] [diff] [review]
TB builds should use the config/printconfigsetting.py from comm-* not mozilla-*

bzexport removed the flag...
Attachment #8705387 - Attachment is obsolete: false
Attachment #8705387 - Flags: review?(rail)
Assignee

Comment 18

3 years ago
(In reply to aleth [:aleth] from comment #16)
> Created attachment 8705399 [details] [diff] [review]
> Ensure TOOLTOOL_DIR is set correctly when running printconfigsetting.py

This patch (similar to those in bug 1221502) should ensure TOOLTOOL_DIR gets the correct value. (Would be overkill if the other patch on its own was enough to fix this, however.)

Comment 19

3 years ago
Thanks for working on this ... Is there any ETA on when builds will work again? As of now they are still failing in exactly same way.

thanks.
Comment on attachment 8705387 [details] [diff] [review]
TB builds should use the config/printconfigsetting.py from comm-* not mozilla-*

Jordan, IIRC you poked around printconfigsetting.py last time. May I ask you to look at this too.

Thanks in advance! ;)
Attachment #8705387 - Flags: review?(rail) → review?(jlund)

Updated

3 years ago
Attachment #8705399 - Flags: review?(rail) → review?(jlund)
Comment on attachment 8705387 [details] [diff] [review]
TB builds should use the config/printconfigsetting.py from comm-* not mozilla-*

Review of attachment 8705387 [details] [diff] [review]:
-----------------------------------------------------------------

hmm, I landed https://hg.mozilla.org/build/buildbotcustom/rev/b99e35634c32 because I thought printconfigsettings.py would be fixed for both FF and TB. Turns out there are further issues using m-c's printconfigsettings.py than I thought:

example log:
========= Started set props: buildid (results: 2, elapsed: 0 secs) (at 2016-01-10 16:34:55.027622) =========
/tools/buildbot/bin/python /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/mach python /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/config/printconfigsetting.py /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/bin/application.ini App BuildID
 in dir /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb (timeout 1200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', '/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/mach', 'python', '/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/config/printconfigsetting.py', '/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/bin/application.ini', 'App', 'BuildID']
Error loading mozconfig: /builds/slave/tb-c-cen-lx-000000000000000000/build/.mozconfig

Evaluation of your mozconfig exited with an error. This could be triggered
by a command inside your mozconfig failing. Please change your mozconfig
to not error and/or to catch errors in executed commands.

mozconfig output:

------BEGIN_MK_OPTION
AUTOCLOBBER=1
------END_MK_OPTION
------BEGIN_AC_OPTION
--enable-crashreporter
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-release
------END_AC_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_BUILD_SYMBOLS=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_L10N_CHECK=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_PACKAGE=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_PACKAGE_TESTS=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_INSTALLER=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPDATE_PACKAGING=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPLOAD=1
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
------END_MK_OPTION
------BEGIN_MK_OPTION
export MOZ_AUTOMATION_SDK=0
------END_MK_OPTION
------BEGIN_MK_OPTION
PATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
------END_MK_OPTION
------BEGIN_AC_OPTION
--enable-elf-hack
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-stdcxx-compat
------END_AC_OPTION
------BEGIN_AC_OPTION
--enable-default-toolkit=cairo-gtk3
------END_AC_OPTION
------BEGIN_MK_OPTION
export FONTCONFIG_PATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/etc/fonts
------END_MK_OPTION
------BEGIN_MK_OPTION
export PANGO_SYSCONFDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/etc
------END_MK_OPTION
------BEGIN_MK_OPTION
export PANGO_LIBDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib
------END_MK_OPTION
------BEGIN_MK_OPTION
export GDK_PIXBUF_MODULE_FILE=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
------END_MK_OPTION
------BEGIN_MK_OPTION
export GDK_PIXBUF_MODULEDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders
------END_MK_OPTION
------BEGIN_MK_OPTION
export LD_LIBRARY_PATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib
------END_MK_OPTION
/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory
program finished with exit code 1
elapsedTime=0.311845
buildid: 'Error loading mozconfig: /builds/slave/tb-c-cen-lx-000000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not error and/or to catch errors in executed commands.\n\nmozconfig output:\n\n------BEGIN_MK_OPTION\nAUTOCLOBBER=1\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-crashreporter\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-release\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_BUILD_SYMBOLS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_L10N_CHECK=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE_TESTS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_INSTALLER=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPDATE_PACKAGING=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD_SYMBOLS=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_SDK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nPATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-elf-hack\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-stdcxx-compat\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-default-toolkit=cairo-gtk3\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport FONTCONFIG_PATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/etc/fonts\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_SYSCONFDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/etc\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_LIBDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULE_FILE=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULEDIR=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport LD_LIBRARY_PATH=/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/usr/local/lib\n------END_MK_OPTION\n/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory'
========= master_lag: 0.01 =========

this causes the unique result of storing something like "Error loading mozconfig: /builds/slave/tb-c-cen-lx-000000000000000000/build/.mozconfig" as the buildid property :\

glandium may know more. I would guess at the very least TB will need a https://dxr.mozilla.org/mozilla-central/source/.mozconfig or m-c's printconfigsettings.py will need to be patched for TB usecase. however, saying that, it is probably better to keep FF and TB separate for both product's sake.
Attachment #8705387 - Flags: review?(jlund) → review+
Attachment #8705399 - Flags: review?(jlund) → review+
Assignee

Comment 22

3 years ago
https://hg.mozilla.org/build/buildbotcustom/rev/77f2f273a89a4989bfdbef0b653f03cfc7cc168f
Bug 1234220 - Ensure TOOLTOOL_DIR is set correctly when running printconfigsetting.py. r=jlund

https://hg.mozilla.org/build/buildbotcustom/rev/ee392b541b017cd4352b83e6f051514aaea0310e
Bug 1234220 - TB builds should use the config/printconfigsetting.py from comm-* not mozilla-*. r=jlund
Assignee

Comment 23

3 years ago
(In reply to Jordan Lund (:jlund) from comment #21)
> this causes the unique result of storing something like "Error loading
> mozconfig: /builds/slave/tb-c-cen-lx-000000000000000000/build/.mozconfig" as
> the buildid property :\
> 
> glandium may know more. I would guess at the very least TB will need a
> https://dxr.mozilla.org/mozilla-central/source/.mozconfig or m-c's
> printconfigsettings.py will need to be patched for TB usecase. however,
> saying that, it is probably better to keep FF and TB separate for both
> product's sake.

The effect of attachment 8705387 [details] [diff] [review] should be that c-c uses its own (older) printconfigsettings.py, but this is indeed an unfortunate hack. Maybe it's not required any more? So I just backed it out again (unfortunately straight after landing it) so we can find out.
Assignee

Comment 25

3 years ago
(In reply to aleth [:aleth] from comment #23)
> (In reply to Jordan Lund (:jlund) from comment #21)
> > this causes the unique result of storing something like "Error loading
> > mozconfig: /builds/slave/tb-c-cen-lx-000000000000000000/build/.mozconfig" as
> > the buildid property :\

To be clear, the immediate cause of this is that an incorrect TOOLTOOL_DIR leads to a mozconfig parse error.
Assignee

Comment 26

3 years ago
Looks like the single patch already fixed the problem for 64b Linux builds! I suppose this means that there are no longer any build machines with an old version of python (which made the separate printconfigsetting for c-c necessary, iirc), which is good news.

Unfortunately there is still a problem on 32b Linux. Now setup.sh is found as tooltool_dir is correct, mozconfig parsing now fails with

/builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
/builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
/builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

This suggests missing 32b libraries. Any ideas as to why this might be happening?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(jlund)
Assignee

Updated

3 years ago
Attachment #8705387 - Attachment is obsolete: true
> This suggests missing 32b libraries. Any ideas as to why this might be
> happening?

I do not sorry. If glandium can't help, it may be worth giving up and revert back to c-c's printconfigsetting.py for all platforms.
Flags: needinfo?(jlund)
Assignee

Comment 28

3 years ago
(In reply to Jordan Lund (:jlund) from comment #27)
> > This suggests missing 32b libraries. Any ideas as to why this might be
> > happening?
> 
> I do not sorry. If glandium can't help, it may be worth giving up and revert
> back to c-c's printconfigsetting.py for all platforms.

I'm actually not sure that would help at all, I don't see how it could be related to 32b library issues in gtk3/setup.sh (but we could try it of course).
Assignee

Comment 29

3 years ago
(In reply to aleth [:aleth] from comment #26)
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter:
> No such file or directory
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF
> interpreter: No such file or directory
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such
> file or directory
> 
> This suggests missing 32b libraries. Any ideas as to why this might be
> happening?

Looks this has just uncovered what was already flagged up here: bug 1232466 comment 20, bug 1232466 comment 22, bug 1232466 comment 27.
Flags: needinfo?(rail)
Assignee

Updated

3 years ago
Blocks: 1232466
yeah, gtk3/setup.sh tries to load some 32-bit libraries, but fails to do so on 64-bit.
Flags: needinfo?(rail)
(In reply to aleth [:aleth] from comment #26)
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter:
> No such file or directory
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF
> interpreter: No such file or directory
> /builds/slave/tb-c-cen-lx-000000000000000000/build/gtk3/setup.sh:
> ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such
> file or directory
> 
> This suggests missing 32b libraries. Any ideas as to why this might be
> happening?

I bet this is running outside the mock environment.
Flags: needinfo?(mh+mozilla)
Assignee

Comment 32

3 years ago
(In reply to Mike Hommey [:glandium] from comment #31)
> I bet this is running outside the mock environment.

Looks like Rail backed it out? https://hg.mozilla.org/build/buildbotcustom/rev/2293d99fd1fc
Flags: needinfo?(rail)
Yes, that change broke the builds unfortunately :/
Flags: needinfo?(rail)
Assignee

Comment 34

3 years ago
(In reply to Rail Aliiev [:rail] from comment #33)
> Yes, that change broke the builds unfortunately :/

So I guess the problem is how to fix that patch so it doesn't break the world...
Assignee

Comment 35

3 years ago
> (In reply to Mike Hommey [:glandium] from comment #31)
> > I bet this is running outside the mock environment.
> 
> Looks like Rail backed it out?
> https://hg.mozilla.org/build/buildbotcustom/rev/2293d99fd1fc

Any ideas what is wrong with this patch?
Flags: needinfo?(mh+mozilla)
Without more details than comment 33, no.
Flags: needinfo?(mh+mozilla)
Assignee

Comment 37

3 years ago
(In reply to Mike Hommey [:glandium] from comment #36)
> Without more details than comment 33, no.

Are there any more details/logs on what exactly went wrong?
Flags: needinfo?(rail)
(In reply to aleth [:aleth] from comment #37)
> (In reply to Mike Hommey [:glandium] from comment #36)
> > Without more details than comment 33, no.
> 
> Are there any more details/logs on what exactly went wrong?

TBH, I couldn't find what happened.... :/ I went through the build logs, release runner logs around Dec 16, but was unable to find something useful.

In any case, we should fix this ASAP.
Flags: needinfo?(rail)
Assignee

Comment 39

3 years ago
(In reply to Rail Aliiev [:rail] from comment #38)
> In any case, we should fix this ASAP.

Thanks, it would be great to sort this out, especially now 45 (the next ESR) is in beta.
Rail, Now that c-c and m-c merge are done, you'll have another go?
Flags: needinfo?(rail)
And https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438 to hopefully get that working. Please retrigger some TB builds to see what state we're in with this.
Assignee

Comment 43

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #42)
> And https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438 to
> hopefully get that working. Please retrigger some TB builds to see what
> state we're in with this.

There's trouble on all Linux builds now, but in a way that maybe indicates progress. The buildid returned appears to contain extraneous stuff, e.g.
"INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
20160126014327
State Changed: unlock buildroot"

http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-linux64/1453805057/comm-central-linux64-bm77-build1-build1.txt.gz
Flags: needinfo?(rail) → needinfo?(nthomas)
Assignee

Comment 44

3 years ago
(In reply to aleth [:aleth] from comment #43)
> There's trouble on all Linux builds now, but in a way that maybe indicates
> progress. 

i.e. it looks like if Mock wrote its status messages to stderr instead of stdout, your patch would have worked?
OK, same problem Firefox has. Lets see how things shake out there and then check TB again.
Flags: needinfo?(nthomas)
Assignee

Updated

3 years ago
Depends on: 1242641
Depends on: 1245828
Assignee

Comment 46

3 years ago
The remaining issue here should be fixed by bug 1242641 and bug 1245828.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 47.0
You need to log in before you can comment on or make changes to this bug.