1242641 - GTK+3 still not working for buildbot builds on beta

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=743745&repo=mozilla-beta

keeping mozilla-beta closed

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This looks a little bogus at the --buildid argument:

Executing: ['python', '/builds/slave/m-beta-lx-00000000000000000000/tools/buildfarm/utils/graph_server_post.py', '--server', 'graphs.mozilla.org', '--selector', '/server/collect.cgi', '--branch', 'Mozilla-Beta', '--buildid', 'Error loading mozconfig: /builds/slave/m-beta-lx-00000000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not e....

which stems from the earlier step:

 ========= Started set props: sourcestamp (results: 2, elapsed: 0 secs) (at 2016-01-25 09:38:14.834835) =========
 /tools/buildbot/bin/python /builds/slave/m-beta-lx-00000000000000000000/build/mach python /builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini App SourceStamp
  in dir /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox (timeout 1200 secs)
  watching logfiles {}
  argv: ['/tools/buildbot/bin/python', '/builds/slave/m-beta-lx-00000000000000000000/build/mach', 'python', '/builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py', '/builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini', 'App', 'SourceStamp']
  environment:
   CCACHE_COMPRESS=1
...
   _=/tools/buildbot/bin/python
  using PTY: False
 Error loading mozconfig: /builds/slave/m-beta-lx-00000000000000000000/build/.mozconfig
 
 Evaluation of your mozconfig exited with an error. This could be triggered
 by a command inside your mozconfig failing. Please change your mozconfig
 to not error and/or to catch errors in executed commands.
 
 mozconfig output:
 
 ------BEGIN_AC_OPTION
 --enable-update-channel=
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-update-packaging
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-google-api-keyfile=/builds/gapi.data
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-google-oauth-api-keyfile=/builds/google-oauth-api.key
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 AUTOCLOBBER=1
 ------END_MK_OPTION
 ------BEGIN_AC_OPTION
 --enable-crashreporter
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-release
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_BUILD_SYMBOLS=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_L10N_CHECK=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_PACKAGE=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_PACKAGE_TESTS=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_INSTALLER=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPDATE_PACKAGING=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPLOAD=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_SDK=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
 ------END_MK_OPTION
 ------BEGIN_AC_OPTION
 --enable-elf-hack
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-stdcxx-compat
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-default-toolkit=cairo-gtk3
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 export FONTCONFIG_PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/etc/fonts
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/etc
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export PANGO_LIBDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export LD_LIBRARY_PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib
 ------END_MK_OPTION
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 program finished with exit code 1
 elapsedTime=0.232137

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Swap to MockProperty

This has rather wider effect than just linux32 builds, and change linux32, linux64, and android builds on m-b, m-r, m-esr38, as well some TB builds. It should do the trick though.

Comment 3

[Justin Wood (:Callek)]

Comment on attachment 8711880 [details] [diff] [review]
[buildbotcustom] Swap to MockProperty

Review of attachment 8711880 [details] [diff] [review]:
-----------------------------------------------------------------

I should note, taht almost every touch of SetProperty for printconfigsettings in these past cycles has broken Thunderbird (and vice versa) so don't be shocked if we get reports from that step-child. But if this works to fix Firefox, I'm happy.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8711880 [details] [diff] [review]
[buildbotcustom] Swap to MockProperty

https://hg.mozilla.org/build/buildbotcustom/rev/31ff0be74400  (default)
https://hg.mozilla.org/build/buildbotcustom/rev/e9d3cb9c5987  (prod)

Reconfig to follow.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Reconfig done, Kwierso is going to push bug 1237179 to see how we do.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Not working, mock doesn't want to start. See also bug 1232466, where we had a very similar issue last cycle.

Comment 7

[Mike Hommey [:glandium]]

Fixing bug 1190860 and finally using setup scripts from tooltool for gtk3 would solve this.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] followup to fix cwd for mock call, and use python from the virtualenv

I landed this patch because it's for because blocker bug
https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438
https://hg.mozilla.org/build/buildbotcustom/rev/d30c4d7a1504
but please take a look for all the context I lost while away.

It may be that we can stop using '/path/to/python mach python /path/to/printconfigsettings.py' everywhere, but I'd created enough bustage in one day.

Reconfig'd, retriggered some builds.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

To explain the previous patch, we were making mock calls like
 mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-beta-lx-d-000000000000000000//builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox
which the 'mock_workdir_prefix=None' fixes by removing the duplicated part before the //.

And then we need to use the python in the virtualenv the build system has created, which is derived from the one installed inside mock, instead of the one which mock can't see in /tools. Using the virtualenv python can maybe be generalized to all platforms, but this code is going to die with build promotion so I'm not convinced it's worth the testing time.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

We are getting properties set now but the values are still bogus, eg:

https://treeherder.mozilla.org/logviewer.html#?job_id=745781&repo=mozilla-beta

 ========= Started set props: buildid (results: 0, elapsed: 4 secs) (at 2016-01-25 23:22:05.112291) =========
 mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox --unpriv --shell '/usr/bin/env 
...
 CCACHE_UMASK="002" '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/mach'"'"' python '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py'"'"' '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini'"'"' App BuildID'
  in dir /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox (timeout 1200 secs)
...
 INFO: mock_mozilla.py version 1.0.3 starting...
 State Changed: init plugins
 INFO: selinux disabled
 State Changed: start
 State Changed: lock buildroot
 State Changed: shell
 '
 State Changed: unlock buildroot
 program finished with exit code 0
 elapsedTime=4.924754
 buildid: 'INFO: mock_mozilla.py version 1.0.3 starting...\nState Changed: init plugins\nINFO: selinux disabled\nState Changed: start\nState Changed: lock buildroot\nState Changed: shell\n20160125145626\nState Changed: unlock buildroot'

Which is a little confusing, as I think we should be at
  http://hg.mozilla.org/build/buildbotcustom/file/d30c4d7a1504/steps/mock.py#l136
and that's only paying attention to stdio. If I run the printconfigsetting command with '2> /dev/null' appended only '20160125145626' is returned.

I'm going to leave this here for now and let someone else pick it up when they start their day. Please be aware the attempted fixes have broken builds on m-esr38 and m-r, and probably impacted Thunderbird too.

Comment 11

[Mike Hommey [:glandium]]

We should just stop papering over this issue by using mock everywhere. The core problem is bug 1188571, which is that the mozconfig for gtk3 is actively doing stuff, and everything that uses the mozconfig ends up doing that stuff. That stuff needs to run in the mock environment for a sum of reasons. There is no patch attached to bug 1188571 to actually finish it, but it's trivial to write one once it can be tested on try... which it can't. Because fixing bug 1188571 relies on tooltool running a setup script that would do that stuff instead of the mozconfig, which means tooltool would need to run in the mock environment. Which is bug 1190860.

So, with bug 1190860 fixed, I can test fixes for bug 1188571, and with both, nothing else than tooltool will need to run under the mock environment to make reading the mozconfig happy.

Comment 12

[Sylvestre Ledru [:Sylvestre]]

Mihai, Rail, Ben, are you able to help here? This is blocking the gtb of 45 beta1.

Comment 13

[Carsten Book [:Tomcat]]

Android seems also to have problems, is this also this bug or do we need for android a seperate bug ?

Comment 14

[Rail Aliiev [:rail]]

I'm not sure if we can resolve this in hours to unblock the beta1 go.

We have been working on porting the release builders to use the same Mozharness scripts with Nightly/Aurora as a part of the Release Promotion project. The current expectation is to go live with next beta1. Sorry, this doesn't solve the current problem :(, but at least gives some hope. :)

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Not sure what you mean, what will happen if I start the build now?

What if we come back on gtk2? Will it fix the issue?

Comment 16

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Milan has some concerns about rolling back to gtk2. Ni'ing him here.

Comment 17

[aleth [:aleth]]

(In reply to Ritu Kothari (:ritu) from comment #16)
> Milan has some concerns about rolling back to gtk2. Ni'ing him here.

45 will be the next ESR release, and so stick around for a long time. That may be part of it.

Comment 18

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #10)
>  INFO: mock_mozilla.py version 1.0.3 starting...
>  State Changed: init plugins
>  INFO: selinux disabled
>  State Changed: start
>  State Changed: lock buildroot
>  State Changed: shell
>  '
>  State Changed: unlock buildroot
>  program finished with exit code 0
>  elapsedTime=4.924754
>  buildid: 'INFO: mock_mozilla.py version 1.0.3 starting...\nState Changed:
> init plugins\nINFO: selinux disabled\nState Changed: start\nState Changed:
> lock buildroot\nState Changed: shell\n20160125145626\nState Changed: unlock
> buildroot'
> 
> Which is a little confusing, as I think we should be at
>  
> http://hg.mozilla.org/build/buildbotcustom/file/d30c4d7a1504/steps/mock.
> py#l136
> and that's only paying attention to stdio. If I run the printconfigsetting
> command with '2> /dev/null' appended only '20160125145626' is returned.

Maybe mock_mozilla (not printconfigsetting) is writing those messages to stdout? If so, one could try changing (as a hack until comment 11 is implemented)
https://github.com/jhford/mock_mozilla/blob/923c9adefb330d45fd5fffa6679ad6788571ec85/py/mock_mozilla/util.py#L316

Comment 19

[Justin Wood (:Callek)]

Attachment: [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

I'm basing this patch on my understanding of what glandium was trying to do it related other patches.

Basically the gtk3 setup only needs to be called once, and this enables that. However it can't be called from outside Mock which is what happened last time we tried doing it this way. The idea here is to make Mock run tooltool instead, which should 'just-work'

We believe doing the setup.sh via tooltool (run inside Mock) will make printconfigsettings and related code that needs to parse the mozconfig continue to work as they once did and all is well.

I confirmed with puppet that the "setup" arg of tooltool is indeed present and should work, we don't need to worry about windows for this use-case at all.

Comment 20

[Michael Shal [:mshal]]

Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

I think it looks fine once the blockers are landed - would be nice to test it on try first when they are fixed.

Comment 21

[Justin Wood (:Callek)]

Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Approval Request Comment
I'm skipping the usual template, but the tl;dr is that if this works as I suspect it will have no product impact vs what we were building with on aurora.

Its basically moving where we invoke a necessary gtk3 setup script from one place (that works fine in mozharness-based jobs) to a different place due to complexities in the many things that parse mozconfigs.

It is also the initial intended way of calling this setup script, however we did not invest in improving the buildbot factories due to Mozharness and upcoming Release Promotion work, since the Buildbot scripts that we need to change will all be obsoleted by those changes.

I'm expecting to land on beta first, then let it ride release+esr, and use try to verify it for inbound+aurora after this sticks on beta.  (beta first to unblock b1, and because we can't test the spot in codepaths where we actually need this with a run on aurora or central/try)

Comment 22

[Mike Hommey [:glandium]]

Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Review of attachment 8712373 [details] [diff] [review]:
-----------------------------------------------------------------

There are a few more manifests with gtk3.tar.xz in them that would need the same change.

Comment 23

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Rubber stamping. This is needed to get gtk3 based builds working for 45.0b1.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8711880 [details] [diff] [review]
[buildbotcustom] Swap to MockProperty

Backed out - http://hg.mozilla.org/build/buildbotcustom/rev/9b9b645d00a1

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8712010 [details] [diff] [review]
[buildbotcustom] followup to fix cwd for mock call, and use python from the virtualenv

Backed out - http://hg.mozilla.org/build/buildbotcustom/rev/360f6f0cf623

Comment 26

[Mike Hommey [:glandium]]

As a followup a bunch of other mockifying patches in the past weeks could be backed out as well.

Comment 27

[Justin Wood (:Callek)]

Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Those other manifests were one under js/*  and then the ones under b2g/*

Since b2g/* is 0-effort suggested for platform team and since b2g has no jobs on beta/release/esr I didn't touch those (and am unlikely to touch when this lands to central). I will file a followup bug for the b2g "team" though.

Comment 28

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #27)
> Comment on attachment 8712373 [details] [diff] [review]
> [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> tooltool
> 
https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

That severity change went along with mozilla-beta reopening. We can start building 45.0b1 at Release Management's leisure.

TODO 
- backport changes to mozharness-builds on m-a and newer
- be nicer to b2g manifests
- comment #26 ?

Comment 30

[aleth [:aleth]]

(In reply to Justin Wood (:Callek) from comment #28)
> (In reply to Justin Wood (:Callek) from comment #27)
> > Comment on attachment 8712373 [details] [diff] [review]
> > [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> > tooltool
> > 
> https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5

Will this also land on mozilla-central? Comm-central will require this patch to build, but currently check-sync-dirs enforces that comm-*'s build/ and tooltool-manifests/ match those in mozilla-*.

Comment 31

[aleth [:aleth]]

https://hg.mozilla.org/releases/comm-beta/rev/ff9752bfd9418909cd707269d4c6cf761a5918eb
Port Bug 1242641 - GTK+3 still not working for buildbot builds on beta. rs=bustage-fix a=aleth

Comment 32

[Justin Wood (:Callek)]

(In reply to aleth [:aleth] from comment #30)
> (In reply to Justin Wood (:Callek) from comment #28)
> > (In reply to Justin Wood (:Callek) from comment #27)
> > > Comment on attachment 8712373 [details] [diff] [review]
> > > [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> > > tooltool
> > > 
> > https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5
> 
> Will this also land on mozilla-central? Comm-central will require this patch
> to build, but currently check-sync-dirs enforces that comm-*'s build/ and
> tooltool-manifests/ match those in mozilla-*.

Yes, and I can take care of that for you when I land elsewhere.

Comment 33

[Justin Wood (:Callek)]

(In reply to Nick Thomas [:nthomas] from comment #29)
> That severity change went along with mozilla-beta reopening. We can start
> building 45.0b1 at Release Management's leisure.
> 
> TODO 
> - backport changes to mozharness-builds on m-a and newer
> - be nicer to b2g manifests
> - comment #26 ?

Assigning to myself to make sure the patches land on m-a/m-c branches. (I'll do a try run) along with a "be nice to b2g" piece. If mozharness-builds need a change I may be able to do that as well.

Comment 34

[Landry Breuil (:gaston)]

Dunno if that's to be tracked here or in a new bug, but it seems in 45.0b1 build1 the source packaging step failed, and this didnt prevent 45.0b1 build1 to be promoted from candidate to release:
http://releases.mozilla.org/pub/firefox/releases/45.0b1/

Per http://releases.mozilla.org/pub/firefox/candidates/45.0b1-candidates/build1/logs/release-mozilla-beta-firefox_source-bm74-build1-build7.txt.gz

===
checking for application to build... browser
checking if app-specific confvars.sh exists... /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/browser/confvars.sh
/builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/configure: line 18050: /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/./gtk3/usr/local/bin/pkg-config: No such file or directory
*** Your version of pkg-config is too old. You need version 0.9.0 or newer.
*** See http://www.freedesktop.org/software/pkgconfig
configure: error: Library requirements (gtk+-3.0 >= 3.4.0 gtk+-unix-print-3.0 glib-2.0 gobject-2.0 ) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them.
===

Distributors somewhat rely on having the source tarball available in the source/ subdir...

Comment 35

[Mike Hommey [:glandium]]

Why are those jobs using a normal mozconfig? They only need configure to run the `make source` target, so they should work with a mozconfig that only contains `ac_add_options --disable-compile-environment`

Comment 36

[Rail Aliiev [:rail]]

(In reply to Mike Hommey [:glandium] from comment #35)
> Why are those jobs using a normal mozconfig?

Never bothered ourselves to create a separate one for source generation I guess?...

> They only need configure to run
> the `make source` target, so they should work with a mozconfig that only
> contains `ac_add_options --disable-compile-environment`

I tried `ac_add_options --disable-compile-environment` "on my laptop" and it still tries to find all required libraries.

Comment 37

[Mike Hommey [:glandium]]

Well, then, they need to run tooltool.

Comment 38

[Justin Wood (:Callek)]

(In reply to Mike Hommey [:glandium] from comment #37)
> Well, then, they need to run tooltool.

No they don't -- Bug 1243448

I'm helping to figure out what is *actually* required in that mozconfig though

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

https://treeherder.mozilla.org/#/jobs?repo=try&exclusion_profile=false&revision=f817b378fb01 for a push with all manifests that use gtk3 to run setup, and tooltool run in mock where it wasn't, plus assorted files that need to be copied into mock to make it work. I don't really understand why the emulator builds need gtk3 at all though

On a previous run the Valgrind job found something (https://treeherder.mozilla.org/logviewer.html#?job_id=16074640&repo=try), which glandium says might be due the missing FONTCONFIG_PATH from https://hg.mozilla.org/try/rev/8fa1085249c7#l13.6 in https://dxr.mozilla.org/mozilla-central/source/build/unix/build-gtk3/build-gtk3.sh#117. We'll have to try a new gtk3 archive for that.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Make that https://treeherder.mozilla.org/#/jobs?repo=try&exclusion_profile=false&revision=33a8ce50834b (for  another android fix).

Comment 41

[aleth [:aleth]]

(In reply to Justin Wood (:Callek) from comment #32)
> > > https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5
> > 
> > Will this also land on mozilla-central? Comm-central will require this patch
> > to build, but currently check-sync-dirs enforces that comm-*'s build/ and
> > tooltool-manifests/ match those in mozilla-*.
> 
> Yes, and I can take care of that for you when I land elsewhere.

Thanks! Just adding a needinfo in case this got overlooked among all the other issues...

Comment 42

[Justin Wood (:Callek)]

Attachment: [central] run gtk3 setup via tooltool, including b2g and mozharness

This was written by nick (as a forward port of my beta patch) and tested on try by him successfully

Comment 43

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/65e246baede4

Comment 44

[Justin Wood (:Callek)]

Comment on attachment 8715794 [details] [diff] [review]
[central] run gtk3 setup via tooltool, including b2g and mozharness

turns out I dont' have an aurora branch handy.

Sheriffs, can you please land this on aurora, a=NPOTDB (and already had a+ for beta)

If there are non-obvious conflicts pop me a n-i and I'll clone and graft to resolve them

[aleth landed the comm part for me already]

Comment 45

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'm backing this out for valgrind failures: https://treeherder.mozilla.org/logviewer.html#?job_id=21079562&repo=mozilla-inbound

https://hg.mozilla.org/integration/mozilla-inbound/rev/979832bbcc99

Comment 46

[Justin Wood (:Callek)]

:glandium, I had missed the relevant part of c#39 here (and mistakenly thought valgrind wasn't tier 1 when I skimmed try's results).

Any chance you can help unblock this? [since it is needed to land before the next uplift and I want to have it off my plate before I end up on leave soon]

Comment 47

[Mike Hommey [:glandium]]

You just need to grab the gtk3 archive from tooltool, extract it, attempt to modify setup.sh to set FONTCONFIG_PATH when running fc-cache, create a new archive, upload it to tooltool, and push a test with it to try.

Comment 48

[Justin Wood (:Callek)]

(In reply to Mike Hommey [:glandium] from comment #47)
> You just need to grab the gtk3 archive from tooltool, extract it, attempt to
> modify setup.sh to set FONTCONFIG_PATH when running fc-cache, create a new
> archive, upload it to tooltool, and push a test with it to try.

Pushed to try with: https://treeherder.mozilla.org/#/jobs?repo=try&revision=81d71b243338

Comment 49

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #48)
> Pushed to try with:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=81d71b243338

...

take 2 -- with proper size: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fe2e5d64a1e

Comment 50

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #49)
> take 2 -- with proper size:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fe2e5d64a1e

Take 3 - without the quotation typo:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f95f411d564f

Comment 51

[Justin Wood (:Callek)]

Ok, still failing after doing everything I can think of, jseward/glandium, ideas/fixes?

https://hg.mozilla.org/try/rev/cc33a8837670

Comment 52

[Justin Wood (:Callek)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cc33a8837670&selectedJob=16423436

Comment 53

[Mike Hommey [:glandium]]

Try adding
LD_LIBRARY_PATH=./usr/local/lib \

Comment 54

[Justin Wood (:Callek)]

(In reply to Mike Hommey [:glandium] from comment #53)
> Try adding
> LD_LIBRARY_PATH=./usr/local/lib \

Nope: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc8de22feda7&selectedJob=16447943

Comment 55

[Justin Wood (:Callek)]

Attachment: setup.sh

OK, finally I was able to coerce it to not fail, BUT I did so by basically adding *everything* in the mozconfig to this invoke, I have no clue whats necessary. And I also did it by marking it as an absolute path rather than a relative one...

Glandium, can you advice on the way to fixup this and the build script to actually keep this result achieved, but removing things likely unnecessary.

Comment 56

[Mike Hommey [:glandium]]

Looks like the HERE change does something, so it's possible that it'd work with $HERE, FONTCONFIG_PATH and LD_LIBRARY_PATH only.

Comment 57

[Justin Wood (:Callek)]

Attachment: [m-c] update gtk3 packages and the build-gtk script to use newer setup

This updates the setup.sh inside the gtk3 archive(s) and uses absolute paths instead of relative ones now.

This is on top of the previously reviewed patch in this bug that adds the setup.sh to the tooltool invoke and adjusted mozharness.

I pushed to try with https://treeherder.mozilla.org/#/jobs?repo=try&revision=27267341f226 ontop of current m-c for sanity

Comment 58

[Mike Hommey [:glandium]]

Comment on attachment 8717230 [details] [diff] [review]
[m-c] update gtk3 packages and the build-gtk script to use newer setup

Review of attachment 8717230 [details] [diff] [review]:
-----------------------------------------------------------------

Your gtk3.tar.xz contains gtk3/usr/local/etc/pango/pango.modules and gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache, which shouldn't be there.

Comment 59

[Justin Wood (:Callek)]

Attachment: [m-c] update gtk3 packages and the build-gtk script to use newer setup (take 2)

That happened because I was using tooltool itself to fetch the old binaries, extracting them, then repacking them. Initially I let it run setup.sh by mistake and I've been doing this based on the last uploaded gtk3 tarball. So I was doing that in a loop by accident.

This batch pulled the currently-in-m-c version and started from scratch. I verified those mentioned files are missing.

Pushed this to try with: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cb931a12351b

Comment 60

[Sylvestre Ledru [:Sylvestre]]

We disabled GTK 3 in 45, blocking 46 now.

Comment 61

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/09064b8c1d07
https://hg.mozilla.org/integration/mozilla-inbound/rev/5a7d53bdedbb

Comment 62

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/09064b8c1d07
https://hg.mozilla.org/mozilla-central/rev/5a7d53bdedbb

Comment 63

[aleth [:aleth]]

Thanks for fixing this!

Comment 64

[aleth [:aleth]]

This should land on aurora as well, but I'm not sure if it would be exactly the same patch.

Comment 66

[Jorg K (CEST = GMT+2)]

Sorry to be impatient. Is anything going to happen for Aurora (FF 46) here?

Comment 67

[aleth [:aleth]]

(In reply to Jorg K (GMT+1) from comment #66)
> Sorry to be impatient. Is anything going to happen for Aurora (FF 46) here?

I believe Callek is on top of this ;)

Comment 68

[Carsten Book [:Tomcat]]

Liz, Sylvestre, also approval for mozilla-aurora ?

Comment 69

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes, I didn't see an approval request probably since there isn't a patch yet.... But please go for it.

Comment 70

[Sylvestre Ledru [:Sylvestre]]

Just like liz said!

Comment 71

[Carsten Book [:Tomcat]]

remote: View your changes here:
remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/7ba5900cf9fd
remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/be58fc73927c