Closed Bug 1242641 Opened 4 years ago Closed 4 years ago

GTK+3 still not working for buildbot builds on beta

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox45-, firefox46blocking fixed, firefox47 fixed)

RESOLVED FIXED
Tracking Status
firefox45 - ---
firefox46 blocking fixed
firefox47 --- fixed

People

(Reporter: philor, Assigned: Callek)

References

Details

Attachments

(5 files, 2 obsolete files)

This looks a little bogus at the --buildid argument:

Executing: ['python', '/builds/slave/m-beta-lx-00000000000000000000/tools/buildfarm/utils/graph_server_post.py', '--server', 'graphs.mozilla.org', '--selector', '/server/collect.cgi', '--branch', 'Mozilla-Beta', '--buildid', 'Error loading mozconfig: /builds/slave/m-beta-lx-00000000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not e....

which stems from the earlier step:

 ========= Started set props: sourcestamp (results: 2, elapsed: 0 secs) (at 2016-01-25 09:38:14.834835) =========
 /tools/buildbot/bin/python /builds/slave/m-beta-lx-00000000000000000000/build/mach python /builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini App SourceStamp
  in dir /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox (timeout 1200 secs)
  watching logfiles {}
  argv: ['/tools/buildbot/bin/python', '/builds/slave/m-beta-lx-00000000000000000000/build/mach', 'python', '/builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py', '/builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini', 'App', 'SourceStamp']
  environment:
   CCACHE_COMPRESS=1
...
   _=/tools/buildbot/bin/python
  using PTY: False
 Error loading mozconfig: /builds/slave/m-beta-lx-00000000000000000000/build/.mozconfig
 
 Evaluation of your mozconfig exited with an error. This could be triggered
 by a command inside your mozconfig failing. Please change your mozconfig
 to not error and/or to catch errors in executed commands.
 
 mozconfig output:
 
 ------BEGIN_AC_OPTION
 --enable-update-channel=
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-update-packaging
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-google-api-keyfile=/builds/gapi.data
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-google-oauth-api-keyfile=/builds/google-oauth-api.key
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --with-mozilla-api-keyfile=/builds/mozilla-desktop-geoloc-api.key
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 AUTOCLOBBER=1
 ------END_MK_OPTION
 ------BEGIN_AC_OPTION
 --enable-crashreporter
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-release
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_BUILD_SYMBOLS=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_L10N_CHECK=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_PACKAGE=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_PACKAGE_TESTS=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_INSTALLER=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPDATE_PACKAGING=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPLOAD=1
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export MOZ_AUTOMATION_SDK=0
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
 ------END_MK_OPTION
 ------BEGIN_AC_OPTION
 --enable-elf-hack
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-stdcxx-compat
 ------END_AC_OPTION
 ------BEGIN_AC_OPTION
 --enable-default-toolkit=cairo-gtk3
 ------END_AC_OPTION
 ------BEGIN_MK_OPTION
 export FONTCONFIG_PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/etc/fonts
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/etc
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export PANGO_LIBDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders
 ------END_MK_OPTION
 ------BEGIN_MK_OPTION
 export LD_LIBRARY_PATH=/builds/slave/m-beta-lx-00000000000000000000/build/gtk3/usr/local/lib
 ------END_MK_OPTION
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 /builds/slave/m-beta-lx-00000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
 program finished with exit code 1
 elapsedTime=0.232137
This has rather wider effect than just linux32 builds, and change linux32, linux64, and android builds on m-b, m-r, m-esr38, as well some TB builds. It should do the trick though.
Attachment #8711880 - Flags: review?(bugspam.Callek)
Assignee: nobody → nthomas
Comment on attachment 8711880 [details] [diff] [review]
[buildbotcustom] Swap to MockProperty

Review of attachment 8711880 [details] [diff] [review]:
-----------------------------------------------------------------

I should note, taht almost every touch of SetProperty for printconfigsettings in these past cycles has broken Thunderbird (and vice versa) so don't be shocked if we get reports from that step-child. But if this works to fix Firefox, I'm happy.
Attachment #8711880 - Flags: review?(bugspam.Callek) → review+
Reconfig done, Kwierso is going to push bug 1237179 to see how we do.
Not working, mock doesn't want to start. See also bug 1232466, where we had a very similar issue last cycle.
See Also: → 1232466
Fixing bug 1190860 and finally using setup scripts from tooltool for gtk3 would solve this.
I landed this patch because it's for because blocker bug
https://hg.mozilla.org/build/buildbotcustom/rev/3b001e0b8438
https://hg.mozilla.org/build/buildbotcustom/rev/d30c4d7a1504
but please take a look for all the context I lost while away.

It may be that we can stop using '/path/to/python mach python /path/to/printconfigsettings.py' everywhere, but I'd created enough bustage in one day.

Reconfig'd, retriggered some builds.
Attachment #8712010 - Flags: review?(bugspam.Callek)
To explain the previous patch, we were making mock calls like
 mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-beta-lx-d-000000000000000000//builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox
which the 'mock_workdir_prefix=None' fixes by removing the duplicated part before the //.

And then we need to use the python in the virtualenv the build system has created, which is derived from the one installed inside mock, instead of the one which mock can't see in /tools. Using the virtualenv python can maybe be generalized to all platforms, but this code is going to die with build promotion so I'm not convinced it's worth the testing time.
We are getting properties set now but the values are still bogus, eg:

https://treeherder.mozilla.org/logviewer.html#?job_id=745781&repo=mozilla-beta

 ========= Started set props: buildid (results: 0, elapsed: 4 secs) (at 2016-01-25 23:22:05.112291) =========
 mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox --unpriv --shell '/usr/bin/env 
...
 CCACHE_UMASK="002" '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/mach'"'"' python '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/config/printconfigsetting.py'"'"' '"'"'/builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox/dist/bin/application.ini'"'"' App BuildID'
  in dir /builds/slave/m-beta-lx-00000000000000000000/build/obj-firefox (timeout 1200 secs)
...
 INFO: mock_mozilla.py version 1.0.3 starting...
 State Changed: init plugins
 INFO: selinux disabled
 State Changed: start
 State Changed: lock buildroot
 State Changed: shell
 '
 State Changed: unlock buildroot
 program finished with exit code 0
 elapsedTime=4.924754
 buildid: 'INFO: mock_mozilla.py version 1.0.3 starting...\nState Changed: init plugins\nINFO: selinux disabled\nState Changed: start\nState Changed: lock buildroot\nState Changed: shell\n20160125145626\nState Changed: unlock buildroot'

Which is a little confusing, as I think we should be at
  http://hg.mozilla.org/build/buildbotcustom/file/d30c4d7a1504/steps/mock.py#l136
and that's only paying attention to stdio. If I run the printconfigsetting command with '2> /dev/null' appended only '20160125145626' is returned.

I'm going to leave this here for now and let someone else pick it up when they start their day. Please be aware the attempted fixes have broken builds on m-esr38 and m-r, and probably impacted Thunderbird too.
Assignee: nthomas → nobody
We should just stop papering over this issue by using mock everywhere. The core problem is bug 1188571, which is that the mozconfig for gtk3 is actively doing stuff, and everything that uses the mozconfig ends up doing that stuff. That stuff needs to run in the mock environment for a sum of reasons. There is no patch attached to bug 1188571 to actually finish it, but it's trivial to write one once it can be tested on try... which it can't. Because fixing bug 1188571 relies on tooltool running a setup script that would do that stuff instead of the mozconfig, which means tooltool would need to run in the mock environment. Which is bug 1190860.

So, with bug 1190860 fixed, I can test fixes for bug 1188571, and with both, nothing else than tooltool will need to run under the mock environment to make reading the mozconfig happy.
Mihai, Rail, Ben, are you able to help here? This is blocking the gtb of 45 beta1.
Flags: needinfo?(rail)
Flags: needinfo?(mtabara)
Flags: needinfo?(bhearsum)
Android seems also to have problems, is this also this bug or do we need for android a seperate bug ?
I'm not sure if we can resolve this in hours to unblock the beta1 go.

We have been working on porting the release builders to use the same Mozharness scripts with Nightly/Aurora as a part of the Release Promotion project. The current expectation is to go live with next beta1. Sorry, this doesn't solve the current problem :(, but at least gives some hope. :)
Flags: needinfo?(rail)
Not sure what you mean, what will happen if I start the build now?

What if we come back on gtk2? Will it fix the issue?
Milan has some concerns about rolling back to gtk2. Ni'ing him here.
Flags: needinfo?(milan)
(In reply to Ritu Kothari (:ritu) from comment #16)
> Milan has some concerns about rolling back to gtk2. Ni'ing him here.

45 will be the next ESR release, and so stick around for a long time. That may be part of it.
(In reply to Nick Thomas [:nthomas] from comment #10)
>  INFO: mock_mozilla.py version 1.0.3 starting...
>  State Changed: init plugins
>  INFO: selinux disabled
>  State Changed: start
>  State Changed: lock buildroot
>  State Changed: shell
>  '
>  State Changed: unlock buildroot
>  program finished with exit code 0
>  elapsedTime=4.924754
>  buildid: 'INFO: mock_mozilla.py version 1.0.3 starting...\nState Changed:
> init plugins\nINFO: selinux disabled\nState Changed: start\nState Changed:
> lock buildroot\nState Changed: shell\n20160125145626\nState Changed: unlock
> buildroot'
> 
> Which is a little confusing, as I think we should be at
>  
> http://hg.mozilla.org/build/buildbotcustom/file/d30c4d7a1504/steps/mock.
> py#l136
> and that's only paying attention to stdio. If I run the printconfigsetting
> command with '2> /dev/null' appended only '20160125145626' is returned.

Maybe mock_mozilla (not printconfigsetting) is writing those messages to stdout? If so, one could try changing (as a hack until comment 11 is implemented)
https://github.com/jhford/mock_mozilla/blob/923c9adefb330d45fd5fffa6679ad6788571ec85/py/mock_mozilla/util.py#L316
I'm basing this patch on my understanding of what glandium was trying to do it related other patches.

Basically the gtk3 setup only needs to be called once, and this enables that. However it can't be called from outside Mock which is what happened last time we tried doing it this way. The idea here is to make Mock run tooltool instead, which should 'just-work'

We believe doing the setup.sh via tooltool (run inside Mock) will make printconfigsettings and related code that needs to parse the mozconfig continue to work as they once did and all is well.

I confirmed with puppet that the "setup" arg of tooltool is indeed present and should work, we don't need to worry about windows for this use-case at all.
Attachment #8712373 - Flags: review?(mshal)
Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

I think it looks fine once the blockers are landed - would be nice to test it on try first when they are fixed.
Attachment #8712373 - Flags: review?(mshal) → review+
Depends on: 1190860, 1188571
Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Approval Request Comment
I'm skipping the usual template, but the tl;dr is that if this works as I suspect it will have no product impact vs what we were building with on aurora.

Its basically moving where we invoke a necessary gtk3 setup script from one place (that works fine in mozharness-based jobs) to a different place due to complexities in the many things that parse mozconfigs.

It is also the initial intended way of calling this setup script, however we did not invest in improving the buildbot factories due to Mozharness and upcoming Release Promotion work, since the Buildbot scripts that we need to change will all be obsoleted by those changes.

I'm expecting to land on beta first, then let it ride release+esr, and use try to verify it for inbound+aurora after this sticks on beta.  (beta first to unblock b1, and because we can't test the spot in codepaths where we actually need this with a run on aurora or central/try)
Attachment #8712373 - Flags: approval-mozilla-beta?
Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Review of attachment 8712373 [details] [diff] [review]:
-----------------------------------------------------------------

There are a few more manifests with gtk3.tar.xz in them that would need the same change.
Attachment #8712373 - Flags: review+
Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Rubber stamping. This is needed to get gtk3 based builds working for 45.0b1.
Attachment #8712373 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Comment on attachment 8711880 [details] [diff] [review]
[buildbotcustom] Swap to MockProperty

Backed out - http://hg.mozilla.org/build/buildbotcustom/rev/9b9b645d00a1
Attachment #8711880 - Attachment description: [buildbot-custom] Swap to MockProperty → [buildbotcustom] Swap to MockProperty
Attachment #8711880 - Flags: checked-in+ → checked-in-
Comment on attachment 8712010 [details] [diff] [review]
[buildbotcustom] followup to fix cwd for mock call, and use python from the virtualenv

Backed out - http://hg.mozilla.org/build/buildbotcustom/rev/360f6f0cf623
Attachment #8712010 - Flags: review?(bugspam.Callek) → checked-in-
As a followup a bunch of other mockifying patches in the past weeks could be backed out as well.
Comment on attachment 8712373 [details] [diff] [review]
[mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in tooltool

Those other manifests were one under js/*  and then the ones under b2g/*

Since b2g/* is 0-effort suggested for platform team and since b2g has no jobs on beta/release/esr I didn't touch those (and am unlikely to touch when this lands to central). I will file a followup bug for the b2g "team" though.
Attachment #8712373 - Flags: checked-in+
(In reply to Justin Wood (:Callek) from comment #27)
> Comment on attachment 8712373 [details] [diff] [review]
> [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> tooltool
> 
https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5
Severity: blocker → normal
That severity change went along with mozilla-beta reopening. We can start building 45.0b1 at Release Management's leisure.

TODO 
- backport changes to mozharness-builds on m-a and newer
- be nicer to b2g manifests
- comment #26 ?
Flags: needinfo?(mtabara)
Flags: needinfo?(milan)
Flags: needinfo?(bhearsum)
(In reply to Justin Wood (:Callek) from comment #28)
> (In reply to Justin Wood (:Callek) from comment #27)
> > Comment on attachment 8712373 [details] [diff] [review]
> > [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> > tooltool
> > 
> https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5

Will this also land on mozilla-central? Comm-central will require this patch to build, but currently check-sync-dirs enforces that comm-*'s build/ and tooltool-manifests/ match those in mozilla-*.
https://hg.mozilla.org/releases/comm-beta/rev/ff9752bfd9418909cd707269d4c6cf761a5918eb
Port Bug 1242641 - GTK+3 still not working for buildbot builds on beta. rs=bustage-fix a=aleth
Flags: needinfo?(bugspam.Callek)
Blocks: 1234220
(In reply to aleth [:aleth] from comment #30)
> (In reply to Justin Wood (:Callek) from comment #28)
> > (In reply to Justin Wood (:Callek) from comment #27)
> > > Comment on attachment 8712373 [details] [diff] [review]
> > > [mozilla-beta] Remove setup.sh from mozconfig use, enable its calling in
> > > tooltool
> > > 
> > https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5
> 
> Will this also land on mozilla-central? Comm-central will require this patch
> to build, but currently check-sync-dirs enforces that comm-*'s build/ and
> tooltool-manifests/ match those in mozilla-*.

Yes, and I can take care of that for you when I land elsewhere.
Flags: needinfo?(bugspam.Callek)
(In reply to Nick Thomas [:nthomas] from comment #29)
> That severity change went along with mozilla-beta reopening. We can start
> building 45.0b1 at Release Management's leisure.
> 
> TODO 
> - backport changes to mozharness-builds on m-a and newer
> - be nicer to b2g manifests
> - comment #26 ?

Assigning to myself to make sure the patches land on m-a/m-c branches. (I'll do a try run) along with a "be nice to b2g" piece. If mozharness-builds need a change I may be able to do that as well.
Assignee: nobody → bugspam.Callek
Dunno if that's to be tracked here or in a new bug, but it seems in 45.0b1 build1 the source packaging step failed, and this didnt prevent 45.0b1 build1 to be promoted from candidate to release:
http://releases.mozilla.org/pub/firefox/releases/45.0b1/

Per http://releases.mozilla.org/pub/firefox/candidates/45.0b1-candidates/build1/logs/release-mozilla-beta-firefox_source-bm74-build1-build7.txt.gz

===
checking for application to build... browser
checking if app-specific confvars.sh exists... /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/browser/confvars.sh
/builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/configure: line 18050: /builds/slave/rel-m-beta-fx_source-000000000/mozilla-beta/./gtk3/usr/local/bin/pkg-config: No such file or directory
*** Your version of pkg-config is too old. You need version 0.9.0 or newer.
*** See http://www.freedesktop.org/software/pkgconfig
configure: error: Library requirements (gtk+-3.0 >= 3.4.0 gtk+-unix-print-3.0 glib-2.0 gobject-2.0 ) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them.
===

Distributors somewhat rely on having the source tarball available in the source/ subdir...
Why are those jobs using a normal mozconfig? They only need configure to run the `make source` target, so they should work with a mozconfig that only contains `ac_add_options --disable-compile-environment`
(In reply to Mike Hommey [:glandium] from comment #35)
> Why are those jobs using a normal mozconfig?

Never bothered ourselves to create a separate one for source generation I guess?...

> They only need configure to run
> the `make source` target, so they should work with a mozconfig that only
> contains `ac_add_options --disable-compile-environment`

I tried `ac_add_options --disable-compile-environment` "on my laptop" and it still tries to find all required libraries.
Well, then, they need to run tooltool.
(In reply to Mike Hommey [:glandium] from comment #37)
> Well, then, they need to run tooltool.

No they don't -- Bug 1243448

I'm helping to figure out what is *actually* required in that mozconfig though
https://treeherder.mozilla.org/#/jobs?repo=try&exclusion_profile=false&revision=f817b378fb01 for a push with all manifests that use gtk3 to run setup, and tooltool run in mock where it wasn't, plus assorted files that need to be copied into mock to make it work. I don't really understand why the emulator builds need gtk3 at all though

On a previous run the Valgrind job found something (https://treeherder.mozilla.org/logviewer.html#?job_id=16074640&repo=try), which glandium says might be due the missing FONTCONFIG_PATH from https://hg.mozilla.org/try/rev/8fa1085249c7#l13.6 in https://dxr.mozilla.org/mozilla-central/source/build/unix/build-gtk3/build-gtk3.sh#117. We'll have to try a new gtk3 archive for that.
(In reply to Justin Wood (:Callek) from comment #32)
> > > https://hg.mozilla.org/releases/mozilla-beta/rev/cb40dba596a5
> > 
> > Will this also land on mozilla-central? Comm-central will require this patch
> > to build, but currently check-sync-dirs enforces that comm-*'s build/ and
> > tooltool-manifests/ match those in mozilla-*.
> 
> Yes, and I can take care of that for you when I land elsewhere.

Thanks! Just adding a needinfo in case this got overlooked among all the other issues...
Flags: needinfo?(bugspam.Callek)
This was written by nick (as a forward port of my beta patch) and tested on try by him successfully
Attachment #8715794 - Flags: review?(rail)
Attachment #8715794 - Flags: review?(rail) → review+
Blocks: 1245828
Comment on attachment 8715794 [details] [diff] [review]
[central] run gtk3 setup via tooltool, including b2g and mozharness

turns out I dont' have an aurora branch handy.

Sheriffs, can you please land this on aurora, a=NPOTDB (and already had a+ for beta)

If there are non-obvious conflicts pop me a n-i and I'll clone and graft to resolve them

[aleth landed the comm part for me already]
Flags: needinfo?(bugspam.Callek)
Attachment #8715794 - Flags: checked-in?
:glandium, I had missed the relevant part of c#39 here (and mistakenly thought valgrind wasn't tier 1 when I skimmed try's results).

Any chance you can help unblock this? [since it is needed to land before the next uplift and I want to have it off my plate before I end up on leave soon]
Flags: needinfo?(bugspam.Callek) → needinfo?(mh+mozilla)
You just need to grab the gtk3 archive from tooltool, extract it, attempt to modify setup.sh to set FONTCONFIG_PATH when running fc-cache, create a new archive, upload it to tooltool, and push a test with it to try.
Flags: needinfo?(mh+mozilla)
(In reply to Mike Hommey [:glandium] from comment #47)
> You just need to grab the gtk3 archive from tooltool, extract it, attempt to
> modify setup.sh to set FONTCONFIG_PATH when running fc-cache, create a new
> archive, upload it to tooltool, and push a test with it to try.

Pushed to try with: https://treeherder.mozilla.org/#/jobs?repo=try&revision=81d71b243338
(In reply to Justin Wood (:Callek) from comment #48)
> Pushed to try with:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=81d71b243338

...

take 2 -- with proper size: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fe2e5d64a1e
(In reply to Justin Wood (:Callek) from comment #49)
> take 2 -- with proper size:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fe2e5d64a1e

Take 3 - without the quotation typo:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f95f411d564f
Ok, still failing after doing everything I can think of, jseward/glandium, ideas/fixes?

https://hg.mozilla.org/try/rev/cc33a8837670
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(jseward)
Try adding
LD_LIBRARY_PATH=./usr/local/lib \
Flags: needinfo?(mh+mozilla)
(In reply to Mike Hommey [:glandium] from comment #53)
> Try adding
> LD_LIBRARY_PATH=./usr/local/lib \

Nope: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc8de22feda7&selectedJob=16447943
Flags: needinfo?(mh+mozilla)
Attached file setup.sh (obsolete) —
OK, finally I was able to coerce it to not fail, BUT I did so by basically adding *everything* in the mozconfig to this invoke, I have no clue whats necessary. And I also did it by marking it as an absolute path rather than a relative one...

Glandium, can you advice on the way to fixup this and the build script to actually keep this result achieved, but removing things likely unnecessary.
Flags: needinfo?(jseward)
Attachment #8715794 - Flags: checked-in?
Looks like the HERE change does something, so it's possible that it'd work with $HERE, FONTCONFIG_PATH and LD_LIBRARY_PATH only.
Flags: needinfo?(mh+mozilla)
This updates the setup.sh inside the gtk3 archive(s) and uses absolute paths instead of relative ones now.

This is on top of the previously reviewed patch in this bug that adds the setup.sh to the tooltool invoke and adjusted mozharness.

I pushed to try with https://treeherder.mozilla.org/#/jobs?repo=try&revision=27267341f226 ontop of current m-c for sanity
Attachment #8716824 - Attachment is obsolete: true
Attachment #8717230 - Flags: review?(mh+mozilla)
Attachment #8717230 - Attachment is patch: true
Comment on attachment 8717230 [details] [diff] [review]
[m-c] update gtk3 packages and the build-gtk script to use newer setup

Review of attachment 8717230 [details] [diff] [review]:
-----------------------------------------------------------------

Your gtk3.tar.xz contains gtk3/usr/local/etc/pango/pango.modules and gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache, which shouldn't be there.
Attachment #8717230 - Flags: review?(mh+mozilla)
That happened because I was using tooltool itself to fetch the old binaries, extracting them, then repacking them. Initially I let it run setup.sh by mistake and I've been doing this based on the last uploaded gtk3 tarball. So I was doing that in a loop by accident.

This batch pulled the currently-in-m-c version and started from scratch. I verified those mentioned files are missing.

Pushed this to try with: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cb931a12351b
Attachment #8717230 - Attachment is obsolete: true
Attachment #8717460 - Flags: review?(mh+mozilla)
Attachment #8717460 - Flags: review?(mh+mozilla) → review+
We disabled GTK 3 in 45, blocking 46 now.
Thanks for fixing this!
This should land on aurora as well, but I'm not sure if it would be exactly the same patch.
Flags: needinfo?(bugspam.Callek)
Duplicate of this bug: 1188571
Sorry to be impatient. Is anything going to happen for Aurora (FF 46) here?
Flags: needinfo?(mh+mozilla)
(In reply to Jorg K (GMT+1) from comment #66)
> Sorry to be impatient. Is anything going to happen for Aurora (FF 46) here?

I believe Callek is on top of this ;)
Flags: needinfo?(mh+mozilla)
Liz, Sylvestre, also approval for mozilla-aurora ?
Flags: needinfo?(sledru)
Flags: needinfo?(lhenry)
Flags: needinfo?(lhenry)
Yes, I didn't see an approval request probably since there isn't a patch yet.... But please go for it.
Just like liz said!
Flags: needinfo?(sledru)
Flags: needinfo?(bugspam.Callek)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.