1196595 - Update the old emulator tooltool configs so it would install gtk3

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ricky Chien [:rickychien] (inactive)]

According to https://bugzilla.mozilla.org/show_bug.cgi?id=1146713#c70, latest b2g desktop depends on gtk3 for some reasons (see also bug 1189125) that break gaia build if we try to upgrade b2g to 43.0a1.

A proper fix is we should install gtk libs on build machines.

Comment 1

[Ricky Chien [:rickychien] (inactive)]

I need TC folks's feedbacks since that blocks bug 1146713.

Comment 2

[Michael Shal [:mshal]]

We install gtk3 using tooltool - it's not on the build machines by default. See for example:

https://dxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/linux64/releng.manifest#13

Comment 3

[Ricky Chien [:rickychien] (inactive)]

(In reply to Michael Shal [:mshal] from comment #2)
> We install gtk3 using tooltool - it's not on the build machines by default.
> See for example:
> 
> https://dxr.mozilla.org/mozilla-central/source/browser/config/tooltool-
> manifests/linux64/releng.manifest#13

ok, so do you know who can let it install on the build machines by default? It's an urgent 2.5 blocker.

Comment 4

[Michael Shal [:mshal]]

I'm not sure I understand - why can't you add gtk3.tar.xz to the relevant tooltool manifest?

Comment 5

[Ricky Chien [:rickychien] (inactive)]

I'm not familiar with b2g or gecko stuffs. I think it's proper to ask releng for help.

Comment 6

[Michael Shal [:mshal]]

Can you point me to a build log on treeherder that this is affecting? From bug 1146713 it sounds like this is for B2G Desktop, but gtk3 should already be installed for those builds from the tooltool manifest:

https://dxr.mozilla.org/mozilla-central/source/b2g/config/tooltool-manifests/linux64/releng.manifest#13

Comment 7

[Ricky Chien [:rickychien] (inactive)]

Here https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=d255da3b284c

And I saw this message:

home/worker/workspace/gaia/b2g_sdk/43.0a1-2015-08-17-15-02-06/b2g/xpcshell: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory

it probably relate to gtk3

Comment 8

[Ricky Chien [:rickychien] (inactive)]

Log from TC

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/7bCnRLKrROyf1imOn3Tilg/0/public/logs/live_backing.log

Comment 9

[Michael Shal [:mshal]]

From that log you can see that gtk3 is installed:

INFO - File gtk3.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /home/worker/workspace/gecko/tmpKTIKLf
...
INFO - untarring "gtk3.tar.xz"

We probably just need to setup LD_LIBRARY_PATH to point to $topsrcdir/gtk3/usr/local/lib when running xpcshell. :glandium, do you have a recommendation here?

Comment 10

[Kim Moir [:kmoir] ET]

changing priority so we stop getting alerts in #buildduty

Comment 11

[Mike Hommey [:glandium]]

How is this xpcshell run?

Comment 12

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Invoke from command line from Make

https://github.com/mozilla-b2g/gaia/blob/master/build/common.mk#L20-L24

Comment 13

[Mike Hommey [:glandium]]

This is not invoked recursively from the gecko build system is it?

Comment 14

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Unfortunately no.

Comment 15

[Blake Wu [:bwu][:blakewu]]

I would like to change the severity back to "blocker" since this bug impacts B2G Emulator KK and many codes on 2.5 cannot be tested now.

Comment 16

[Mike Hommey [:glandium]]

Then you have to set the right variables on your own. LD_LIBRARY_PATH should be enough.

Comment 17

[Phil Ringnalda (:philor)]

In a releng bug component, the severity blocker on an unassigned bug means it produces alerts all the time.

Comment 18

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Mike Hommey [:glandium] from comment #16)
> Then you have to set the right variables on your own. LD_LIBRARY_PATH should
> be enough.

Just to be on the safe side: if this is really necessary, we should think of some way to carry that variable from Gecko build system to Gaia, not hardcode it...

Comment 19

[Blake Wu [:bwu][:blakewu]]

(In reply to Phil Ringnalda (:philor) from comment #17)
> In a releng bug component, the severity blocker on an unassigned bug means
> it produces alerts all the time.
Thanks for this info and the help to change the component.

Comment 20

[Blake Wu [:bwu][:blakewu]]

Set this bug to be 2.5 blocker per comment 15.

Comment 21

[Ricky Chien [:rickychien] (inactive)]

I think LD_LIBRARY_PATH should depend on user's machine. Is it possible to set LD_LIBRARY_PATH from buildbot script, not hardcode it from gecko or gaia?

Comment 22

[Ricky Chien [:rickychien] (inactive)]

Problem of LD_LIBRARY_PATH we ran into didn't happened on local machine, it happened on treeherder only. So we need your help for setting an appropriate ENVs on build machine.

Comment 23

[Mike Hommey [:glandium]]

Import .mozconfig.mk from the gecko build?

Comment 24

[Ricky Chien [:rickychien] (inactive)]

Gregory, do you know what's the best way to setup LD_LIBRARY_PATH for b2g desktop build?

Comment 25

[Ricky Chien [:rickychien] (inactive)]

:glandium, would you mind taking this bug? thanks :)

Comment 26

[Mike Hommey [:glandium]]

I can't take this bug. I know nothing of those builds. You obviously have gtk3 pulled from tooltool, so it's there, somewhere, wherever tooltool is run from. Set LD_LIBRARY_PATH to that_path/gtk3/usr/local/lib from wherever it makes sense in the build (the TC script, the wrapper script if there is one, gaia build system...)

Comment 27

[Mike Hommey [:glandium]]

Essentially, I'm repeating comment 9.

Comment 28

[Julien Wajsberg [:julienw]]

Ricky, isn't the issue at [1] ?

[1] https://github.com/mozilla-b2g/gaia/blob/a8a462ab783a5bbab508d3c29483cff260672e3c/Makefile#L340

Maybe you can replace:

XULRUNNERSDK := LD_LIBRARY_PATH="$(dir $(XPCSHELLSDK))"

by

XULRUNNERSDK := LD_LIBRARY_PATH="$(dir $(XPCSHELLSDK)):$(LD_LIBRARY_PATH)"

(or anything that makes it work)

Comment 29

[Mike Hommey [:glandium]]

Note, you don't want a trailing : when LD_LIBRARY_PATH is empty.

Comment 30

[Ricky Chien [:rickychien] (inactive)]

If it means LD_LIBRARY_PATH has already set from buildbot or TC machine, I can export it to Gaia build system. I hope that it will work.

Thank you Julien, I'll try it later.

Comment 31

[Ricky Chien [:rickychien] (inactive)]

Try status

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e46de7e4ff61

still don't know why there are lots of test suites broke


/usr/local/lib/node_modules/taskcluster-vcs/build/bin/tc-vcs.js:57
        throw err;
              ^
Error: Error running command: git clone https://github.com/rickychien/gaia /home/worker/gaia
    at Error (<anonymous>)
    at run$ (/usr/local/lib/node_modules/taskcluster-vcs/build/run.js:122:15)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at GeneratorFunctionPrototype.invoke (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:209:22)
    at tryCatch (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:53:40)
    at Function.step (/usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/regenerator-6to5/runtime.js:103:22)
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1283:41
    at /usr/local/lib/node_modules/taskcluster-vcs/node_modules/6to5/node_modules/core-js/shim.js:1293:10
    at process._tickCallback (node.js:442:13)

It seems to relate to task-cluster modules...

Comment 32

[Ricky Chien [:rickychien] (inactive)]

Push to try again

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9bfcfde85581

Comment 33

[Gregory Szorc [:gps]]

Others seem to have this under control.

Comment 34

[Ricky Chien [:rickychien] (inactive)]

Try passed.

Comment 35

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Thanks for taking care of this!

Comment 36

[Ricky Chien [:rickychien] (inactive)]

Patch was landed in https://bugzilla.mozilla.org/show_bug.cgi?id=1146713#c86 but it was backed out for similar failures as last time. 

Here is my patch https://github.com/mozilla-b2g/gaia/commit/61385bf8a2063f256b546ea5dffb90a53c12876c

I wonder why this patch passed on try but failed on b2g-inbound, it was not supposed to happen.

Mike, do you have any idea?

Comment 37

[Ricky Chien [:rickychien] (inactive)]

Try

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3f61845707a

b2g-inbound

https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=cfb6e4519984

Comment 38

[Ricky Chien [:rickychien] (inactive)]

Pushed to try again: https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb03b7d2208d without applying XULRUNNERSDK := LD_LIBRARY_PATH="$(dir $(XPCSHELLSDK)):$(LD_LIBRARY_PATH)" and I expect it will throw gtk failures as same as on b2g-inbound.

Comment 39

[Ricky Chien [:rickychien] (inactive)]

Compare with my first patch and second patch, result of B2G Desktop Linux x64 on b2g-inbound is definitely different, see also

First patch:
https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=d255da3b284c

Second patch:
https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=cfb6e4519984

I don't see any libgtk-3.so errors throw on B2G Desktop Linux x64 in second patch, so I guess that issue is still machine related.

Comment 40

[Mike Hommey [:glandium]]

Without the command lines appearing in the logs, I can't tell anything.

Comment 41

[Ricky Chien [:rickychien] (inactive)]

I don't understand why you can't see command lines in the logs. Just click on raw log button

first patch:

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/7bCnRLKrROyf1imOn3Tilg/0/public/logs/live_backing.log

it threw this error: libgtk-3.so.0: cannot open shared object file

second patch:

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/gwJIZoMaSaSLihCrvyYZEw/0/public/logs/live_backing.log

it passed without failures

Comment 42

[Mike Hommey [:glandium]]

(In reply to Ricky Chien [:rickychien] from comment #41)
> I don't understand why you can't see command lines in the logs. Just click
> on raw log button

There is no command line running xpcshell with LD_LIBRARY_PATH in the log. There is a test -f /path/to/xpcshell, and then the error.

Comment 43

[Julien Wajsberg [:julienw]]

Yep because all xpcshell invocations are well hidden in the Makefile (using $(shell)) and as a result we don't see them in the output.

I agree that the builds on B2K Desktop look good now. Is it possible that the image builds machine actually miss GTK3 ? I don't see the same commands about "tooltool" in their logs.

See http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-emulator/1440641647/b2g_b2g-inbound_emulator_dep-bm72-build1-build129.txt.gz
(we can see this one ^ installs gtk2)

See https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/cNbzvtQnR6avb8mxqilRYA/0/public/logs/live_backing.log
(I don't see any tooltool invocaton on this one).


So I'd say _these_ build environments don't have GTK3, and that Ricky patch is OK.

Comment 44

[Mike Hommey [:glandium]]

Both logs in comment 41 have tooltool running and pulling/installing gtk3.tar.xz.

Comment 45

[Tim Guan-tin Chien [:timdream] (please needinfo)]

I checked the log with Ricky and we found the LD_LIBRARY_PATH is set on the commend-line itself when calling |./build.sh|, not exported to the shell. We would need to carry that from gaia/Android.mk and Ricky is doing exactly that right now.

Comment 46

[Tim Guan-tin Chien [:timdream] (please needinfo)]

The way B2G build system and Gaia build script are called is very confusing, I hope we could improve that *someday*.

Comment 47

[Julien Wajsberg [:julienw]]

Mike, logs in comment 41 are not the ones failing with the current patch from Ricky.
Logs in comment 41 show the effect from his patch on the same build -- that works now.
(I agree it was not the best to show the current situation)

Comment 48

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Here is what we should do per IRC conversation with Julien:

1. On taskcluster VMs like [1] we have gtk3 installed, but Gaia have failed to consume LD_LIBRARY_PATH from B2G build system. Ricky need to produce a patch to do that on bug 1146713.
2. On ICS Emulator old buildbot VMs we have only gtk2 installed via apt-get [2]. We need releng to install gtk3 on these machine too.

[1] https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/B5lGXwVXQ5OW9gatMw0LZA/0/public/logs/live_backing.log
[2] http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-emulator/1440641647/b2g_b2g-inbound_emulator_dep-bm72-build1-build129.txt.gz

Switching this bug over to releng again.

Comment 49

[Mike Hommey [:glandium]]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #48)
> Here is what we should do per IRC conversation with Julien:
> 
> 1. On taskcluster VMs like [1] we have gtk3 installed, but Gaia have failed
> to consume LD_LIBRARY_PATH from B2G build system. Ricky need to produce a
> patch to do that on bug 1146713.

If gtk3 is installed at the OS level, then LD_LIBRARY_PATH is not needed.

> 2. On ICS Emulator old buildbot VMs we have only gtk2 installed via apt-get
> [2]. We need releng to install gtk3 on these machine too.

Buildbot slaves use a Centos 6 environment to build. There are no gtk3 packages for that, and that's why we have a tooltool package in the first place.

Comment 50

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #49)
> Buildbot slaves use a Centos 6 environment to build. There are no gtk3
> packages for that, and that's why we have a tooltool package in the first
> place.

And that's why you need LD_LIBRARY_PATH to be set with the gtk3 path from the tooltool unpack.

Comment 51

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Mike Hommey [:glandium] from comment #49)
> (In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to
> queue) from comment #48)
> > Here is what we should do per IRC conversation with Julien:
> > 
> > 1. On taskcluster VMs like [1] we have gtk3 installed, but Gaia have failed
> > to consume LD_LIBRARY_PATH from B2G build system. Ricky need to produce a
> > patch to do that on bug 1146713.
> 
> If gtk3 is installed at the OS level, then LD_LIBRARY_PATH is not needed.
> 

Just to reaffirm, gtk3 is not installed at the OS level in either TC nor builtbots; it's only available from tooltool.

Comment 52

[Ricky Chien [:rickychien] (inactive)]

Attachment: Add gtk3.tar.xz in emulator

Comment 53

[Ricky Chien [:rickychien] (inactive)]

Push to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=503e0fa6c835

and I saw gtk3 appeared on B2G ICS Emulator from above link.

Comment 54

[Justin Wood (:Callek)]

glandium answered the releng needs with https://bugzilla.mozilla.org/show_bug.cgi?id=1196595#c49 and c#50 --> not a blocker

Comment 55

[Mike Hommey [:glandium]]

Comment on attachment 8653367 [details] [diff] [review]
Add gtk3.tar.xz in emulator

Review of attachment 8653367 [details] [diff] [review]:
-----------------------------------------------------------------

::: b2g/config/emulator/releng-emulator.tt
@@ +10,5 @@
> +"size": 12057960,
> +"digest": "6105d6432943141cffb40020dc5ba3a793650bdeb3af9bd5e56d3796c5f03df9962a73e521646cd71fbfb5e266c1e74716ad722fb6055589dfb7d35175bca89e",
> +"algorithm": "sha512",
> +"filename": "gtk3.tar.xz",
> +"setup": "setup.sh",

You can remove the setup line from all these. That's not relevant for the emulator builds (and currently doesn't do anything anyways).

Comment 56

[Ricky Chien [:rickychien] (inactive)]

Attachment: Bug 1196595 - Add gtk3.tar.xz in emulator

Comment 57

[Ricky Chien [:rickychien] (inactive)]

Ryan, could you help me check-in that patch? thanks

Comment 58

[Ryan VanderMeulen [:RyanVM]]

Please fix your hg config so that patches are generated with all of the necessary commit information.

Comment 59

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/3ddedf58352e

Comment 60

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3ddedf58352e