1013634 - All emulators in automation report "Failed to load libGL.so"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jonathan Griffin (:jgriffin)]

Every emulator test job in buildbot is reporting the following error when the emulator is launched:

emulator: Initializing hardware OpenGLES emulation support
Failed to load libGL.so
error libGL.so: cannot open shared object file: No such file or directory
Failed to load libGL.so
error libGL.so: cannot open shared object file: No such file or directory

This happens with both successful and failing jobs.  It's nice that this isn't causing us to fail, but it may be causing us to run more slowly than we would otherwise, and could be the source of other problems.

The emulator expects this file to live at /usr/lib/x86_64-linux-gnu/libGL.so (for 64-bit linux) and should be symlinked from wherever it is to that location.

Comment 1

[Andrew Halberstadt [:ahal]]

Good catch.

For reference:
> sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so
(s/i386/x86_64)

From https://developer.mozilla.org/en-US/Firefox_OS/Firefox_OS_build_prerequisites#Emulator_build_issues

Comment 2

[Ting-Yu Chou [:ting] (away)]

An alternative is to have libgl1-mesa-dev installed, but I am not sure how to do this?

Comment 3

[Ting-Yu Chou [:ting] (away)]

Andrew, do you know how should I fix this for test machine, is this about mozharness?

Comment 4

[Ting-Yu Chou [:ting] (away)]

The code seems in mozharness [1], I tried to import mozharness to my mecurial repository, but it doesn't show up. Tried also Bitbucket, but the build was failed with HTTP 404 [2], probably the build server is not allow to access external resource.

[1] http://dxr.mozilla.org/build:mozharness/source/scripts/android_emulator_build.py#309
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5a6d75f91d2

Comment 5

[Rail Aliiev [:rail]]

Try tries (sic!) to "probe" https://bitbucket.org/janus926/mozharness/rev/bug-1013634 per http://hg.mozilla.org/build/tools/file/0d8d4f943f6e/buildfarm/utils/repository_manifest.py#l148, fails to retrieve it and doesn't try to clone the repo after that. If you use your user repo on hg.m.o it should work as expected.

Comment 6

[Ting-Yu Chou [:ting] (away)]

Thanks for the information! I've already filed bug 1115600 for the issue of cloning mozharness to my user repo.

Comment 7

[Ting-Yu Chou [:ting] (away)]

The code I pointed out at comment 4 is not the right one, the same error message still shows up.

Comment 8

[Ting-Yu Chou [:ting] (away)]

Attachment: patch v1

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2d3572748f6

Comment 9

[Andrew Halberstadt [:ahal]]

This would work, but seems like it would be better if the machines just came pre-configured with this. Rail, how difficult would it be to get them set up with this symlink?

> sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

Comment 10

[Rail Aliiev [:rail]]

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> This would work, but seems like it would be better if the machines just came
> pre-configured with this. Rail, how difficult would it be to get them set up
> with this symlink?
> 
> > sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

Definitely it would be better than adding hacks to the code. libgl1-mesa-dev has the file in the package, installing it should fix the problem. One of the downsides though will be some other extra packages installed (all of them are *-dev, so should not interfere with other libraries).

I'll create a puppet patch for this.

Comment 11

[Rail Aliiev [:rail]]

Attachment: [puppet] install libgl1-mesa-dev

Comment 12

[Rail Aliiev [:rail]]

Grabbing the bug to track the deployment.

Comment 13

[Rail Aliiev [:rail]]

I'm going to deploy this ... not today. :) In the worst case scenario on Mon/Tue.

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8542970 [details] [diff] [review]
[puppet] install libgl1-mesa-dev

remote:   https://hg.mozilla.org/build/puppet/rev/e9d1afd8c374
remote:   https://hg.mozilla.org/build/puppet/rev/2cf318e35f3c

This will show up in production tomorrow.

Comment 15

[Rail Aliiev [:rail]]

:ting, can you verify that the problem is fixed without applying your patch (attachment 8542490 [details] [diff] [review])?

Comment 16

[Andrew Halberstadt [:ahal]]

Comment on attachment 8542490 [details] [diff] [review]
patch v1

Thanks Rail!

Comment 17

[Andrew Halberstadt [:ahal]]

I'm still seeing the libGL error in recent inbound qemu.logs:
http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/fd46662f417e689374c9724b5f47399cf60e44268109bf72eb43ee8e815b7094cb4f96c599f6b350f0c467a4f231e7671d891bfdf2dde7b3696fcaceae5debfe

Comment 18

[Rail Aliiev [:rail]]

Hmm, the patch symlinks 32-bit libraries, probably we need to install libgl1-mesa-dev:i386. I'll poke this next week. Meanwhile I'll backout the puppet patch.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8542970 [details] [diff] [review]
[puppet] install libgl1-mesa-dev

remote:   https://hg.mozilla.org/build/puppet/rev/f6e23a227371
remote:   https://hg.mozilla.org/build/puppet/rev/37806d3f195a

Comment 20

[Ting-Yu Chou [:ting] (away)]

It'd be nice if we can fix this by the end of this week.

Comment 21

[Rail Aliiev [:rail]]

Attachment: install-mesa-32bit.diff

this is what's expected to be installed:

$ apt-get install libgl1-mesa-dev:i386
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libdrm-dev:i386 libkms1:i386 libpthread-stubs0:i386 libpthread-stubs0-dev:i386 libx11-dev:i386 libx11-doc libxau-dev:i386 libxcb1-dev:i386 libxdmcp-dev:i386 libxext-dev:i386
  mesa-common-dev:i386 x11proto-core-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
  libxcb-doc:i386
Recommended packages:
  libx11-doc:i386
The following NEW packages will be installed:
  libdrm-dev:i386 libgl1-mesa-dev:i386 libkms1:i386 libpthread-stubs0:i386 libpthread-stubs0-dev:i386 libx11-dev:i386 libx11-doc libxau-dev:i386 libxcb1-dev:i386
  libxdmcp-dev:i386 libxext-dev:i386 mesa-common-dev:i386 x11proto-core-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev xorg-sgml-doctools xtrans-dev
0 upgraded, 18 newly installed, 0 to remove and 121 not upgraded.
Need to get 4893 kB of archives.
After this operation, 19.1 MB of additional disk space will be used.

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8546392 [details] [diff] [review]
install-mesa-32bit.diff

remote:   https://hg.mozilla.org/build/puppet/rev/d7eaf090016f
remote:   https://hg.mozilla.org/build/puppet/rev/561fab128519

We'll see the results tomorrow.

Sorry for the delay, upcoming releases and merge day keep me busy.

Comment 23

[Rail Aliiev [:rail]]

This should be fixed now. \o/

Comment 24

[Ting-Yu Chou [:ting] (away)]

Thank you Rail!