Install emulator-with-codecs onto linux32 test machines

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
4 months ago

People

(Reporter: joduinn, Unassigned)

Tracking

Details

(URL)

Attachments

(2 attachments, 4 obsolete attachments)

Once jgriffin has verified exact toolchain in bug#789650, RelEng needs to install this "emulator-with-codecs" onto our linux32 test machines. This is a prerequesite to having tbpl show percheckin "B2G builds being tested within the emulator-with-codecs". 

The current location of the emulator to be deployed is:
https://releases.mozilla.com/b2g/2012-09-07/emulator-arm_linux_2012-09-07.zip
...but per email, this may be revised after testing, and there may also be additional libraries needed by the emulator, so do nothing until bug#789650 has worked out all the exact details.


Per meeting today w/jgriffin, cjones:
* this emulator will need to be updated infrequently (at most monthly?), so the process to deploy this emulator should be similar to the process we use for updating other toolchain components like talos, etc.
* it is possible to run this emulator on other OS (like OSX, win32) but as linux32 is most stable, lets just install this on linux32 slaves.
If there is *any* risk in updating the emulator for "older" trains, say we update it for a trunk change and it can hurt in *any* way the release channel builds, I'm in favor of doing the emulator via tooltool, if it is possible to do so.
Can you also document how to build the emulator?
Also, please ensure that the emulator can be downloaded/unpacked at runtime.
I've been able to get an emulator running successfully on a linx32 test slave, which I temporarily have access to courtesy of bug 789650.  I only had to install one package to get it to work:  mesa-libGL-devel (this in turns pulls in a few other packages).

The emulator build I used:  https://releases.mozilla.com/b2g/latest/emulator-arm_linux_2012-09-11.zip

I think you're OK to grab this emulator and install on linux32 slaves, along with the above-mentioned package.
OS: Mac OS X → Linux
Hardware: x86 → All
================================================================================
 Package                   Arch        Version                Repository   Size
================================================================================
Installing:
 mesa-libGL-devel          i686        7.6-0.13.fc12          fedora      459 k
Installing for dependencies:
 kernel-headers            i686        2.6.31.5-127.fc12      fedora      739 k
 libX11-devel              i686        1.3-1.fc12             fedora      1.0 M
 libXau-devel              i686        1.0.5-1.fc12           fedora       13 k
 libXdamage-devel          i686        1.1.2-1.fc12           fedora      8.7 k
 libXdmcp-devel            i686        1.0.3-1.fc12           fedora      9.6 k
 libXext-devel             i686        1.1-1.fc12             fedora       77 k
 libXfixes-devel           i686        4.0.4-1.fc12           fedora       11 k
 libXxf86vm-devel          i686        1.1.0-1.fc12           fedora       17 k
 libdrm-devel              i686        2.4.15-4.fc12          fedora       70 k
 libxcb-devel              i686        1.4-1.fc12             fedora      118 k
 xorg-x11-proto-devel      noarch      7.4-34.fc12            fedora      249 k

Transaction Summary
================================================================================
Install      12 Package(s)
Upgrade       0 Package(s)

Total download size: 2.8 M
Is this ok [y/N]: n
Exiting on user Command
Complete!
Created attachment 665664 [details] [diff] [review]
first stab

Am I on the right track here?
Do I need to list all the dependent rpms, or will puppet find them?
Will this affect fedora64 too?
Attachment #665664 - Flags: feedback?(rail)
Comment on attachment 665664 [details] [diff] [review]
first stab

(In reply to Aki Sasaki [:aki] from comment #6)
> Created attachment 665664 [details] [diff] [review]
> first stab
> 
> Am I on the right track here?
> Do I need to list all the dependent rpms, or will puppet find them?

No. RPM provider doesn't install dependencies. :/ yum provider would.

> Will this affect fedora64 too?

Yeah. If you want to install the packages for i686 only, you need to wrap it with something like
case ${hardwaremodel} : {
    'i686': {
      package {....}
    }
}
Attachment #665664 - Flags: feedback?(rail) → feedback-
Created attachment 666651 [details] [diff] [review]
second stab

If this looks good, I'm going to start asking about how to stage/test this.
Attachment #665664 - Attachment is obsolete: true
Attachment #666651 - Flags: feedback?(rail)
Attachment #666651 - Flags: feedback?(rail) → feedback+
testing on talos-r3-fed-010:

err: Could not retrieve catalog: Could not parse for environment production: Could not match '${hardwaremodel}' at /etc/puppet/manifests/os/talos_fedora.pp:26
warning: Not using cache on failed catalog

# removed case statement, s,RPMS,RPMs,g

err: //Node[talos-r3-fed-010]/talosslave/talos_fedora/Package[libXfixes-devel]/ensure: change from absent to libXfixes-devel-4.0.4-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://staging-puppet.build.mozilla.org/staging/fedora12-i686/test/RPMs/libXfixes-devel-4.0.4-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	pkgconfig(x11) is needed by libXfixes-devel-4.0.4-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:76
err: //Node[talos-r3-fed-010]/talosslave/talos_fedora/Package[libXxf86vm-devel]/ensure: change from absent to libXxf86vm-devel-1.1.0-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://staging-puppet.build.mozilla.org/staging/fedora12-i686/test/RPMs/libXxf86vm-devel-1.1.0-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	pkgconfig(x11) is needed by libXxf86vm-devel-1.1.0-1.fc12.i686
	pkgconfig(xext) is needed by libXxf86vm-devel-1.1.0-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:76
Created attachment 666753 [details] [diff] [review]
wip 3

[root@talos-r3-fed-010 ~]# puppetd --test --server staging-puppet.build.mozilla.org  --logdest console
err: Could not retrieve catalog: Default already defined for Package { provider }; cannot redefine at /etc/puppet/manifests/os/talos_fedora.pp.orig:4 on node talos-r3-fed-010.build.mozilla.org
warning: Not using cache on failed catalog

bleah
Attachment #666651 - Attachment is obsolete: true
Created attachment 666758 [details] [diff] [review]
works!

This works on i686.
Now I need to figure out how to limit to 32bit only, or I need to download the 64 bit rpms as well.
Attachment #666753 - Attachment is obsolete: true
Created attachment 666769 [details] [diff] [review]
works, and is limited to 32 bit
Attachment #666758 - Attachment is obsolete: true
Attachment #666769 - Flags: review?(rail)
Attachment #666769 - Flags: review?(rail) → review+
ugh, talos-r3-fed-007

err: //Node[talos-r3-fed-007]/talosslave/talos_fedora/Package[libxcb-devel]/ensure: change from absent to 1.4-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://scl-production-puppet.build.scl1.mozilla.com/production/fedora12-i686/test/RPMs/libxcb-devel-1.4-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	libxcb = 1.4-1.fc12 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-composite.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-damage.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-dpms.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-glx.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-randr.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-record.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-render.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-res.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-screensaver.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-shape.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-shm.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-sync.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xevie.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xf86dri.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xfixes.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xinerama.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xselinux.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xtest.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xv.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb-xvmc.so.0 is needed by libxcb-devel-1.4-1.fc12.i686
	libxcb.so.1 is needed by libxcb-devel-1.4-1.fc12.i686
	pkgconfig(xau) >= 0.99.2 is needed by libxcb-devel-1.4-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:87
info: //Node[talos-r3-fed-007]/talosslave/talos_fedora/buildslave::cleanup/Tidy[/home/cltbld/.mozilla/firefox/console.log]/ensure: Tidy target does not exist; ignoring
err: //Node[talos-r3-fed-007]/talosslave/talos_fedora/Package[libX11-devel]/ensure: change from absent to 1.3-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://scl-production-puppet.build.scl1.mozilla.com/production/fedora12-i686/test/RPMs/libX11-devel-1.3-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	libX11 = 1.3-1.fc12 is needed by libX11-devel-1.3-1.fc12.i686
	libX11-xcb.so.1 is needed by libX11-devel-1.3-1.fc12.i686
	libX11.so.6 is needed by libX11-devel-1.3-1.fc12.i686
	libXau-devel is needed by libX11-devel-1.3-1.fc12.i686
	libXdmcp-devel is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(kbproto) is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xcb) is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xcb) >= 1.1.92 is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xproto) is needed by libX11-devel-1.3-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:87
talos-r3-fed-012 looks better:

err: //Node[talos-r3-fed-012]/talosslave/talos_fedora/Package[libX11-devel]/ensure: change from absent to 1.3-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://scl-production-puppet.build.scl1.mozilla.com/production/fedora12-i686/test/RPMs/libX11-devel-1.3-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	libXau-devel is needed by libX11-devel-1.3-1.fc12.i686
	libXdmcp-devel is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(kbproto) is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xcb) is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xcb) >= 1.1.92 is needed by libX11-devel-1.3-1.fc12.i686
	pkgconfig(xproto) is needed by libX11-devel-1.3-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:87

err: //Node[talos-r3-fed-012]/talosslave/talos_fedora/Package[libxcb-devel]/ensure: change from absent to 1.4-1.fc12 failed: Could not update: Execution of '/bin/rpm -i --oldpackage http://scl-production-puppet.build.scl1.mozilla.com/production/fedora12-i686/test/RPMs/libxcb-devel-1.4-1.fc12.i686.rpm' returned 1: error: Failed dependencies:
	pkgconfig(xau) >= 0.99.2 is needed by libxcb-devel-1.4-1.fc12.i686
 at /etc/puppet/manifests/os/talos_fedora.pp:87
stupid rpm provider... I wish we had a yum repo mirror and used yum provider. :/
[10:54]	<aki>	wondering if it's difficult to build a statically linked emulator
[10:54]	<aki>	hitting large dependency ugliness trying to install the dependencies on the test farm https://bugzilla.mozilla.org/show_bug.cgi?id=789651
[10:56]	<mwu>	aki: probably difficult, but you also don't need the package suggested there just to run the emulator
[10:56]	<aki>	oh?
[10:56]	<mwu>	you just need mesa-libGL for x86
[10:57]	<mwu>	the devel package is only necessary for building the emulator

back to square one, but with a most-likely easier-to-install rpm.
Comment on attachment 666769 [details] [diff] [review]
works, and is limited to 32 bit

[root@talos-r3-fed-008 talos-slave]# yum install mesa-libGL
Loaded plugins: presto, refresh-packagekit
Setting up Install Process
Package mesa-libGL-7.6-0.13.fc12.i686 already installed and latest version
Nothing to do

Backed out: http://hg.mozilla.org/build/puppet-manifests/rev/edb6f2d051cc
Attachment #666769 - Flags: checked-in+ → checked-in-
We're going to be installing the emulator per script run in bug 774535.
It looks like we already have mesa-libGL installed on our fedora pool.
I think that's all that needs to be done here; resolving.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
I'm slightly skeptical of this.  The emulator needs libGL.so.  

sudo yum provides "*/libGL.so"
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN Satellite or RHN Classic support will be disabled.
jenkins/filelists                                                                                | 5.4 kB     00:00

mozilla/filelists                                                                                | 785 kB     00:00

puppetlabs/filelists                                                                             | 618 kB     00:00

rpmforge/filelists                                                                               | 1.6 MB     00:00

rpmforge-extras/filelists                                                                        | 303 kB     00:00

mesa-libGL-devel-7.11-3.el6.x86_64 : Mesa libGL development package
Repo        : installed
Matched from:
Filename    : /usr/lib64/libGL.so



mesa-libGL-devel-7.11-3.el6.i686 : Mesa libGL development package
Repo        : installed
Matched from:
Filename    : /usr/lib/libGL.so


On the linux32 test slave I checked out and tested the emulator with, the emulator would not run because of missing libGL.so.
[cltbld@talos-r3-fed-020 ~]$ ls /usr/lib/libGL*
/usr/lib/libGL.so.1    /usr/lib/libGLU.so.1
/usr/lib/libGL.so.1.2  /usr/lib/libGLU.so.1.3.070700

is libGL.so.1 ok?
(In reply to Aki Sasaki [:aki] from comment #21)
> [cltbld@talos-r3-fed-020 ~]$ ls /usr/lib/libGL*
> /usr/lib/libGL.so.1    /usr/lib/libGLU.so.1
> /usr/lib/libGL.so.1.2  /usr/lib/libGLU.so.1.3.070700
> 
> is libGL.so.1 ok?

I don't know; maybe if it was symlinked to libGL.so?  As I said, the emulator wouldn't run on a linux32 test slave in its default configuration.
I'd be happy to check out another test slave to investigate.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 667115 [details] [diff] [review]
softlink libGL.so -> libGL.so.1.2

tested on staging-puppet.
Attachment #667115 - Flags: review?(rail)
Comment on attachment 667115 [details] [diff] [review]
softlink libGL.so -> libGL.so.1.2

Please remove alias => "create_libGL_symlink" if it's not used anywhere.
Attachment #667115 - Flags: review?(rail) → review+
Deployed.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED

Updated

6 years ago
Depends on: 797425
Great success. I verified that after symlinking libGL.so.1.2. to libGL.so, the emulator works without the installation of any additional libs.  I also found that in order to run, I had to export DISPLAY:=0, which we may need to do in the mozharness scripts, unless buildbot sets that up automatically.
In a desktop build, I see this in the log:

========= Started set props: basedir (results: 0, elapsed: 35 secs) (at 2012-10-03 11:20:29.022128) =========
bash -c pwd
 in dir /home/cltbld/talos-slave/test/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'pwd']
 environment:
  CVS_RSH=ssh
  DISPLAY=:0.0
...

so I'm guessing DISPLAY=:0.0 will be set automatically and we don't have to explicitly set it in our mozharness scripts.
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

4 months ago
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.