589814 - Install ANGLE libraries on test machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Benoit Jacob [:bjacob] (mostly away)]

Short version: could you consider installing OSMesa on the test machines?

In bug 582053 we are discussing a new Mochitest covering WebGL.

A requirement for that is to have some sort of OpenGL (aka 3D graphics) capability, either in hardware (GPU) or in software.

OSMesa is a software (i.e. not relying on the GPU) implementation of OpenGL. It is available on all platforms that we need to support, as far as I know.

Having OSMesa is *not* a requirement to run the WebGL Mochitest. However:

 * if both native OpenGL and OSMesa is available, the Mochitest will run its sub-tests twice, with native OpenGL and with OSMesa. This results in more extensive/reliable test coverage. Hence, having OSMesa is always desirable.

 * if some of your buildslaves don't have usable OpenGL, then OSMesa is a good alternative.


*** Where to get OSMesa from ***

Windows 32 bit:
  http://people.mozilla.com/~vladimir/misc/OSMESA32.zip

Linux package names:
  Debian: libosmesa6
  Ubuntu: libosmesa6
  Fedora: mesa-libOSMesa
  openSUSE: Mesa

Other platforms: builds/links coming soon :-)


*** How to inform Firefox of where OSMesa is installed ***

This is controlled by the preference webgl.osmesalib. For example, on linux, typically all you have to do is set webgl.osmesalib=libOSMesa.so.6.

When running the Mochitest, one can pass that pref by setting this environment variable:

  EXTRA_TEST_ARGS='--setpref=webgl.osmesa=\"libOSMesa.so.6\"'

Comment 1

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We need OSMesa installed on the test boxes pretty soon (not an option), so that we can turn on these tests.  We can provide OSMesa binaries for windows/mac/linux, because I believe we are using old versions of Fedora/CentOS that likely have a pretty old version of mesa.  Can you let us know if you'd prefer a package with OSMesa for various architectures, or if you'd rather install the packages directly on the linux boxes, can you let us know what the versions are?

Comment 2

[Aki Sasaki (not active)]

Is there someone that has run these tests all the way through with the above settings?

Comment 3

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to comment #2)
> Is there someone that has run these tests all the way through with the above
> settings?

I have.

Also, some important news from bug 582053: while trying to land this mochitest we realized that differences between native OpenGL implementations make it currently hard to have the same test succeeding and failing everywhere (of course in the long term we want all tests succeeding everywhere, but I'm talking short term). So we decided to make the mochitest not try native OpenGL for now, by default: see bug 582053 comment 55. So by default, on a machine that does not have OSMesa, the mochitest does just nothing (and does not report error). While this should allow us to land the mochitest immediately, this also means that having OSMesa becomes more important than before! Basically it is the "main" way of running this mochitest.

Comment 4

[Armen [:armenzg]]

Taking as I have been talking with Vlad and Benoit about it.

As per discussion this has higher urgency than OpenGL for Mac.
Let's try to get this as soon as possible for Windows and let's see how things work from there.

Vlad should this block beta5 or betaN+? or no need to specify anything?

Comment 5

[Benoit Jacob [:bjacob] (mostly away)]

I just filed bug 590729 with a big syncing of our copy of the tests, no unreliable timeouts anymore, and parsable console output, see the first comment on that bug.

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

Armen: I suggest that you apply the patch from that bug 590729 locally until it's landed.

Comment 7

[Benoit Jacob [:bjacob] (mostly away)]

Armen: bug 590729 has been landed. See bug 590729 comment 0 for an explanation of how to run and how to parse the console output.

Comment 8

[Armen [:armenzg]]

This is what I have done to run it manually:
* wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1282820815/firefox-4.0b5pre.en-US.win32.zip
* wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1282820815/firefox-4.0b5pre.en-US.win32.tests.zip
* unzip -o firefox-4.0b5pre.en-US.win32.zip
* unzip -o firefox-4.0b5pre.en-US.win32.tests.zip
* python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --total-chunks=5 --this-chunk=1 --chunk-by-dir=4 --test-path=content/canvas/test/webgl/test_webgl_conformance_test_suite.html --setpref=webgl.osmesalib=\"C:\libs\OSMESA32.DLL\"

All I am adding is --test-path which I believe ignores all the chunk-by-dir stuff and set --setpref.

I believe this proves that it can work manually and running through the harness.

Now I have to write the patches to modify the mochitest suites to add the setpref by default (NOTE that this will be different per platform).
Does this sound reasonable?

Thanks Benoit for landing the upstream fixes and the logging.

Comment 9

[Armen [:armenzg]]

Attachment: enable osmesa for win32

This is all I needed to make this work.
For now we are only worrying about win32.
For now we are not turning this on since there is a known crash which turns mochitests-1/5 orange. bjacob is working on it.

I now have to work on a patch to deploy OSMESA to the Windows slaves (for Win7 we will have to do it through ssh since we don't have OPSI in there).

For reference:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1282846081.1282846827.16760.gz&fulltext=1

Comment 10

[Armen [:armenzg]]

Attachment: deploy OSMESA32.dll to Win7 slaves

Ben: once we have the final dll we check it on the tools repo, right?

TODO: Document how to deploy OSMESA32.dll in the reference pages and deploy it on the reference platform (the machine is down)

Comment 11

[Armen [:armenzg]]

bug 590729 did fix one crash but it seems that there is now another one.
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1282918198.1282918970.29371.gz&fulltext=1

I am moving to the OpenGL one until this one is crash free and we have a final DLL. Lowering priority to indicate this.

I hope it makes sense. Ask me if you have any questions.

Comment 12

[Benoit Jacob [:bjacob] (mostly away)]

This crash is probably the same as bug 576620. We must fix this.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 469555 [details] [diff] [review]
enable osmesa for win32

Nit: the path looks like c:\libs\osmesa32.dll\ -- can you use single quotes around the string to make it a bit more readable?
'--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'

r=me

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 469601 [details]
deploy OSMESA32.dll to Win7 slaves

It's O.K. to pull it from here as a one-off, but please store this DLL somewhere permanent in the releng network. I suggest putting it in the CVS 'ref-platforms' repository.

Comment 15

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to comment #13)
> Comment on attachment 469555 [details] [diff] [review]
> enable osmesa for win32
> 
> Nit: the path looks like c:\libs\osmesa32.dll\ -- can you use single quotes
> around the string to make it a bit more readable?
> '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'

At least here on linux, the double quotes need to be escaped, even though all that is already inside of single quotes. Seems to be a mozilla thing, not a shell thing.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to comment #15)
> (In reply to comment #13)
> > Comment on attachment 469555 [details] [diff] [review] [details]
> > enable osmesa for win32
> > 
> > Nit: the path looks like c:\libs\osmesa32.dll\ -- can you use single quotes
> > around the string to make it a bit more readable?
> > '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
> 
> At least here on linux, the double quotes need to be escaped, even though all
> that is already inside of single quotes. Seems to be a mozilla thing, not a
> shell thing.

I'm not actually suggesting anything functionally different, just more readable. They both do the same thing in the python interpreter:
>>> print "--setpref=webgl.osmesalib=\"c:\libs\osmesa32.dll\""
--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"
>>> print '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"

Comment 17

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #13)
> > > Comment on attachment 469555 [details] [diff] [review] [details] [details]
> > > enable osmesa for win32
> > > 
> > > Nit: the path looks like c:\libs\osmesa32.dll\ -- can you use single quotes
> > > around the string to make it a bit more readable?
> > > '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
> > 
> > At least here on linux, the double quotes need to be escaped, even though all
> > that is already inside of single quotes. Seems to be a mozilla thing, not a
> > shell thing.
> 
> I'm not actually suggesting anything functionally different, just more
> readable. They both do the same thing in the python interpreter:
> >>> print "--setpref=webgl.osmesalib=\"c:\libs\osmesa32.dll\""
> --setpref=webgl.osmesalib="c:\libs\osmesa32.dll"
> >>> print '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
> --setpref=webgl.osmesalib="c:\libs\osmesa32.dll"

What I was suggesting was:

'--setpref=webgl.osmesalib=\"c:\libs\osmesa32.dll\"'

I.e. use single quotes AND escaping.

I've been running the mochitest with

EXTRA_TEST_ARGS='xxxxxx'    ... make mochitest-plain

and had to follow that convention in the xxxxxx. Otherwise my pref just wasn't set. But maybe things are just different when you run mochitests with python!

Comment 18

[[e-mail] (:bhearsum)]

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > (In reply to comment #13)
> > > > Comment on attachment 469555 [details] [diff] [review] [details] [details] [details]
> > > > enable osmesa for win32
> > > > 
> > > > Nit: the path looks like c:\libs\osmesa32.dll\ -- can you use single quotes
> > > > around the string to make it a bit more readable?
> > > > '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
> > > 
> > > At least here on linux, the double quotes need to be escaped, even though all
> > > that is already inside of single quotes. Seems to be a mozilla thing, not a
> > > shell thing.
> > 
> > I'm not actually suggesting anything functionally different, just more
> > readable. They both do the same thing in the python interpreter:
> > >>> print "--setpref=webgl.osmesalib=\"c:\libs\osmesa32.dll\""
> > --setpref=webgl.osmesalib="c:\libs\osmesa32.dll"
> > >>> print '--setpref=webgl.osmesalib="c:\libs\osmesa32.dll"'
> > --setpref=webgl.osmesalib="c:\libs\osmesa32.dll"
> 
> What I was suggesting was:
> 
> '--setpref=webgl.osmesalib=\"c:\libs\osmesa32.dll\"'
> 
> I.e. use single quotes AND escaping.
> 
> I've been running the mochitest with
> 
> EXTRA_TEST_ARGS='xxxxxx'    ... make mochitest-plain
> 
> and had to follow that convention in the xxxxxx. Otherwise my pref just wasn't
> set. But maybe things are just different when you run mochitests with python!

Ah, I see. Armen, do you know if that's necessary here? If we need to do it, let's do it :-).

Comment 19

[Armen [:armenzg]]

I will test both scenarios once I pick this bug again.

For now we are waiting for the right OSMESA32.dll to be delivered so we can deployed. From IRC conversation, the test where hiding some bugs on OSMesa.

vlad: "OSMesa seemed like it was going to work, but we had bugs in our own code that were masking these problems before"

Please poke us again once we are ready to tackle it again.

Comment 20

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Ok, let's morph this a bit.  Instead of OSMESA32, let's deploy two libraries -- libEGL.dll and libGLESv2.dll.  The tests will assume they can be found in C:\libs (same place as where osmesa was going to go.)

Comment 21

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Attachment: ANGLE libraries for deployment

The libraries for deployment.  For the record, this is angle svn r411.

Comment 22

[Armen [:armenzg]]

Attachment: deploy angle libs to win7 machines

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 471276 [details]
deploy angle libs to win7 machines

>mkdir C:\libs

Maybe "mkdir -p" here in case C:\libs already exists?

Script looks fine, although I wish we'd push harder on getting OPSI deployed to these Win7 slaves.

Comment 24

[Armen [:armenzg]]

This got deployed to all talos-r3-w7-0{01-53} slaves except the following:
* talos-r3-w7-slave040

Comment 25

[Armen [:armenzg]]

I deployed this as well into talos-r3-w7-ref.

I documented this in:
https://wiki.mozilla.org/ReferencePlatforms/Test/Win7

Comment 26

[Armen [:armenzg]]

Oops. Reopened by mistake.

Comment 27

[Armen [:armenzg]]

Filed bug 593040 for last remaining slave.

Comment 28

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Awesome, thanks Armen!