692989 - QA cross-process layers

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

The configurations that are important to test are

 1. Windows 7 machine with a GPU/driver that supports Direct3D10/Direct2D.  You can check this in a nightly build by going to about:support and looking at the "Graphics" section.

 2. Windows XP machine with a GPU/driver that supports Direct3D9.  Again, about:support will tell you this.

 3. OS X (10.6? 10.7? both?) machine that supports OpenGL layers.

 4. Linux-x11 machine that supports OpenGL layers.  Latest Ubuntu release + NVIDIA GPU and proprietary NVIDIA drivers is usually the most reliable combination, sadly.

In that order of importance.

(It would be useful at some point to check a machine without GPU layers, but the benchmarks below are so slow without GPU support that it doesn't really matter.  Plus the non-GPU path is what we've been optimizing in fennec for the last year.)

To enable out-of-process content in desktop Firefox, you have to build Firefox with the option --enable-e10s-compat, and then in about:config you have to set the pref "browser.tabs.remote" to true and restart.  To return to in-process content, just set that pref back to false and restart again.

The first thing we'll want to QA is checking large, complicated sites for functional bugs, like gmail.  We have a fairly comprehensive set of automated unit tests, but running through the litmus tests that we have, probably in addition to the talos5 pageset, would be a great start.

KNOWN BUGS:
 - pages that use plugins are highly unlikely to work properly.  That's being tracked separately.
 - pages that use <select> elements probably won't work correctly.  That's also being tracked separately.

The second thing we'll need to manually QA is benchmark performance.  When eideticker[1] is ready, we can test performance automatically, but until then they need to be run manually.  Here's a list of the benchmarks I know of that are relatively interesting

 o JSGameBench
 o GUIMark2 HTML5
 o Asteroids HTML5 Canvas 2D
 o Psychedelic Browsing
 o Hardware Acceleration Stress Test
 o FishIE
 o WebGL FishIE

gfx guys, am I leaving out any interesting tests?

Each of these tests reports some kind of composite score or fps.  We want to compare the scores with in-process vs. out-of-process content.  See above for how to flip that pref.

KNOWN BUGS:
 - on windows XP (Direct3D9), we expect a 3-4x performance regression on WebGL benchmarks.  There's nothing we can do about that.

Virgil/Ioana, Let me know if you have questions.  If it helps we can move this content to its own wiki pages.  Feel free to file depedent bugs on this one to track work items as you see fit, or we can discuss further how to break up the work.

[1] https://wiki.mozilla.org/Project_Eideticker

Comment 1

[Ioana (away)]

Chris,

Is there a Litmus test suite created especially for this 
or should I run older test suites that somehow incorporate it (e.g. Web App Compatibility tests)?

Do you want me to run the Litmus tests on all the configurations you specified 
or is it enough if I do exploratory testing on all of them?

Should I also run the tests on these configurations with both out-of-process content and in-process content 
or is that just for the performance tests?

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Ioana Budnar [QA] from comment #1)
> Chris,
> 
> Is there a Litmus test suite created especially for this 
> or should I run older test suites that somehow incorporate it (e.g. Web App
> Compatibility tests)?
> 

Not to my knowledge, but I know almost nothing about Litmus.  Running older suites is a great start.  If there's not already a suite that includes the benchmarks in comment 0, I think we should add one.  I took that list partially from the Tom's Hardware browser comparisons.  (They're not all good benchmarks, but they're what people are using.)

> Do you want me to run the Litmus tests on all the configurations you
> specified 
> or is it enough if I do exploratory testing on all of them?
> 

For now, exploratory testing is what we need.  We want to understand where the big correctness and performance bugs are.

> Should I also run the tests on these configurations with both out-of-process
> content and in-process content 
> or is that just for the performance tests?

That's just needed for the performance tests.

Comment 3

[Ioana (away)]

Chris,
 Because of more higher priority work on beta 3, I only got to do some exploratory testing on a Windows 7 machine with a GPU/driver that supports Direct3D10. You can find the tests here: http://bit.ly/pz3TkC.

There are no functionality issues on complex sites, but they seem to work very slow sometimes so there probably are some performance issues.

Before going further with the testing I need some information from you: in comment #0 you were talking about nightly so I tested on a nightly build. Is this ok? Should I run the tests on an electrolysis build or on any other builds?

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Ioana Budnar [QA] from comment #3)
> Chris,
>  Because of more higher priority work on beta 3, I only got to do some
> exploratory testing on a Windows 7 machine with a GPU/driver that supports
> Direct3D10. You can find the tests here: http://bit.ly/pz3TkC.
> 

That document is private.

> Before going further with the testing I need some information from you: in
> comment #0 you were talking about nightly so I tested on a nightly build. Is
> this ok? Should I run the tests on an electrolysis build or on any other
> builds?

I don't think you can enable out-of-process content in mozilla-central nightly builds.  Pretty sure you need a custom build.

Felipe, is this documented somewhere?  Are there e10s nightly builds that Ioana can use?

Comment 5

[Virgil Dicu [:virgil] [QA]]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #4)
> (In reply to Ioana Budnar [QA] from comment #3)
> > Chris,
> >  Because of more higher priority work on beta 3, I only got to do some
> > exploratory testing on a Windows 7 machine with a GPU/driver that supports
> > Direct3D10. You can find the tests here: http://bit.ly/pz3TkC.
> > 
> 
> That document is private.

The document should be viewable now.

I will continue with what Ioana started to do. I have logged in Bug 694991 when attempting to go through the test cases from the spreadsheet, but not sure as how it regards this feature or not.

Comment 6

[Virgil Dicu [:virgil] [QA]]

I've created a test plan based on the informations provided here: https://wiki.mozilla.org/Platform/Features/ElectrolysisTextureSharing/TestPlan#Electrolysis_Texture_Sharing

Chris, could you take a look and see if I missed anything or something might be inaccurate?

Also, could you possibly provide a Try Server build with --enable-e10s-compat? That would save us a lot of time in the process.

Comment 7

[:Felipe Gomes (needinfo for replies!)]

(In reply to Virgil Dicu [QA] from comment #6)
> Also, could you possibly provide a Try Server build with
> --enable-e10s-compat? That would save us a lot of time in the process.

I can create the tryservers build for you. I will post the link here as soon as the builds are ready

 
> I don't think you can enable out-of-process content in mozilla-central
> nightly builds.  Pretty sure you need a custom build.
> 
> Felipe, is this documented somewhere?  Are there e10s nightly builds that
> Ioana can use?

We don't have nightly builds using the flag because the goal has been to remove the flag as soon as possible, so custom builds are needed. I wrote an overview on how to use it here: http://felipe.wordpress.com/2011/07/20/initial-electrolysis-patches-for-desktop-firefox-now-in-mozilla-central/

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Virgil Dicu [QA] from comment #6)
> I've created a test plan based on the informations provided here:
> https://wiki.mozilla.org/Platform/Features/ElectrolysisTextureSharing/
> TestPlan#Electrolysis_Texture_Sharing
> 
> Chris, could you take a look and see if I missed anything or something might
> be inaccurate?
> 

This looks good.  The one missing bit is Litmus: do you know which parts of the plan are already in Litmus suites?  If not, which team typically creates Litmus suites?  It would be easier to have QA instructions that are "run Litmus suites X, Y" instead of "browse these 20 pages then run these 10 benchmarks" :).

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Try builds here: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-1e5a8c2cadba

Please be aware that the first tab in each window is *not* a remote tab (yet). All other tabs open on any window are remote.

Comment 10

[Virgil Dicu [:virgil] [QA]]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #8)
> This looks good.  The one missing bit is Litmus: do you know which parts of
> the plan are already in Litmus suites?  If not, which team typically creates
> Litmus suites?  It would be easier to have QA instructions that are "run
> Litmus suites X, Y" instead of "browse these 20 pages then run these 10
> benchmarks" :).

Yes, true, I'll copy the Litmus suite there. I was thinking about the Web App Compatibility test suite. This test suite is ran weekly on every beta, but never on nightly. The test cases contain navigation and functionality verification of yahoo, gmail, top 10 alexa sites and other general popular sites. This is the link to the test cases there (one or two in the list might be disabled): http://bit.ly/o2veoO

Layout BFTs test suite can also be used:http://bit.ly/qqMx9A

Litmus doesn't have a Nightly branch, so the tests there can only be ran on Aurora and older versions of Firefox. 
I can create separate Litmus suites, but as there is no nightly branch there, I will have to wait until the feature reaches Aurora to run them there. Then, I can create a new category, write new created test cases from the spreadsheet into that new category.
So for the moment I could manually copy those tests upon which we agree from Litmus into our spreadsheet, along with newly created test cases and use the spreadsheet for results and bug links.

How does that sound? I'll add some links to Litmus soon and update the test plan.


(In reply to Felipe Gomes (:felipe) from comment #9)
> Try builds here:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-
> 1e5a8c2cadba
> 
> Please be aware that the first tab in each window is *not* a remote tab
> (yet). All other tabs open on any window are remote.

Thanks!

Comment 11

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Virgil Dicu [QA] from comment #10)
> (In reply to Chris Jones [:cjones] [:warhammer] from comment #8)
> > This looks good.  The one missing bit is Litmus: do you know which parts of
> > the plan are already in Litmus suites?  If not, which team typically creates
> > Litmus suites?  It would be easier to have QA instructions that are "run
> > Litmus suites X, Y" instead of "browse these 20 pages then run these 10
> > benchmarks" :).
> 
> Yes, true, I'll copy the Litmus suite there. I was thinking about the Web
> App Compatibility test suite. This test suite is ran weekly on every beta,
> but never on nightly. The test cases contain navigation and functionality
> verification of yahoo, gmail, top 10 alexa sites and other general popular
> sites. This is the link to the test cases there (one or two in the list
> might be disabled): http://bit.ly/o2veoO
> 
> Layout BFTs test suite can also be used:http://bit.ly/qqMx9A
> 

Sounds good.

> Litmus doesn't have a Nightly branch, so the tests there can only be ran on
> Aurora and older versions of Firefox.

Hmm, being able to run Litmus tests on nightlies seems very useful ... is this something we can fix?  Who should we talk to?  This could be quite useful for platform and frontend developers, too.
 
> So for the moment I could manually copy those tests upon which we agree from
> Litmus into our spreadsheet, along with newly created test cases and use the
> spreadsheet for results and bug links.
> 
> How does that sound? I'll add some links to Litmus soon and update the test
> plan.
> 

This sounds OK for now.  I'd like to help change things so that we run the tests on nightlies, but until then this plan sounds good.

Comment 12

[Virgil Dicu [:virgil] [QA]]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #11)
> Hmm, being able to run Litmus tests on nightlies seems very useful ... is
> this something we can fix?  Who should we talk to?  This could be quite
> useful for platform and frontend developers, too.

Adding Anthony, he might now of a way to solve this issue (is a new Nightly branch possible? or a specific Electrolysis branch for the project?)

Comment 13

[u279076]

Our Nightly branch in Litmus is Aurora. You can add your tests there.

On a side note, I'm curious why this sort of discussion is taking place in Bugzilla. I'm asking because we traditionally coordinate feature and test strategy planning via email, etherpad, and wikis. It seems oddly out of place in Bugzilla.

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

Only a tiny fraction of Mozilla development discussion happens on mailing lists; a lot more happens in Bugzilla. It's a lot easier to keep track of items in Bugzilla. Just sayin' --- I don't have any opinion specifically about this discussion.

Comment 15

[u279076]

Granted, this does not appear to be a development discussion though. It appears to be a test strategy discussion. I'm not against it and will not mandate against using bugzilla for this purpose. It is just extremely unorthodox and exists as a "closed" discussion to those on the cc list of this bug.

Comment 16

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #13)
> Our Nightly branch in Litmus is Aurora. You can add your tests there.
> 

Can you point us at where the Litmus code lives?  I don't quite understand --- will the tests added there run against a build of Firefox from mozilla-central?  That's all we need for this work.

Comment 17

[u279076]

Litmus is not "code", it is plain-text steps to reproduce and expected results (see http://litmus.mozilla.org). If you want to point the community to a group of tests to run I would suggest creating a testgroup in the Aurora branch called Electrolysis and a subgroup called "Cross Process Layers". Tests can be added to this subgroup and testers can be pointed to it.

It's hard to explain without a proof of concept set up. I suggest Virgil to get something set up with a couple tests as an example.

Comment 18

[Virgil Dicu [:virgil] [QA]]

I've ran the benchmark tests from comment 0 on the try server build provided by Felipe. You can find the results in the same spreadsheet with the exploratory on a separate sheet: http://bit.ly/sEUinx   

The configurations used are listed there. For some types of benchmarks different tests were needed (Vector, text and bitmap for guimark2, for example). I listed them all there.  

While loading the psychedelic benchmark, Firefox crashed on all platforms except Ubuntu 11.10 (the page didn't load though there). You can find these stated in the spreadsheet. I'll file a bug for this tomorrow, if you agree.


(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #17)
> Litmus is not "code", it is plain-text steps to reproduce and expected
> results (see http://litmus.mozilla.org). If you want to point the community
> to a group of tests to run I would suggest creating a testgroup in the
> Aurora branch called Electrolysis and a subgroup called "Cross Process
> Layers". Tests can be added to this subgroup and testers can be pointed to
> it.
> 
> It's hard to explain without a proof of concept set up. I suggest Virgil to
> get something set up with a couple tests as an example.

Yes, Litmus is used for manual testing throughout the community. Once tests for electrolysis are added there(this will be the next step) community members can test this issue accordingly. 
Chris, were you thinking about running the tests from litmus on a build with electerolysis enabled (as it is now the case) or would you expect them to be ran there once this reaches the official nightly?

Comment 19

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Virgil Dicu [QA] from comment #18)
> While loading the psychedelic benchmark, Firefox crashed on all platforms
> except Ubuntu 11.10 (the page didn't load though there). You can find these
> stated in the spreadsheet. I'll file a bug for this tomorrow, if you agree.
> 

Yes please! :)

> Chris, were you thinking about running the tests from litmus on a build with
> electerolysis enabled (as it is now the case) or would you expect them to be
> ran there once this reaches the official nightly?

Both.  We'll want to use these tests to make sure e10s builds are up to parity with m-c before e10s is enabled by default, and then check for regressions after enabled.

Comment 20

[Will Setc]

[testday-20111028]
open about:support caused nightly to crash
 Crash is understandable since this machine doesn't meet minimal specs but submitted a crash report anyway.
id:bp-a2e39839-9b2e-4960-a4ff-87c792111028
10/29/201112:51 AM
Signature: i915_dri.so@0x256b7

Will try again on both XP and linux / nvidia GeForce 6600 gpu / nvidia driver on linux

Comment 21

[Virgil Dicu [:virgil] [QA]]

Is this part of the Electrolysis project on hold? As seen here (https://bugzilla.mozilla.org/show_bug.cgi?id=697762#c1) some parts got suspended.

I'm asking because there hasn't been much action here lately and just wanted to now from were to start. I'll follow up with some exploratory on complex pages if things are on schedule.

Comment 22

[Gregory Pappas [:gregp]]

Inactive metabug.