941809 - AreWeSlimYet for B2G

Status: RESOLVED DUPLICATE of bug 899194

(Core :: General, defect)

Description

[Andrew McCreight [:mccr8]]

We need some kind of automated regression testing for B2G.

Comment 1

[Nicolas B. Pierron [:nbp]]

I already have some automation where I can start webpages within the browser app and collect results from elements of the page.  After I can upload them where we want.

Just tell me what page (that I can host my-self) to run and where to upload the result and I will setup the automation for you ;)

Comment 2

[Nicholas Nethercote [inactive]]

johns and kats are the experts here.

Comment 3

[Robert Wood [:rwood]]

Hey guys I am already working on this, see Bug 899194

Comment 5

[Andrew McCreight [:mccr8]]

Thanks, I figured we must have a bug for this already, I just couldn't remember what it was.

Comment 6

[Nicolas B. Pierron [:nbp]]

(In reply to Robert J. Wood [:rwood] from comment #3)
> Hey guys I am already working on this, see Bug 899194

I am sorry, but talos tp5 is not a good representative set of what Gaia apps are made of.
The problem raised here is not that we want exactly the same thing as AreWeSlimYet, but we want something which is capable of warning us when worker's size is increasing.  I don't think any of the tp5 has any worker.

Even if we can reuse the infrastructure of either AreWeSlimYet or AreWeFastYet, we at least need to find a benchmark which is focus on highlighting the features of the blocked bug / Gaia.

Comment 7

[Robert Wood [:rwood]]

Hey all, sorry I will add more context to what I am currently working on for AWSY & Firefox OS.

No, we are not using the TP5 benchmarks. Instead of just testing the Firefox OS browser, we want to test all the gaia apps in order to ensure the main b2g process (at minimum) doesn't have a memory regression.

The test itself is an orangutan-based test that runs on the b2g emulator. The test drives the gaia-ui and simply starts each gaia app by tapping the app icon, leaves the app open for 10 seconds, and then presses the home button to minimize the app. Then it is repeated for each gaia app. The test will repeat for 6 hours. About_memory dumps will be taken at intervals and provided to AWSY.

The test will run on a per-CI basis using TBPL mozilla-central builds. We are setting up 4 amazon VMs to drive the automation via jenkins.

Regards,

Rob

Comment 8

[Nicolas B. Pierron [:nbp]]

(In reply to Robert J. Wood [:rwood] from comment #7)
> Hey all, sorry I will add more context to what I am currently working on for
> AWSY & Firefox OS.
> 
> No, we are not using the TP5 benchmarks. Instead of just testing the Firefox
> OS browser, we want to test all the gaia apps in order to ensure the main
> b2g process (at minimum) doesn't have a memory regression.

Ok, then this is indeed a duplicated bug.

> The test itself is an orangutan-based test that runs on the b2g emulator.

Orangutan-based?  Why choosing an interface which is so low level, this would be difficult to maintain when somebody adds a new icons to the home screen?

Comment 9

[Robert Wood [:rwood]]

(In reply to Nicolas B. Pierron [:nbp] from comment #8)
> (In reply to Robert J. Wood [:rwood] from comment #7)
> > Hey all, sorry I will add more context to what I am currently working on for
> > AWSY & Firefox OS.
> > 
> > No, we are not using the TP5 benchmarks. Instead of just testing the Firefox
> > OS browser, we want to test all the gaia apps in order to ensure the main
> > b2g process (at minimum) doesn't have a memory regression.
> 
> Ok, then this is indeed a duplicated bug.
> 
> > The test itself is an orangutan-based test that runs on the b2g emulator.
> 
> Orangutan-based?  Why choosing an interface which is so low level, this
> would be difficult to maintain when somebody adds a new icons to the home
> screen?

Hi, thanks Nicolas. We chose orangutan as we want to use a minimum amount of overhead as possible, and reduce the chance of having memory regressions caused by the automation itself. Yes true, however with other tools we would probably also need to update the test occasionally anyway according to gaia changes.

> 
> *** This bug has been marked as a duplicate of bug 899194 ***