Closed Bug 914983 Opened 11 years ago Closed 10 years ago

[Marionette] Proposal to add running forever till timed-out functions

Categories

(Remote Protocol :: Marionette, defect)

Product:
Remote Protocol
Component:
Marionette
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wachen, Unassigned)

Details

I am thinking about doing --hours options for run_tests() to run test for as long as we want it. I don't know if it is proper or any opinions are welcomed.
I don't think that this function is proper for a test harness, since the ideal would be that tests will always pass until an unexpected failure occurs and we can call that a regression. This assumes that "something is wrong" and helps diagnose the issue, which is a different function.

If there is another reason for adding --hours that is related to running tests and not diagnosing the problem, then we should add it, but I don't think --hours is appropriate to add just for test diagnosis.
Dave, I actually really want to agree with you on that. However, since we think that stability should be something that we can show data of how many running hours did we finish to the telecompanies. I really want it to be more precise. Otherwise, we might need to adjust multiple times so that we can reach our goal to go above certain hours.

Also, we'd like it to run without restart and we'd like it to imitate user's behavior instead of launch/kiss an app multiple times. In that case, we want to write something different. Furthermore, we think about adding some more options into marionette client. The shuffle was almost for the same reason.


Malini, hey, yes, we do have a reason for that. We want the stability tests to run by hours. We want to shuffle all the tests and rerun them over and over again. In that case, we can better emulate the user behavior and show a good data by "hours" to telecompany and some other results to our company (maybe memory leak over hours, app effienciency over hours, and so on)
I think it makes more sense to add this option to the gaia endurance framework, rather than core marionette.
(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> I think it makes more sense to add this option to the gaia endurance
> framework, rather than core marionette.

+1
However, we are not going to use gaia endurance framework... Or, we need to change(modify) the gaia endurance framework to run in different ways
(In reply to Walter Chen from comment #6)
> However, we are not going to use gaia endurance framework... Or, we need to
> change(modify) the gaia endurance framework to run in different ways

Ah, ctalbert, jgriffin and I met with Bruce Weng yesterday and we discussed that we were going to modify the gaia endurance test harness (which uses gaiatest/gaia-ui-test) to accommodate your needs. We hope to send over an ateam member to Taipei to get you up and running as well. 

To get this working, Bruce would like to address the following:
- Execute asynchronous JavaScript could cause b2g hang, need to fix slowdown issue: Bug 915489

CCing Bruce Weng if there are any clarifications needed.

- Test cases need to re-write
1.Zac provided Gaia UI test cases at here https://github.com/mozilla/gaia-ui-tests/tree/master/gaiatest/tests
We can check these test cases and see how much we can leverage. (not an ateam specific task)

-Prevent a reset when a test case is completed and move to next test case
Bug 915572 

-Should keep the data and saved in the file system
Bug 915575
Hi, Brian and I are in charge of this as for now.
I still think that we need to do a by hour functions here.
I can do that if everyone agrees on that.
Yes, I agree that we should provide a way to run by hours, but this should live in an endurance framework, instead of runtests.py becoming a catch-all for every new function we want to add, or it will quickly get out of hand.

Right now, our endurance framework uses gaiatest, but that framework isn't specific to endurance, it's just used to run tests against the gaia ui.

Instead of having our runtests.py take on more and more specific responsibilities, I propose we create a marionette endurance runner (much like the runtests.py file used in gaiatest) which will work as the usual runtests.py does, but will also provide the --hours option, as well as any other options needed that are specific to endurance (ie: a run-until-fail option, with appropriate output upon failure? ), and you can use this runner. Thoughts?
This is just a general idea, but perhaps the right approach would be to have some kind of mixin for each new class of concerns want to add to our harnesses instead of doing new runtests.py for each. It will take a bit of rejiggering, but it seems it might be worth it so we can just add the EnduranceMixin to the core runner, so we'll have marionette tests with endurance profiling, or say a GaiaRunnerMixin with the EnduranceMixin and then we'd get the gaia-ui-test runner with all the fun of the endurance runner.
We won't do this by endurance framework. Endurance framework is a different thing than the MTBF test we are doing now. 

1. I think I talked with Tony and some other people here in Taipei about this. I think Tony agreed on this one. It's a long explanation. However, basically, we want all the gaia-ui-tests cases can be reused and we don't need to simply create a new framework upon this

2. In comment 10, I should then start to create MTBFMixin in this case, right? I really want to just do a simple modification and go. However, if this is what it takes, and you can agree on this. Then, I will do since I need to complete something really fast.

How do you think?
The endurance and gaia-ui-test frameworks have been merged back together and currently live in the gaia repo: https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/gaiatest

I like the mixin idea but think that will require a lot of refactoring to support, so if you need something quickly, I suggest going with comment #9, and creating something like runmtbf.py that sublcasses GaiaTestRunner and GaiaTestOptions, unless Malini has a different idea.
(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> The endurance and gaia-ui-test frameworks have been merged back together and
> currently live in the gaia repo:
> https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/
> gaiatest
> 
> I like the mixin idea but think that will require a lot of refactoring to
> support, so if you need something quickly, I suggest going with comment #9,
> and creating something like runmtbf.py that sublcasses GaiaTestRunner and
> GaiaTestOptions, unless Malini has a different idea.

+1, I'm happy to delay the Mixin work in favour of speed. Getting a new version of runtests.py will work for now. I've filed Bug 925398 so we can get around to doing the mixin change eventually.
jgriffin, mdas,
Thanks for the suggestions. I will go with comment #9 now. Later, I will be on mdas's Mixin work!

I think I need a little bit time to pick up the new structure, and thanks again.
(In reply to Walter Chen[:ypwalter][:wachen] from comment #14)
> jgriffin, mdas,
> Thanks for the suggestions. I will go with comment #9 now. Later, I will be
> on mdas's Mixin work!
> 
> I think I need a little bit time to pick up the new structure, and thanks
> again.

Have you had any progress with this? I may have a need for this functionality soon and can either help you, or take on this work if that's the case.
Update: The option to run tests for a specific duration is being added to a MTBF test driver rather than core marionette. The MTBF test driver initiates/controls tests that are based on a new MTBF gaiatest case class.
We are now using MTBF driver to get this running as for now.
You may find the driver at https://github.com/Mozilla-TWQA/MTBF-Driver

Thanks for Paul Yang's helping.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.