656388 - Rename endurance tests to support running them as restart tests

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We want to restart the application between each endurance test by default. In order for this we will need to rename the tests appropriately.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Rename endurance tests to support running them as restart tests. v1

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 531690 [details] [diff] [review]
Rename endurance tests to support running them as restart tests. v1

Sorry, this inadvertently contains the patch from bug 656258. I'll fix that and provide a new version soon.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Rename endurance tests to support running them as restart tests. v1

This patch should be cleaner.

Comment 4

[u279076]

Comment on attachment 531693 [details] [diff] [review]
Rename endurance tests to support running them as restart tests. v1

Review of attachment 531693 [details] [diff] [review]:
-----------------------------------------------------------------

This looks okay to me -- r+

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 531693 [details] [diff] [review]
Rename endurance tests to support running them as restart tests. v1

>+++ b/tests/endurance/testAddons/test1.js

I don't think this level is enough. If you plan to have more tests for a specific subgroup you should add one more level. Also think about that in some circumstances it would be necessary to have multiple tests for one specific restart test.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to comment #5)
> Comment on attachment 531693 [details] [diff] [review] [review]
> Rename endurance tests to support running them as restart tests. v1
> 
> >+++ b/tests/endurance/testAddons/test1.js
> 
> I don't think this level is enough. If you plan to have more tests for a
> specific subgroup you should add one more level. Also think about that in
> some circumstances it would be necessary to have multiple tests for one
> specific restart test.

Do you have any suggestions? Should I make the current depth more descriptive or introduce a further level of depth? One proposal is below:

tests/endurance/
  testAddons/testOpenAndCloseAddonManager/test1.js
  testTabbedBrowsing/testOpenNewTabs/test1.js
  testTabView/testOpenNewTabs/test1.js
  testTabView/testSwitchTabs/test1.js

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This proposal looks fine. We only have to make sure to have the subgroups as separate subfolders.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It's not currently possible to add this additional level. I have raised bug 657798 to ensure we revisit this when Mozmill 2.0 is released, however for now we should land this as-is (assuming it otherwise passes review).

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 531693 [details] [diff] [review]
Rename endurance tests to support running them as restart tests. v1

>diff --git a/tests/endurance/testAddons/test1.js b/tests/endurance/testAddons/test1.js

We would still have to give the subfolder a meaningful name. In this case s/testAddons/testOpenCloseAddonsManager/. Otherwise how would you name the next test which is add-ons related? Same applies to all the other tests. It's not ideal right now, but we should start this move in a good and clean state.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Rename endurance tests to support running them as restart tests. v1.1

(In reply to comment #9)
> Comment on attachment 531693 [details] [diff] [review] [review]
> Rename endurance tests to support running them as restart tests. v1
> 
> >diff --git a/tests/endurance/testAddons/test1.js b/tests/endurance/testAddons/test1.js
> 
> We would still have to give the subfolder a meaningful name. In this case
> s/testAddons/testOpenCloseAddonsManager/. Otherwise how would you name the
> next test which is add-ons related? Same applies to all the other tests.
> It's not ideal right now, but we should start this move in a good and clean
> state.

Done.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 533214 [details] [diff] [review]
Rename endurance tests to support running them as restart tests. v1.1

>+++ b/tests/endurance/testEnterAndLeavePrivateBrowsing/test1.js

As discussed with Dave on IRC we should name those sub folder like 'testPrivateBrowsing_EnterAndLeave'. That way we have the right sorting in that folder per subgroup and makes it easier for us later to fold them into their own subfolder once Mozmill 2 has been released.

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Rename endurance tests to support running them as restart tests. v1.2

(In reply to comment #11)
> Comment on attachment 533214 [details] [diff] [review] [review]
> Rename endurance tests to support running them as restart tests. v1.1
> 
> >+++ b/tests/endurance/testEnterAndLeavePrivateBrowsing/test1.js
> 
> As discussed with Dave on IRC we should name those sub folder like
> 'testPrivateBrowsing_EnterAndLeave'. That way we have the right sorting in
> that folder per subgroup and makes it easier for us later to fold them into
> their own subfolder once Mozmill 2 has been released.

Done.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed as:
http://hg.mozilla.org/qa/mozmill-tests/rev/4a6c0d23b08b (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/f8aaa2af0bcc (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/7776071fd853 (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/9c35ffa32c2b (2.0)