637306 - Restructure the mozmill tests repository

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With more and more testruns coming up we should restructure the whole Mozmill tests repository. Each of the testruns should become a top-level folder and should not be located under Firefox anymore.

Here the proposal:
* addons
* endurance
* lib (formerly shared-modules/)
* general (formerly firefox/)
* l10n
* restart (probably not necessary with mozmill 2.0)
* test-files
* update

One concern raised by Geo in the last meeting would be the 'general' folder. Which name could we use for it? One possible solution is to keep 'firefox'. That would allow us to not have to update all of our tests, which would be a big pro. What do you think?

Comment 1

[Aaron Train [:aaronmt]]

I agree with the comment from Geo, except that than all these folders are, in essence related to Firefox, would a subfolder named 'firefox' be a best fit? Otherwise, this current structure would be better with a parent folder named 'firefox', with this current listing underneath.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The whole repository only contains tests for Firefox, so no top folder named 'firefox' should be used to stick all other folders beneath.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Keeping 'firefox' as a top level directory would be confusing if the various testruns folders are raised to the top level. I see Geo's point about general, but am struggling to come up with a better name for this folder...

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Henrik's point about keeping naming in sync was very good. I wasn't thinking about the implications of that at all when I made that suggestion.

If it's a huge PITA to change names on reports, URLs, so forth, let's just keep it as "general".

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, talked one more time with Geo yesterday and we came up with a slight different approach. Reason for that is, that the restart tests are not really a different test-run but are part of the general testrun and for Mozmill 2.0 there will even be no difference because of the manifest files. So we propose the following structure:

* addons
* endurance
* general
** restart
* l10n
* lib
* test-files
* update

The only remaining question I have now is if we want to include 'test' into the folder name:

* files
* lib
* testAddons
* testEndurance
* testGeneral
** restart
* testL10n
* testUpdate


(In reply to comment #4)

> If it's a huge PITA to change names on reports, URLs, so forth, let's just keep
> it as "general".

It's not such a problem. I have regex entries for all testrun types and I would only have to add a new one to catch a new name.

Comment 6

[u279076]

If we are breaking it down to different test-types, then why not call general what it really is: "functional"

All tests which exist in general (firefox currently) are BFTs (basic FUNCTIONAL tests).

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's a good idea. Also the run_ prefix should probably be used to better reflect what the sub-folders are. Anyone objections? 

files
lib
run_addons
run_endurance
run_functional
  restart
run_l10n
run_update

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I think functional is a good idea, but I'm not keen on the run_ prefix. As these are all folders containing tests, could we not have a structure where 'tests' is a top level?

files
lib
tests
  addons
  functional
    restart
  l10n
  update

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sound fair and should also work with the regex to assign sent reports to a known testrun. Lets try to finalize it in todays meeting.

Comment 10

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

I don't like the run_ prefixes either. I'm fine with Dave's proposal.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We agreed on:

data
lib
tests
  addons
  functional
    restart
  l10n
  update

Comment 12

[u279076]

Now that these changes are checked in the documentation needs to be updated.  These changes should also be communicated to the community.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

All documentation has been updated. I will now write a blog post.

This bug has been fixed now. Thanks for your help!