604230 - Script to synchronize cherry-picked mozmill-tests with mozilla-central

Status: RESOLVED INCOMPLETE

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Jeff Hammel]

Attachment: list of current tests in m-c for bb

for instance, consider the basic list of what is in there now (attached)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 483014 [details] [diff] [review]
list of current tests in m-c for bb

Sorry that I have missed that yesterday, but you should also include all the shared modules. Also I wonder if we could place this file in a sub folder. I would like to have a script here too which helps us with synching tests between both repositories (for the beginning only m-t -> m-c).

Comment 2

[Jeff Hammel]

Attachment: list of current tests in m-c for bb as well as a script to update from them and generate them

Comment 3

[Jeff Hammel]

(In reply to comment #1)
> Comment on attachment 483014 [details] [diff] [review]
> list of current tests in m-c for bb
> 
> Sorry that I have missed that yesterday, but you should also include all the
> shared modules. Also I wonder if we could place this file in a sub folder. I
> would like to have a script here too which helps us with synching tests between
> both repositories (for the beginning only m-t -> m-c).

Updated attachment posted.  I didn't include the shared modules as all of these should be copied over from m-t regardless of whether they're on m-c or not (as discussed on IRC).  The new attachment also includes a script for updating m-c or updating the list of tests from m-c

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 483271 [details] [diff] [review]
list of current tests in m-c for bb as well as a script to update from them and generate them

>+firefox/testPreferences/testSetToCurrentPage.js
>+firefox/testToolbar/testBackForwardButtons.js
>+firefox/restartTests/testDefaultBookmarks/test1.js
>+firefox/testTechnicalTools/testAccessPageInfoDialog.js
>+firefox/testPrivateBrowsing/testStartStopPBMode.js
>+firefox/test-files/password_manager/login_form.html
[..]
>+firefox/test-files/awesomebar/copypaste.html
>+firefox/testAwesomeBar/testGoButton.js

Can you please write this text file sorted by folder and file name? That would make it easier for us to check which tests aren't part of m-c yet.

>diff --git a/update_mozilla-central.py b/update_mozilla-central.py

Please rename that script to buildbot_update.py. Both files should be located right next to each other. And once we branch for 2.0 it will not be mozilla-central anymore. So the current name is not what we want to use.

>+# copy the files
>+for line in file(TESTS).readlines():

Before you copy the files from the m-t repository you will have to delete all files in m-c first. Otherwise removed tests from the txt file will be still existent. Same to what you are doing with shared modules.

Otherwise it looks nice. I will test it once the comments above have been addressed.

Comment 5

[Jeff Hammel]

Will need to update m-c before this is actionable; there are a few other issues to clear up too

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This bug is really about the script to copy over the selected files to mozilla-central. Renaming summary to make it easier to find.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Jeff, would be great if you could come up with a patch soon, so we can fix bug 611844. A lot of stuff has been changed meanwhile, and I don't want to manually mess-up with the tests. Thanks.

Comment 8

[Jeff Hammel]

(In reply to comment #4)
> Comment on attachment 483271 [details] [diff] [review]
> list of current tests in m-c for bb as well as a script to update from them and
> generate them
> 
> >+firefox/testPreferences/testSetToCurrentPage.js
> >+firefox/testToolbar/testBackForwardButtons.js
> >+firefox/restartTests/testDefaultBookmarks/test1.js
> >+firefox/testTechnicalTools/testAccessPageInfoDialog.js
> >+firefox/testPrivateBrowsing/testStartStopPBMode.js
> >+firefox/test-files/password_manager/login_form.html
> [..]
> >+firefox/test-files/awesomebar/copypaste.html
> >+firefox/testAwesomeBar/testGoButton.js
> 
> Can you please write this text file sorted by folder and file name? That would
> make it easier for us to check which tests aren't part of m-c yet.

Done: http://k0s.org/mozilla/hg/mozmill-tests-patches/rev/b8a24f83b5bc

> >diff --git a/update_mozilla-central.py b/update_mozilla-central.py
> 
> Please rename that script to buildbot_update.py. Both files should be located
> right next to each other. And once we branch for 2.0 it will not be
> mozilla-central anymore. So the current name is not what we want to use.

I'm not sure I understand this comment.  The tests currently live in mozilla-central.  Is that not the plan?  The tests aren't purely for buildbot;  that is just the current usecase.

Comment 9

[Jeff Hammel]

(In reply to comment #1)
> Comment on attachment 483014 [details] [diff] [review]
> list of current tests in m-c for bb
> 
> Sorry that I have missed that yesterday, but you should also include all the
> shared modules. Also I wonder if we could place this file in a sub folder. I
> would like to have a script here too which helps us with synching tests between
> both repositories (for the beginning only m-t -> m-c).

Moving forward, at least for the m-c case, its probably a better idea to move to manifests whereby tests can be enabled/disabled via editting the file instead of pushing tests and backing out.  Then the entire set of tests can be synced and more easily enabled/disabled as needed.

I hesitate putting a lot of logic in what should be an interim ad hoc script.  We should really refine what our syncing strategy is, not just for mozmill-tests but in general for:
- test harnesses that aren't native to m-c
- tests that aren't native to m-c

Comment 10

[Jeff Hammel]

Another thing to consider is hg.  This script currently does nothing wrt version control.  Even if we e.g. remove files, you will still have to hg remove them

Comment 11

[Jeff Hammel]

Attachment: list of current tests in m-c for bb as well as a script to update from them and generate them

updated script;  still not sure what the strategy is for the hg issue (and to reiterate, i'd prefer to put effort into having manifests)

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #8)
> > Please rename that script to buildbot_update.py. Both files should be located
> > right next to each other. And once we branch for 2.0 it will not be
> > mozilla-central anymore. So the current name is not what we want to use.
> 
> I'm not sure I understand this comment.  The tests currently live in
> mozilla-central.  Is that not the plan?  The tests aren't purely for buildbot; 
> that is just the current usecase.

You are using the 'buildbot_list.txt' file to store the tests which have to be copied. The script itself references mozilla-central. It's hard to see that both files are bind to each other. I would still prefer to use buildbot because:

* mozilla-central will be branched in the next couple of months and our repository too. As result we will have a new branch named mozilla-2.0. Having a name of mozilla-central in the script name, is confusing. 'buildbot' will stay the same across all possible branches in the future.

(In reply to comment #11)
> updated script;  still not sure what the strategy is for the hg issue (and to
> reiterate, i'd prefer to put effort into having manifests)

Personally I don't have any problem with manually doing the hg stuff. It would be even safer as having too much magic in the script. Regarding manifests, those will not be available before Mozmill 2.0 so I wouldn't consider those as an option in the next months. One idea we have discussed yesterday in the Automation meeting was to put a line into the files executed by buildbot, which state that this test runs in buildbot. That would at least reduce the cost to maintain a separate list of files.

Leaving review request open for now until we are clear about the strategy.

Comment 13

[Jeff Hammel]

(In reply to comment #12)
> (In reply to comment #8)
> > > Please rename that script to buildbot_update.py. Both files should be located
> > > right next to each other. And once we branch for 2.0 it will not be
> > > mozilla-central anymore. So the current name is not what we want to use.
> > 
> > I'm not sure I understand this comment.  The tests currently live in
> > mozilla-central.  Is that not the plan?  The tests aren't purely for buildbot; 
> > that is just the current usecase.
> 
> You are using the 'buildbot_list.txt' file to store the tests which have to be
> copied. The script itself references mozilla-central. It's hard to see that
> both files are bind to each other. I would still prefer to use buildbot
> because:
> 
> * mozilla-central will be branched in the next couple of months and our
> repository too. As result we will have a new branch named mozilla-2.0. Having a
> name of mozilla-central in the script name, is confusing. 'buildbot' will stay
> the same across all possible branches in the future.
> 
> (In reply to comment #11)
> > updated script;  still not sure what the strategy is for the hg issue (and to
> > reiterate, i'd prefer to put effort into having manifests)
> 
> Personally I don't have any problem with manually doing the hg stuff. It would
> be even safer as having too much magic in the script. Regarding manifests,
> those will not be available before Mozmill 2.0 so I wouldn't consider those as
> an option in the next months. One idea we have discussed yesterday in the
> Automation meeting was to put a line into the files executed by buildbot, which
> state that this test runs in buildbot. That would at least reduce the cost to
> maintain a separate list of files.
> 
> Leaving review request open for now until we are clear about the strategy.

Maybe we should rename the script and the manifest to `update-firefox-tests.py` and `firefox-tests.py` ? I dunno, its a pretty ambiguous name too.  buildbot really shouldn't be referenced at all, so buildbot_list.txt should go.

If it is worth thinking about at this point (not sure what the larger plan is) we could build a minimal manifest for our purpose now with the goal of having something useful going forward.  Or...something.  It shouldn't be hard to implement a minimal manifest, even if it doesn't have all the features.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #13)
> Maybe we should rename the script and the manifest to `update-firefox-tests.py`
> and `firefox-tests.py` ? I dunno, its a pretty ambiguous name too.  buildbot
> really shouldn't be referenced at all, so buildbot_list.txt should go.

Why you don't wanna reference buildbot? A name which includes Firefox will not be logical to users of our external repository. Lets go forward with buildbot. If we have a problem with it, we can change it later though. And btw. some of our projects have 'buildbot' in the description, so it would make sense for us and contributors.

> If it is worth thinking about at this point (not sure what the larger plan is)
> we could build a minimal manifest for our purpose now with the goal of having
> something useful going forward.  Or...something.  It shouldn't be hard to
> implement a minimal manifest, even if it doesn't have all the features.

Lets tackle that in a follow-up bug. We are still discussing a way to make it more obvious which tests are part of the buildbot test-run.

Comment 15

[Jeff Hammel]

I'd prefer not to reference buildbot since the tests are gneeral purpose.  While currently they are only used by buildbot see e.g. bug 568642

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So what about firefox-tree? Geo, do you have another idea?

Comment 17

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Honestly? I'd have gone with buildbot. 

We're talking about pulling out the subset that's going to be run by buildbot, and naming the script after the goal makes the most sense to me. If buildbot went away and was replaced by something else, I'd just...rename the script.

Short of that? Dunno, update_releng? Point is that it's the subset release engineering runs for us. However we want to express that works for me.

Comment 18

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Oh, should add, totally agree with Henrik's point re: branching vs. naming. Branch names should never be reflected in the filenames IMO.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Move of Mozmill Test related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 491394 [details] [diff] [review]
list of current tests in m-c for bb as well as a script to update from them and generate them

>+# update shared module
>+MODULES_TARGET=os.path.join(TARGET, SHARED_MODULES)
>+os.makedirs(MODULES_TARGET)

This creates another 'shared-modules' subfolder inside the 'shared-modules' folder. Otherwise it looks good.

Please update the filenames as spoken above to include buildbot.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Nothing we are working on at the moment. Even not sure when we will have the next attempt or if we want to. Closing as incomplete for now.