Closed Bug 1080772 Opened 10 years ago Closed 9 years ago

Convert a few mozmill tests to Marionette as POC

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: whimboo)

References

Details

These should be tests that don't rely on in-progress Marionette enhancements; Henrik should be able to recommend some.
I started an empty repo for the tests:
https://github.com/ahal/marionette-release-tests

We can move it under the mozilla organization, or into hg.m.o or wherever later. Also I'm not crazy about the name but I don't have any better ideas.

You'll notice a 'harness' subdirectory.. I expect this to be mostly a very thin wrapper around the marionette runner. I mainly just wanted it to provide an entry point to running the tests that lived alongside the tests.
Just noting that our tests will not only be release tests. We also test any nightly build of Firefox.
Yes, feel free to suggest better names for it :).

I transferred this to the 'mozilla' organization and gave the automation team permissions for it:
https://github.com/mozilla/marionette-release-tests

Finally I added a single POC test (test_back_forward.py). To run it, install the package and then do:
run-release-tests -b <path to firefox> tests/manifest.ini

It works, but there are a few things that need to be fixed:
1) Host local test files so we don't hit the net (I think marionette already sets up a webserver)
2) Marionette has no "waitForPageLoad" function. Instead it implicitly waits when you call navigate(), go_forward() or go_back(). But we don't want to use those functions when testing the back/forward buttons, so we may need some kind of general purpose wait_for_page_load. In this case we don't really care whether content has finished loading or not.. so maybe it's not needed.
3) foward.is_displayed() is returning False when it should be returning True.. need to investigate.

I'm PTO this afternoon and holiday on Monday. So won't have time to look into this further until mid next week.
(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Yes, feel free to suggest better names for it :).
> 
> I transferred this to the 'mozilla' organization and gave the automation
> team permissions for it:
> https://github.com/mozilla/marionette-release-tests
> 
> Finally I added a single POC test (test_back_forward.py). To run it, install
> the package and then do:
> run-release-tests -b <path to firefox> tests/manifest.ini
> 

Nice!

> It works, but there are a few things that need to be fixed:
> 1) Host local test files so we don't hit the net (I think marionette already
> sets up a webserver)

Yep, it does.  A subclass could override "server_root" to set the location of static files to serve, see e.g., https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/runtests.py#L52

> 2) Marionette has no "waitForPageLoad" function. Instead it implicitly waits
> when you call navigate(), go_forward() or go_back(). But we don't want to
> use those functions when testing the back/forward buttons, so we may need
> some kind of general purpose wait_for_page_load. In this case we don't
> really care whether content has finished loading or not.. so maybe it's not
> needed.

We probably want a ReleaseTestMixin (or whatever) that could be home to this method.

> 3) foward.is_displayed() is returning False when it should be returning
> True.. need to investigate.
> 
> I'm PTO this afternoon and holiday on Monday. So won't have time to look
> into this further until mid next week.
The latest commit sets the webserver to the 'tests/resources' directory and updates the test to use locally hosted content. We may want to look into creating a server-locations.txt, but I guess that's not really necessary.

I think it's pretty much ready for some additional POC tests to get converted. Any tests that don't depend on the dependencies of bug 1080766 are fair game. I'm going to shift focus to bug 1080764 though.
I will get started with this tomorrow. Sorry for the delay but there is a pressing mozmill release upcoming.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Ok here's where I'm at with this bug for now:

1) There are two working tests, one that tests the toolbar a bit [1] and one that clicks the new-tab button [2]. Note that [2] depends on the un-landed patch for anonymous node support in bug 1080764.
2) I created a new "firefox marionette automation libary" [3] as a place to store shortcuts for finding commonly used elements and manipulating the Firefox UI. Both tests already use this library.
3) I think both these repos should get moved into mozilla-central eventually.. though github makes sense for now.
4) Their names don't have to be final, but everyone in an irc discussion agreed that "firefox-greenlight-tests" was better than "marionette-release-tests".

My plan is to shift focus back to landing anonymous node support, and then come back to this to start filling in some holes that will be needed by the tests.


[1] https://github.com/mozilla/firefox-greenlight-tests/blob/master/greenlight/tests/functional/toolbar/test_back_forward.py
[2] https://github.com/mozilla/firefox-greenlight-tests/blob/master/greenlight/tests/functional/tabs/test_new_tab.py
[3] https://github.com/ahal/firefox-puppeteer
(In reply to Andrew Halberstadt [:ahal] from comment #7)
> 2) I created a new "firefox marionette automation libary" [3] as a place to
> store shortcuts for finding commonly used elements and manipulating the
> Firefox UI. Both tests already use this library.
> 3) I think both these repos should get moved into mozilla-central
> eventually.. though github makes sense for now.

I would like to mention and what we have discussed in our initial meeting, this is a totally different approach you are going here in separating those out, and trying to get them into mozilla-central. I did explain our reason why we don't have them in the tree. Github may be an option if we find a python library/client to assist us with cloning, and which can easily be distributed via our environment. 

Is there a real reason why we should directly start with two repositories? One for ui and backend libs, and another for tests? In case of necessary changes it will be hard to synchronize between those two. What are the benefits?

Given that this bug was just to create a simple list of tests, which have to work in Marionette I feel we are getting way far ahead here. Can't we concentrate us for now on creating the tests and libs in the same temporary repository? 

> 4) Their names don't have to be final, but everyone in an irc discussion
> agreed that "firefox-greenlight-tests" was better than
> "marionette-release-tests".

I'm fine with that name. It's temporary maybe even the repository. I have to flesh out later how to best wrap our automation around it.

> [3] https://github.com/ahal/firefox-puppeteer

That would stay in conflict with https://github.com/zackw/firefox-puppeteer, and as mentioned above can we get the ui lib back into the firefox-greenlight-tests repo please?
Flags: needinfo?(ahalberstadt)
No, I'm 100% opposed to the ui lib living alongside the tests. The reason being is that this lib will be useful to anyone who wants to use marionette to drive Firefox. Since a goal of marionette is to replace webdriver's FirefoxDriver, this library could be the beginnings of that. I kind of envision this library being an extension to marionette-client specifically for Firefox. So I think it should live in m-c alongside marionette-client.

As far as moving the tests into m-c is concerned, I guess ultimately it's up to the person who has to maintain them. If you will be the maintainer and want them out of tree, then I won't stop you :).
Flags: needinfo?(ahalberstadt)
If it turns out that others want to use that code or if we have to share it as you say it will replace webdriver for Firefox, I'm happy to search for solutions. m-c might still be a problem, but I would feel better to see it all working first.

Would you have anything against it when I move the ui libs back into firefox-greenlight-tests?
Yes, I just said I would have something against it :).

These tests already depend on marionette-client which lives in m-c, so I don't see what the problem is on them depending on an additional library. It doesn't make the setup any more difficult than it already is.
On second thought, the "greenlight" tests could also be the unit tests for the firefox automation library, so if we think about it that way, combining the two might make sense. If we went this route, I'd really like to see them live in m-c though..
The problem is that in the current state it is kinda complicated to maintain two separate repositories when we quickly want to build up tests and libraries. Updates would be needed for both, and you cannot do that in a single changeset but have to code blindly and hope the test will work then. So in the current state I really feel this is totally overhead and slows us or especially me down.

I really don't see why you are forcing that at this state of the project, as long as I haven't audited all of our other repositories and tools if those would easily be converted to support that.
(In reply to Andrew Halberstadt [:ahal] from comment #12)
> On second thought, the "greenlight" tests could also be the unit tests for
> the firefox automation library, so if we think about it that way, combining
> the two might make sense. If we went this route, I'd really like to see them
> live in m-c though..

We will not have the the automation scripts live together with the transformed Mozmill tests in the same repository. The scripts don't require branching into all the different release branches, and will always work on the latest state of Marionette. All that I have explicitly mentioned in the meeting and we agreed on that (as mentioned above). So I feel it strange how this goes now...
Ok, I didn't understand you meant temporarily. If having the tests in the same repo makes things easier, then let's do that for now. I'm worried that the library will somehow get intricately tied to this one test suite and it will require effort to re-generalize it at a later date.
(In reply to Henrik Skupin (:whimboo) from comment #14)
> We will not have the the automation scripts live together with the
> transformed Mozmill tests in the same repository. The scripts don't require
> branching into all the different release branches, and will always work on
> the latest state of Marionette.

Maybe they don't require branching, but they have stuff like this:
http://hg.mozilla.org/qa/mozmill-tests/file/2b3167614438/firefox/tests/functional/testTabbedBrowsing/testNewTab.js#l50
(In reply to Henrik Skupin (:whimboo) from comment #14)
> All that I have explicitly mentioned in the
> meeting and we agreed on that (as mentioned above). So I feel it strange how
> this goes now...

I'm sorry, it didn't occur to me at the time.
(In reply to Andrew Halberstadt [:ahal] from comment #16)
> Maybe they don't require branching, but they have stuff like this:
> http://hg.mozilla.org/qa/mozmill-tests/file/2b3167614438/firefox/tests/
> functional/testTabbedBrowsing/testNewTab.js#l50

This was necessary so we were able to test builds of the australis project branch. This should have been removed meanwhile. I know that this is not ideal, but there is not an easy solution for that. Our tests are somewhat special with a close focus on our community, and cannot be compared to tests which are part of m-c, which are only run by buildbot and devs. There are always pros and cons, and we have decided to not force our testers in QA to completely download the whole tree(s) for testing Firefox in its flavors. There is 1MB worth of tests compared to >1GB of data to download. Compare those numbers and you can imagine why we did that.

So my proposal really is to use firefox-greenlight-tests for building up a list of tests and libs to use for testing our requirements against Marionette. Everything else should be discussed and decided later, so we are not hold back from getting this transition done as quick and good as possible.
(In reply to Henrik Skupin (:whimboo) from comment #18)
> So my proposal really is to use firefox-greenlight-tests for building up a
> list of tests and libs to use for testing our requirements against
> Marionette. Everything else should be discussed and decided later, so we are
> not hold back from getting this transition done as quick and good as
> possible.

Sounds good. Using the github to get started was also my intention. You're right, we can talk about the other stuff later.
So I miss various files in the current green light repository which don't let me get started. I tried to copy the library files as created in firefox-puppeteer into the the current repo, but there is still at least the .testcase module for FirefoxTestCase missing.

Ahal, can you please sync your local repo so that I can get started? Thanks.
Flags: needinfo?(ahalberstadt)
I talked with Andrew and IRC and he pushed the necessary file. So I'm good to get started. As we talked I temporarily removed the dependency to puppeteer, so we can get a faster pace.
Flags: needinfo?(ahalberstadt)
I'm going to close this bug in favor of bug 1119715 and its dependencies. It will allow us to better manage the conversation when we drill it down to specific P1 areas.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.