1080092 - Add test framework

Status: RESOLVED FIXED

(bugzilla.mozilla.org Graveyard :: Bugzilla Anthropology Metrics, defect)

Description

[Kyle Lahnakoski [:ekyle]]

MoDevMetrics is now large enough to require tests.  Page tests looking for simple failure will go a long way to identifying bugs early.

Summary of Problem
------------------

MoDevMetrics has two main parts:  The individual dashboard pages, and the core library.  When a page is changed, the core library is modified to support new features or fix bugs.  This change to the core could break other pages, so the work done on pages and the core is in a separate branch.  When work is complete, that branch is posted its own directory on my people page.

At this time, pages and their (unique) core code, inhabit eight different branches (plus 3 in charts.mozilla.org).  Merges are done, from time to time, so the core code stays synchronized.  Merges are only done *into* the branch I am working on so the manual tests can be limited to the pages I am working on, and I can ensure the merge did not introduce bugs. If a lot of time passed between merges I can totally forget what i did to the core to cause a breakage, which increases my debugging time.

Adding tests will not reduce the number of branches in the short or medium term: Branches provide too excellent isolation that only the most serious test suites could provide.  Tests *will* identify many bugs early while my mind is fresh.

If you can help with this problem, either by guiding me to the right test framework, or suggesting an even better strategy, please comment.

Thanks

Comment 1

[Kyle Lahnakoski [:ekyle]]

Malini, Dave

Your names came up as I discussed my problem with others:

My thought is to use Selenium to test these pages.  But maybe Mozilla has a similar tool; and I can benefit from using the same tech as the rest of the team.  What is the name of what I looking for?

Thanks!

Comment 2

[David Burns :automatedtester]

We have the next generation of Selenium testing being built into Firefox. This is called Marionette and is used to test parts of Firefox and B2G. If you are testing a web site stick with Selenium for now and we can easily switch it marionette later as it should be seemless

Comment 3

[Kyle Lahnakoski [:ekyle]]

For those that are curious:

Here are a list of the branches
https://github.com/klahnakoski/MoDevMetrics/branches

You will require the MozillaVPN to access some of these charts
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=30769829

Example of some bug counting, (one branch)
http://people.mozilla.org/~klahnakoski/modevmetrics/Bug-Counts.html#sampleInterval=week&sampleMin=2014-06-01&programFilter=B2G+2.1

Count reviews (another branch)
http://people.mozilla.org/~klahnakoski/review/Review-byTop.html#teamFilter=ctalbert%40mozilla.com

Here is an old games benchmarking charts, Perfy. (yet another branch) 
http://people.mozilla.org/~klahnakoski/perfy/Perfy-Overview.html#

Comment 4

[Kyle Lahnakoski [:ekyle]]

It appears Selenium can not get all the console logs out of Firefox.  Whether this is Selenium, or Firefox, I do not know.  It may still be possible with Java, and setting some System properties, so an output file can be generated; but what I read is too vague.  Also it may be possible with installing Firebug, an extension for it, and then setting config properties to use a specific profile; again that was too vague, and no (full) examples exist.

Here is the branch that has been stripped to run a test:
https://github.com/klahnakoski/MoDevMetrics/tree/test_hello_world

Comment 5

[Kyle Lahnakoski [:ekyle]]

It appears Selenium does not retrieve non-visible text from a page, despite it easily retrieving non-visible elements.  I regret loosing many hours of my life to making this surprising discovery.

Here is a branch I would have thought works, but does not.
https://github.com/klahnakoski/MoDevMetrics/tree/selenium_test

Comment 6

[Kyle Lahnakoski [:ekyle]]

A rough test framework is in place

https://github.com/klahnakoski/MoDevMetrics/tree/a51485b3c7273fe66b6b0c1796fe951d93312553/tests