581211 - Automate running of Firebug testsuite

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

This is a follow up of Bug 559466.

The Firebug unit tests have been automated and running without problems for several weeks now.  Results of the tests can be seen here: http://getfirebug.com/tests/content/testbot/results/

While the current setup works great, the automated testing should have a more permanent home in the standard buildbot framework.

Some things to consider:

1) The buildbot step will call into our existing python Firebug test runner (http://github.com/ahal/Firebug-Test-Runner)

2) The Firebug test files themselves can be checked into and tracked in a repository of our choice so we can see if a failed test was recently modified.  Currently, the repository of tests is co-located on the webserver.

3) The python module will need to hit an apache webserver inside the Mozilla network.  It can use qm-codecover-02 (which buildbot already depends on)

4) The Firebug tests hosted on the webserver in (3) will be updated by our fb_update script which will pull down the latest Firebug tests and store them in the repo mentioned in (2).  We can schedule this update to occur with whatever frequency we desire.

At the moment the Firebug Test Runner runs every hour, pulls the latest tinderbox builds and checks for new changesets before running the FBTests on that build.  I don't know what a patch to buildbot should look like, please let me know what you think.

Comment 1

[[e-mail] (:bhearsum)]

Assign to John for prioritization

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

summary of meeting with bmoss, ctalbert

1) This firebug test suite is working on some ATeam machines. They are not clones of RelEng refimages, but are Vista/centos5/ubuntu9.x.  Ctalbert thinks there is very little installed there - should be "easy" to get running on ref image machines. joduinn will lend ctalbert 3 machines to make sure it runs on ref image machines before we start switching networks, etc.

2) This will need a standalone apache webserver accessible within the build network, because the test suites grabs pages and configuration settings from this server. We might be able to bring over the experimental one being used by ATeam, still TBD.

3) Its unclear how frequently we want/need to run this. Per-checkin might be too much, but maybe nightly? weekly? monthly? Given we can trigger this suite on existing builds, we can run this less frequently, and then if a problem is discovered, go back and test on older builds to find when it broke. The real goal is to make sure we do not ship a Firefox that breaks Firebug.

4) The work left to do here is automate running of this existing suite, so morphing summary to match.

Comment 3

[Andrew Halberstadt [:ahal]]

1) Once we get the ref image machines, I'll move over the test runner and ensure that it is stable.  When will these be available?

3) From watching the Firebug tests run over the last month or so I've noticed that for the most part tests don't break from day to day.  I think that weekly should be enough. We could always adjust the frequency as needed once we get this up and running.

Comment 4

[Andrew Halberstadt [:ahal]]

Are we waiting for the ref image machines to become available?  Or where can I find them?

Thanks,
Andrew

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) Found during triage. Sorry for not being better about dealing with this before now.

2) Per email with ctalbert, honza, this is no longer needed, as this same need has since been addressed by separate systems that ATeam setup, and which feel "good enough". We're closing as WONTFIX, because its closest fit - this is not exactly "FIXED" or "DUP" or "WORKSFORME".

If I've missed something, please reopen with details.