Automate running of Firebug testsuite

RESOLVED WONTFIX

Status

Release Engineering
General
RESOLVED WONTFIX
8 years ago
5 years ago

People

(Reporter: ahal, Assigned: joduinn)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

This is a follow up of Bug 559466.

The Firebug unit tests have been automated and running without problems for several weeks now.  Results of the tests can be seen here: http://getfirebug.com/tests/content/testbot/results/

While the current setup works great, the automated testing should have a more permanent home in the standard buildbot framework.

Some things to consider:

1) The buildbot step will call into our existing python Firebug test runner (http://github.com/ahal/Firebug-Test-Runner)

2) The Firebug test files themselves can be checked into and tracked in a repository of our choice so we can see if a failed test was recently modified.  Currently, the repository of tests is co-located on the webserver.

3) The python module will need to hit an apache webserver inside the Mozilla network.  It can use qm-codecover-02 (which buildbot already depends on)

4) The Firebug tests hosted on the webserver in (3) will be updated by our fb_update script which will pull down the latest Firebug tests and store them in the repo mentioned in (2).  We can schedule this update to occur with whatever frequency we desire.

At the moment the Firebug Test Runner runs every hour, pulls the latest tinderbox builds and checks for new changesets before running the FBTests on that build.  I don't know what a patch to buildbot should look like, please let me know what you think.
Assign to John for prioritization
Assignee: nobody → joduinn
summary of meeting with bmoss, ctalbert

1) This firebug test suite is working on some ATeam machines. They are not clones of RelEng refimages, but are Vista/centos5/ubuntu9.x.  Ctalbert thinks there is very little installed there - should be "easy" to get running on ref image machines. joduinn will lend ctalbert 3 machines to make sure it runs on ref image machines before we start switching networks, etc.

2) This will need a standalone apache webserver accessible within the build network, because the test suites grabs pages and configuration settings from this server. We might be able to bring over the experimental one being used by ATeam, still TBD.

3) Its unclear how frequently we want/need to run this. Per-checkin might be too much, but maybe nightly? weekly? monthly? Given we can trigger this suite on existing builds, we can run this less frequently, and then if a problem is discovered, go back and test on older builds to find when it broke. The real goal is to make sure we do not ship a Firefox that breaks Firebug.

4) The work left to do here is automate running of this existing suite, so morphing summary to match.
Summary: Integrate Firebug Automated tests with buildbot → Automate running of Firebug testsuite
1) Once we get the ref image machines, I'll move over the test runner and ensure that it is stable.  When will these be available?

3) From watching the Firebug tests run over the last month or so I've noticed that for the most part tests don't break from day to day.  I think that weekly should be enough. We could always adjust the frequency as needed once we get this up and running.
Are we waiting for the ref image machines to become available?  Or where can I find them?

Thanks,
Andrew
Depends on: 631659, 631665

Updated

7 years ago
Depends on: 635019
1) Found during triage. Sorry for not being better about dealing with this before now.

2) Per email with ctalbert, honza, this is no longer needed, as this same need has since been addressed by separate systems that ATeam setup, and which feel "good enough". We're closing as WONTFIX, because its closest fit - this is not exactly "FIXED" or "DUP" or "WORKSFORME".

If I've missed something, please reopen with details.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.