532445 - Create new Unittest suite: unit test with Firebug installed

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[(not currently active) Ted Mielczarek]

There are a lot of unknowns about how extensions impact our codebase. We should work on ways to make this less scary. For a start, let's take Firebug, which is one of our most popular extensions (and critical to developers). We should try running our unit test suites (at least the Mochitest variants, reftest and crashtest, and probably the js browser tests) with Firebug installed. Probably once a day or so would be fine for starting out. We could grab the nightly build + test package, the latest Firebug XPI (or Firebug development XPI even, if there is such a thing), and tweak the command lines to run the tests with Firebug installed.

I'll need to sort out the precise details of how to run the test suites in this way, I'll put this back in the pool once I'm done with that.

Comment 1

[Boris Zbarsky [:bzbarsky]]

We should also run leak tests in this configuration, and ideally talos (at least the rss tests).

Comment 2

[(not currently active) Ted Mielczarek]

Should probably file a separate bug for Talos. Leak tests we can probably do, although we don't currently run them in a packaged-build configuration.

Comment 3

[Rob Campbell [:rc] (:robcee)]

thanks for the CC. At a bare minimum, this might reveal some weird incompatibility regressions early on.

Agree talos should be filed separately so I will do so.

Comment 4

[John J. Barton]

We discussed this bug in our Firebug conf call today
https://wiki.mozilla.org/Firebug/WeeklyUpdates/2010-03-23

Running the Firefox test suite with Firebug installed, active on all tabs, and active for all pages, would go a long ways to address crashing bugs on trunk for Firebug (tho these have become quite a bit rarer). 

Lots of Firebug function only runs on demand. We have a test system (user-interface focused of course) which we use for our releases, with about 120 test files. This system opens a XUL window and drives Firefox to load pages, apply  Firebug operations, and check results. 

Ideally we should run this test suite on FF nightly builds. However this is very time consuming: if the test fails we have to debug Firefox and Firebug. One new suggestion: we could work with someone from the platform team for a set time slot each week to triage any problems.

At present we have one user who tests Firebug latest on Firefox nightly builds. Well they did it once, and hopefully again.

Comment 5

[Rob Campbell [:rc] (:robcee)]

alternatively, someone in QA might be able to help out with running these things as part of a weekly checkup.

ctalbert: anybody on your team with a couple of cycles to help with this?

Comment 6

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Here is a wiki page about how to run automated tests for Firebug.
http://getfirebug.com/wiki/index.php/Running_Automated_Test_Suite

Quick summary of how to test Firebug 1.6:
1) Install latest Firebug 1.6: http://getfirebug.com/releases/firebug/1.6X/
2) Install latest FBTest 1.6: http://getfirebug.com/releases/fbtest/1.6/
3) Open FBTest window using "Firebug -> Open Test Console" menu item.
4) Click Run All button in the FBTest window.

Honza

Comment 7

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

People appear to be talking about two different things here. Comment 0 talks about running the fire*fox* tests with firebug installed. Comment 6 talks about running the fire*bug* tests.

I filed bug 559466 on running the fire*bug* tests automatically.

Comment 8

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I think we have made good progress on bug 559466 - Integrate firebug test suite with automated tests.

Any progress on this one? Can I help somehow to have firebug installed in testing profiles?

Honza

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Bugcleanup.

This seems to be about developing a unittest-suite running firefox-with-firebug-installed. This is very similar to bug#532637, which is about developing a Talos suite measuring firefox-with-firebug-installed. Tweaked summary and moved this bug to Testing:General for suite development. 

Once this new suite is developed, please file a separate bug in
mozilla.org/ReleaseEngineering to enable running the new suite on builds
automatically.

Comment 10

[Geoff Brown [:gbrown]]

Mass closing bugs with no activity in 2+ years. If this bug is important to you, please re-open.