Closed Bug 559466 Opened 14 years ago Closed 12 years ago

Integrate firebug test suite with automated tests

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: sicking, Assigned: ahal)

Details

Attachments

(1 file)

Firebug has an automated test suite which tests that firebug works as it should. However this test suite is currently run manually and so regressions caused by changes in firefox are only detected after the fact.

We should fix this.

Currently the steps to run these tests are:

1) Install latest Firebug 1.6: http://getfirebug.com/releases/firebug/1.6X/
2) Install latest FBTest 1.6: http://getfirebug.com/releases/fbtest/1.6/
3) Open FBTest window using "Firebug -> Open Test Console" menu item.
4) Click Run All button in the FBTest window.

So what we need is to create a profile with these two addons installed, and then create a way for the buildbot to start firefox such that the tests automatically run and report pass/fail back.

(Ideal would of course be to know which exact test failed, like we do with our current test suites)
Note that this is different from bug 532445. That bug is about running the existing tests under additional conditions (with firebug installed). This bug is about running additional tests.
Yes, sorry for the confusion. (I'd be happy with either outcome ;-)

If you run with the two extensions above and command line option 
"-runFBTests https://getfirebug.com/tests/content/testlists/firebug1.6.html", 
then the FBTest window will come up and run all of the tests in the given list, 
then print two lines Passed # and Failed #
then exit.

We can print the summary of failures in the console or post them somewhere or whatever. The current format is shown by the examples posted by tasoss:
http://getfirebug.com/wiki/index.php/Test_Results_with_Firefox.next 

More info http://getfirebug.com/wiki/index.php/Firebug_Tests
We should publish the test list with the XPI to insure that Firefox.next is tested against a known-good list. We don't want Firebug project changes to the list to break the Firefox.next build.

A secondary concern are the tests themselves. We try to dup them and not change them but we have no safe guards.  We could have the bot checkout the test directory at a given revision or tag.  That would also solve the first problem.
Yeah, I don't think we want to do any sort of automatically updating test suite from Firebug. Starting with taking known-good drops seems like the way to go. Both for simplicity reasons, and the firebug-shouldn't-break-firefox reason you mention.

I'd love to hear from releng what's involved with adding additional test suites though. In order to avoid this bug just being wishful thinking :)
I have a user-repo of work I started on this here:

http://hg.mozilla.org/users/rcampbell_mozilla.com/fbtestmaster/

currently linux-only and I believe the semantics for running FBTests have changed slightly since that implementation. Should be a good starting point for integration however.
(In reply to comment #5)
> I have a user-repo of work I started on this here:
> 
> http://hg.mozilla.org/users/rcampbell_mozilla.com/fbtestmaster/
> 
> currently linux-only and I believe the semantics for running FBTests have
> changed slightly since that implementation. Should be a good starting point for
> integration however.
This is a great help to us, and we can probably leverage a bunch of your code in combination with a lot of existing code to put this together pretty easily.  I'm thinking we can reuse the existing buildbot code for downloading/installing the builds.  We can use Mozrunner (the runner code that is used to create profiles and start the application for mozmill, thunderbird and jetpack testing) to create the profiles and start the browser.   

Essentially, once we figure out the hard part of generalizing the up and coming solution in bug 516984, this would not be too hard (technically speaking) to "buildbot-ize" and deploy in conjunction with RelEng.
(In reply to comment #6)
> (In reply to comment #5)
> > I have a user-repo of work I started on this here:
> > 
> > http://hg.mozilla.org/users/rcampbell_mozilla.com/fbtestmaster/
> > 
> > currently linux-only and I believe the semantics for running FBTests have
> > changed slightly since that implementation. Should be a good starting point for
> > integration however.
> This is a great help to us, and we can probably leverage a bunch of your code
> in combination with a lot of existing code to put this together pretty easily. 

hopefully!

> I'm thinking we can reuse the existing buildbot code for downloading/installing
> the builds.  We can use Mozrunner (the runner code that is used to create
> profiles and start the application for mozmill, thunderbird and jetpack
> testing) to create the profiles and start the browser.

Sounds good. You might want to check with releng about the downloader code. I cooked that up for the original talos stuff a few years back and I'm pretty sure it's been deprecated/replaced.

Then again if it serves the purpose, feel free to keep it. :)

> Essentially, once we figure out the hard part of generalizing the up and coming
> solution in bug 516984, this would not be too hard (technically speaking) to
> "buildbot-ize" and deploy in conjunction with RelEng.

awesome! I hope this helps.
So, Andrew has done the following:
1. Written a script to pull down the firebug XPIs and firebug tests from getfirebug.com and Google Code
2. Crafted a way to run the firebug tests from the command line
3. Honza is working on a way to export the firebug test results into a format buildbot can understand.

= Status =
Item 1 is done.
Item 2 is 90% done.
Item 3 is in progress, I don't know the complete status.  Honza, can you update?

= The Plan =
== The Server ==
* Install the script from item 1 onto a web server inside the colo. (I'm sure we can find a machine - it's small and the tests are small).
* The script will run in chron once a day, pulling down the various versions of firebug.
* The script downloads the firebug xpi's and tests and hosts them on the webserver.

== The Buildbot Slave Step ==
* It downloads the firebug xpi for the version of firefox it is testing (might be one or more firebug version per version of firefox)
* Using the mozrunner abstraction from bug 516984, we run a script (already written) with a single commandline to call into mozrunner to register the firebug.xpi as well as the firebugtest.xpi.  Then it starts firefox and runs firebug tests. 
* Output occurs to the console now, but it is not in a format that buildbot will understand (that's waiting on part 3).

= The Rationale =
* We don't want buildbot slaves depending on external network components - this way, we pull down the firebug code once a day to a web server we control.  If we lose connection to the external resources, we'll still have a copy to test against (it might be old, but it will be there) and we won't burn any trees.
* We wanted to make the buildbot impact very light - once the mozrunner code lands in m-c in bug 516984 (as part of the larger mozmill codebase), this becomes a very simple script for buildbot to call.  If needed, we can remove this dependency, but it was the fastest, simplest way to write this.
 
We'll start attaching WIP patches soon.
I actually don't think we should be updating the Firebug XPI and test suite automatically on a daily basis. What would we do if a bug arises in firebug or its test suite? Would that turn tinderbox perma orange with no means of recourse for 24 hours?

It seems much safer that we pull from known good points, just like we do when we update other libraries that we have in our code and/or test suite.
I believe the plan is daily download of known good points.
Changing the known point will require attention to the test results.We need to work out how to do that.
Assignee: nobody → ahalberstadt
(In reply to comment #8)
> So, Andrew has done the following:
> 1. Written a script to pull down the firebug XPIs and firebug tests from
> getfirebug.com and Google Code
> 2. Crafted a way to run the firebug tests from the command line
> 3. Honza is working on a way to export the firebug test results into a format
> buildbot can understand.
> 
> = Status =
> Item 1 is done.
> Item 2 is 90% done.
> Item 3 is in progress, I don't know the complete status.  Honza, can you
> update?
FBTest export is done.

Here is an example of a log:

FIREBUG INFO | Firebug: 1.6X.0a9
FIREBUG INFO | FBTest: 1.6b12
FIREBUG INFO | App Name: Firefox
FIREBUG INFO | App Version: 3.6.5pre
FIREBUG INFO | App Platform: 1.9.2.5pre
FIREBUG INFO | App Build ID: 20100505042444
FIREBUG INFO | Locale: en-US
FIREBUG INFO | OS Name: Windows_NT
FIREBUG INFO | OS Version: 6.0
FIREBUG INFO | Export Date: Thu, 06 May 2010 12:02:52 GMT
FIREBUG INFO | Test Suite: http://getfirebug.com/.../firebug1.6.html
FIREBUG INFO | Total Tests: 122

FIREBUG INFO | console/log.html | [START] Console API log method.
FIREBUG TEST-PASS | console/log.html | [DONE]

FIREBUG INFO | Test Suite Finished: Thu, 06 May 2010 12:07:06 GMT
FIREBUG INFO | Passing: 1 FIREBUG INFO | Failing: 0

The log is stored into a file (after all tests are finished) that is located within the current Firefox profile.

Here is a syntax description of the log:
<log-row> := <testname><space><RESULT_TYPE>|<message>
<testname> := FIREBUG
<message> := <fbtest-name>|<message>
<RESULT_TYPE> := INFO | TEST-PASS | TEST-KNOWN-FAIL | TEST-UNEXPECTED-FAIL


I have also built a new version of the FBTest harness, it's available here:
http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b13.xpi

The log is created only if tests (Firefox) are launched from the command line.

Example: 
firefox.exe -P test001 -no-remote -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.6.html

Log dir in windows:
C:\Users\Honza\AppData\Roaming\Mozilla\Firefox\Profiles\<profile-id>\firebug\fbtest\logs\

Honza
(In reply to comment #10)
> I believe the plan is daily download of known good points.
> Changing the known point will require attention to the test results.We need to
> work out how to do that.

Yes, I also think that the download should be done from specified revisions. So, changes (and possible bugs) don't break the testing.

Possible solution could be:

1) Download latest Firebug + FBTest combo release (e.g. Firebug 1.6a10 which is always tied to specific revision), which is provided by Firebug team and available online on getfirebug.com. In case the test-bot script wants to download the source code from SVN instead (and use it directly), it should be done from the same revision point.

2) Download all test files from the same revision point.

3) Run tests.

Does this make sense?

Honza
What I meant was: 
  We need to design a procedure for updating the known-good revision points so that failures in that update do not trigger reports to the rest of the team.

I think the dev team has a established process for committing code and watch that your commit was "green". We need to adapt it for updating this test.
(In reply to comment #12)
> Possible solution could be:
> 
> 1) Download latest Firebug + FBTest combo release (e.g. Firebug 1.6a10 which is
> always tied to specific revision), which is provided by Firebug team and
> available online on getfirebug.com. In case the test-bot script wants to
> download the source code from SVN instead (and use it directly), it should be
> done from the same revision point.

Currently the script downloads the Firebug extension from getfirebug.com and the FBTest extension from SVN (as per your suggestion).  I didn't realise that they were meant to be run with a specific revision.  Is there any way to obtain the revision from the getfirebug.com/releases/firebug directory?

Also fyi, the script currently downloads all (latest) major versions of the Firebug and FBTest extensions.  Then you can specify which version you wish to run the tests on later via a configuration file or a command line parameter
Rather than automating finding good pulls, why not make it simple for ourselves at first and just do manual pulls every so often. It'd be nice to start simple until we know that this works well.
(In reply to comment #14)
> Currently the script downloads the Firebug extension from getfirebug.com and
> the FBTest extension from SVN (as per your suggestion).
FBTest should be also downloaded as xpi. 

So, you need to download three things:
1) Firebug extension (*.xpi) from http://getfirebug.com/releases/firebug/1.6X/
2) FBTest extension (*.xpi) from http://getfirebug.com/releases/fbtest/1.6/
3) Test files (*.html, *.js, ...) from SVN, specific revision

Note that, FBTest extension represents Firebug test harness that is responsible for loading 'Test files' and executing them.

> I didn't realise that they were meant to be run with a specific revision.
> Is there any way to obtain the revision from the
> getfirebug.com/releases/firebug directory?
Currently no.

What about to create a config file (located on getfirebug.com), which would specify all these three things you need to download. It's content would be something as like follows:

FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a9.xpi
FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b9.xpi
SVN_REVISION=6025

- The test-bot script can read this file and see what to download.
- As soon as Firebug team releases new Firebug version, it also updates this file with new info.

> Also fyi, the script currently downloads all (latest) major versions of the
> Firebug and FBTest extensions.  Then you can specify which version you wish to
> run the tests on later via a configuration file or a command line parameter
The command line parameter could be URL of the config file on getfirebug.com. This way allows to specify a different config file, in cases when an older version of Firebug should be tested. However, not sure how Firebug team is supposed to alter the command line...

Honza
(In reply to comment #16)
> What about to create a config file (located on getfirebug.com), which would
> specify all these three things you need to download. It's content would be
> something as like follows:
> 
> FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a9.xpi
> FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b9.xpi
> SVN_REVISION=6025
> 
> - The test-bot script can read this file and see what to download.
> - As soon as Firebug team releases new Firebug version, it also updates this
> file with new info.
This might be the cleanest path to take.  If I understand you correctly, then each major release of Firebug would have its own config file in the appropriate directory (rather than one master in the root).

> The command line parameter could be URL of the config file on getfirebug.com.
> This way allows to specify a different config file, in cases when an older
> version of Firebug should be tested. However, not sure how Firebug team is
> supposed to alter the command line...
To clarify there are currently two scripts:
- One which downloads all the required files and hosts them on a local webserver
- Another which runs the FBTests

We're trying to avoid accessing getfirebug.com when running the tests.  I don't know if testing older versions of Firebug is something that people are interested in or not, but if it is.. the download script could just update all versions at the same time, ensuring we always have the most recent extensions and test files of each version.
(In reply to comment #17)
> (In reply to comment #16)
> > What about to create a config file (located on getfirebug.com), which would
> > specify all these three things you need to download. It's content would be
> > something as like follows:
> > 
> > FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a9.xpi
> > FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b9.xpi
> > SVN_REVISION=6025
> > 
> > - The test-bot script can read this file and see what to download.
> > - As soon as Firebug team releases new Firebug version, it also updates this
> > file with new info.
> This might be the cleanest path to take.  If I understand you correctly, then
> each major release of Firebug would have its own config file in the appropriate
> directory (rather than one master in the root).
It depends on what is more suitable for you. But note that, for example, all Firebug 1.6 releases (all alpha and beta XPIs files) are located in the same directory:
http://getfirebug.com/releases/firebug/1.6X/

So, in case of more config files (one config file per released firebug xpi), we would have to use different file names. For instance:

For "firebug-1.6X.0a1.xpi" release there would be "firebug-1.6X.0a1.config" file.

In such case the test-bot script needs to figure out what is the latest config file. Also, when we finish Firebug 1.6 and start on 1.7 all releases go into:
http://getfirebug.com/releases/firebug/1.7X/

So, the test-bot script needs to be updated to reflect the new location (but correct me if a new location is not a problem).

I think that the situation is simpler in case of just one config file (with fixed URL).
E.g.: http://getfirebug.com/releases/firebug/test-bot.config

In this case the test-bot script loads always this file (updated regularly by Firebug team upon releasing a new version), and downloads necessary files from proper locations.

This way, the script has always up to date info where to find the right files and doesn't depend on Firebug's team decisions about new XPI locations.

> > The command line parameter could be URL of the config file on getfirebug.com.
> > This way allows to specify a different config file, in cases when an older
> > version of Firebug should be tested. However, not sure how Firebug team is
> > supposed to alter the command line...
> To clarify there are currently two scripts:
> - One which downloads all the required files and hosts them on a local
> webserver
> - Another which runs the FBTests
I see

> We're trying to avoid accessing getfirebug.com when running the tests.
Great, that's correct. These online files are automatically synced with FB SVN and so changing continuously as we are committing patches.
  
> I don't know if testing older versions of Firebug is something that people are
> interested in or not, but if it is..
I have been also thinking about this, but not sure how this would work.

My current view on the test-bot process is as follows:

- The script reads latest test-bot.config file and downloads all necessary files for testing.
- The other script runs FBTests, gets the output log, parses and publishes results.
- This is done every day using Firefox nightly build.

Btw. how Firebug team will be notified about test results?

So, the same (latest) version of Firebug is tested with every new Firefox nightly, until the test-bot.config file is updated and new Firebug release (with updated test harness and files) comes into the place (which can happen, approximately once in couple of weeks).

Now let's say that we (FB team) want to test older version (e.g. Firebug 1.5.4). What should actually happen? Of course we could just update the master config file to point to the older version and so, force the test bot to download it. But yes this would break regular testing of the latest version.

Can we instruct the test-bot to launch another process that uses another configuration and runs other tests in parallel? Note that in case of FB 1.5 we want to test with Firefox 3.6 not 3.7 so, this would have to be also reflected in the config file.

I would keep it simple at first, see how it works (testing only the latest Fb with Fx nightly) and we can expand later. The main goal here is to check compatibility between latest Fb and Fx.

> the download script could just update all
> versions at the same time, ensuring we always have the most recent extensions
> and test files of each version.

---

One addition. As I mentioned, FBTest extension needs one input parameter, which is URL of the test-list file (*.html, list of tests to run). This file is part of the SVN (included in the same directory as all test files).

#svn/tests/content - all test files
#svn/tests/content/testlists/firebug1.6.html

So, since the script is downloading all test files using given revision, it'll also have up to date test-list file.

There is always one test-list file per Firebug branch (so for Firebug 1.7, there will be #svn/tests/content/testlists/firebug1.7.html).

You need to compute the final URL (passed into FBTest) and reflect both the firebug branch version and URL of your server. But I think you already know this.

Honza
(In reply to comment #18)
> For "firebug-1.6X.0a1.xpi" release there would be "firebug-1.6X.0a1.config"
> file.
Sorry for the confusion.  I meant having one config file for each directory,
i.e "firebug-1.6.config", but your method of keeping all of this in one main
config file works too and probably makes more sense.  I'm just looking for
information on the most recent release of each directory (i.e "1.4", "1.5", "1.6" etc...)

An example config file:
[Firebug1.6]
FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a9.xpi
FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b9.xpi
SVN_REVISION=6025
[Firebug1.5]
FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.5.4.xpi
FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.5b3.xpi
SVN_REVISION=5840
[Firebug1.4]
etc...

With this information, we can download and maintain the extensions and test
files for each major release locally (i.e the most recent in each directory).
Then testing an older version of Firebug is as simple as passing a parameter
into the test runner script.

Note: Currently my script already parses the extension file names to retrieve
the one with the highest version number, but by providing the names in the
config file it would ensure we don't accidentally grab the wrong one.

> In such case the test-bot script needs to figure out what is the latest config
> file. Also, when we finish Firebug 1.6 and start on 1.7 all releases go into:
> http://getfirebug.com/releases/firebug/1.7X/
> 
> So, the test-bot script needs to be updated to reflect the new location (but
> correct me if a new location is not a problem).
This is not a problem.  The script already iterates through the parent
directory to retrieve a complete list of version numbers.

> I think that the situation is simpler in case of just one config file (with
> fixed URL).
> E.g.: http://getfirebug.com/releases/firebug/test-bot.config
Agree.  Sorry for the miscommunication.

> I have been also thinking about this, but not sure how this would work.
> 
> My current view on the test-bot process is as follows:
> 
> - The script reads latest test-bot.config file and downloads all necessary
> files for testing.
> - The other script runs FBTests, gets the output log, parses and publishes
> results.
> - This is done every day using Firefox nightly build.
Correct.  My intention is to have the download script maintain older releases
of Firebug as well, just in case someone wants to test it (presumably with an
older release of FF)

> Btw. how Firebug team will be notified about test results?
Clint can probably answer this better than me, but my impression is that the
build bot expects logging info to be written to stdout, so all we have to do is
send your log file there.

> Now let's say that we (FB team) want to test older version (e.g. Firebug
> 1.5.4). What should actually happen? Of course we could just update the master
> config file to point to the older version and so, force the test bot to
> download it. But yes this would break regular testing of the latest version.
> 
> Can we instruct the test-bot to launch another process that uses another
> configuration and runs other tests in parallel? Note that in case of FB 1.5 we
> want to test with Firefox 3.6 not 3.7 so, this would have to be also reflected
> in the config file.
Yes.  This is answered above.

> I would keep it simple at first, see how it works (testing only the latest Fb
> with Fx nightly) and we can expand later. The main goal here is to check
> compatibility between latest Fb and Fx.
I agree.  However I don't see any harm in downloading and maintaining older
versions of Firebug for the moment, so long as the test runner script only runs
1.6X for now.

> You need to compute the final URL (passed into FBTest) and reflect both the
> firebug branch version and URL of your server. But I think you already know
> this.
Yep. As of now everything is working.  I can download and run the tests without
issue.  The only thing I need to do next is ensure that the correct SVN
revision is being checked out for the corresponding Firebug release.

Let me know if anything I said isn't quite clear,
-ahal
(In reply to comment #19)

> An example config file:
> [Firebug1.6]
> FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a9.xpi
> FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b9.xpi
> SVN_REVISION=6025
> [Firebug1.5]
> FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.5.4.xpi
> FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.5b3.xpi
> SVN_REVISION=5840
> [Firebug1.4]
> etc...
I like this.

> With this information, we can download and maintain the extensions and test
> files for each major release locally (i.e the most recent in each directory).
> Then testing an older version of Firebug is as simple as passing a parameter
> into the test runner script.
Sounds good.

> Correct.  My intention is to have the download script maintain older releases
> of Firebug as well, just in case someone wants to test it (presumably with an
> older release of FF)
Should we add an info about what Firefox version should be used with specific Firebug version? So the script knows what FF version to actually run?

The current support is as follows:
Firebug 1.4 is for Firefox 3.0 and 3.5
Firebug 1.5 is for Firefox 3.5 and 3.6
Firebug 1.6 is for Firefox 3.6 and ...


> > Btw. how Firebug team will be notified about test results?
> Clint can probably answer this better than me, but my impression is that the
> build bot expects logging info to be written to stdout, so all we have to do > is send your log file there.
Yes.
I meant how FB team detect that there are actually some tests failing? What online page we should regularly check out to see test-bot results (and also the log from FBTest)? Or do we get an automated email with results?


Let me know as soon as you need the config file on getfirebug.com and I'll commit it there.

Not sure about the config file name (test-bot.config?), but here is real info for Firebug 1.6

[Firebug1.6]
FIREBUG_XPI=http://getfirebug.com/releases/firebug/1.6X/firebug-1.6X.0a10.xpi
FBTEST_XPI=http://getfirebug.com/releases/fbtest/1.6/fbTest-1.6b13.xpi
SVN_REVISION=6715

Honza
(In reply to comment #20)
> Should we add an info about what Firefox version should be used with specific
> Firebug version? So the script knows what FF version to actually run?
I've already been asked how we'll know this, so I like this idea.  Maybe it could be a comma separated list in the config file?
i.e. 
[Firebug1.5]
FIREFOX_VERSION=3.5,3.6

> I meant how FB team detect that there are actually some tests failing? What
> online page we should regularly check out to see test-bot results (and also the
> log from FBTest)? Or do we get an automated email with results?
Oops, misread that as Firefox team.  What are your preferences?  I'm sure it wouldn't be hard to send an automated e-mail with the log file as an attachment if you want.  Then again I may not be the best person to ask about this.

> Let me know as soon as you need the config file on getfirebug.com and I'll
> commit it there.
Whenever you have some spare time would be great!  (As for name test-bot.config works for me)

Thanks,
-Ahal
> [Firebug1.5]
> FIREFOX_VERSION=3.5,3.6
I like it


> Oops, misread that as Firefox team.  What are your preferences?  I'm sure it
> wouldn't be hard to send an automated e-mail with the log file as an attachment
> if you want.  Then again I may not be the best person to ask about this.

Clint what is your proposal on this?


> > Let me know as soon as you need the config file on getfirebug.com and I'll
> > commit it there.
> Whenever you have some spare time would be great!  (As for name test-bot.config
> works for me)
Done, the file is online here:
https://getfirebug.com/releases/firebug/test-bot.config

Honza
Status: NEW → ASSIGNED
Ahal, is there anything I could help with yet? Or are you successful already with the automation? 

Honza
(In reply to comment #22)
> > [Firebug1.5]
> > FIREFOX_VERSION=3.5,3.6
> I like it
> 
> 
> > Oops, misread that as Firefox team.  What are your preferences?  I'm sure it
> > wouldn't be hard to send an automated e-mail with the log file as an attachment
> > if you want.  Then again I may not be the best person to ask about this.
> 
> Clint what is your proposal on this?
> 
This thing is going to be reporting to the Tinderbox Push Log with the other
tests.  Buildbot has the ability to mail results to various places.  I know
that it can mail out results to a fixed newsgroup address (see the
mozilla.dev.tree-management newsgroup as an example).  So, it may be possible
to conditionally mail a firebug newsgroup if a test fails so that you do not
have to always watch tbpl which could easily become a full time job.

Chris Atlee (catlee) is the guy who set up the automated email for the
tree-management group, and so he probably knows whether or not this is possible
here.  I think at the very least, we could have it mail the tree-management
newsgroup, then you and the other firebug folks could watch that group with a
filter to throw away everything except the "Firebug" messages.  But ideally, it
could be set up to mail your own group as well.
> 
> > > Let me know as soon as you need the config file on getfirebug.com and I'll
> > > commit it there.
> > Whenever you have some spare time would be great!  (As for name test-bot.config
> > works for me)
> Done, the file is online here:
> https://getfirebug.com/releases/firebug/test-bot.config
Awesome, you guys are doing great stuff here.  So, it looks like we've come to
the point where we are ready to integrate this into BuildBot.  Chris, what do
you need from us?  Here's what I imagine we'll need to do:
1. Install Andrew's "downloader" script onto a box with a webserver on it
inside the colo someplace.  I can handle that.  We can tune the frequency of
the downloading to our liking.  Maybe we start that manual, maybe it's once a
month or something like that.
2. Create a buildbot step for firebug tests.  This step will do the following:
** Download the firebug runner script from the webserver in point 1.
** Download the proper firebug extension and fbtest extension from the
webserver in point 1 (or perhaps this is something our "runner" script could
handle?)
** Call the runner script to run the tests
** pull the information off of stdout, upload to tbpl (automatic)
** shoot off an email to some newsgroup if the tests fail.

How does this sound?  Do you want me to make a buildbot step patch for this or
is this something you guys can do?
Currently the script can be found here: http://github.com/ahal/Firebug-Test-Runner 

In order to use the script you have to do a few things.

1) Set up a local directory webserver (no index file)
2) Configure settings in "fb-test-runner.config"
  - [update]->serverpath: should be set to the server directory in your file system (i.e /var/www/)
  - [run]->serverpath: should be set to the web address the above path corresponds to (e.g http://localhost/)
  - [run]->profile: for now this needs to be set to the Firefox default profile (I'm planning to fix this to be used with any profile, but I've been busy with some other things).

Please note, despite having the correct revision checked out and extensions installed, running the FBTests from my local server still results in several additional failures as opposed to running them off of "http://getfirebug.com" (this happens when running them manually as well, i.e is not caused by the script).  I don't know if this is an artifact of the way my server is configured or not.

In addition, even when running them manually off of "http://getfirebug.com" there are still a handful of tests that fail. This means that integrating it into the build system at this point in time will cause a flag.
> Chris Atlee (catlee) is the guy who set up the automated email for the
> tree-management group, and so he probably knows whether or not this is possible
> here.  I think at the very least, we could have it mail the tree-management
> newsgroup, then you and the other firebug folks could watch that group with a
> filter to throw away everything except the "Firebug" messages.
Sounds good to me.

> But ideally, it could be set up to mail your own group as well.
Yes, I think that the results should be sent to FWG group (this is a less public group used only Firebug developers).
http://groups.google.com/group/firebug-working-group

> How does this sound?  Do you want me to make a buildbot step patch for this or
> is this something you guys can do?
Anyone already assigned on this? (I am probably not the right one to do it).

Honza
> Please note, despite having the correct revision checked out and extensions
> installed, running the FBTests from my local server still results in several
> additional failures as opposed to running them off of "http://getfirebug.com"
> (this happens when running them manually as well, i.e is not caused by the
> script).  I don't know if this is an artifact of the way my server is
> configured or not.
Can you post an info about which particular tests fail for you? Just right click within the test-console (FBTest) UI and pick "Copy All Errors", this will copy report into the clipboard.

> In addition, even when running them manually off of "http://getfirebug.com"
> there are still a handful of tests that fail.
What are your results in this case?

We need to be able to reproduce these failures in order to fix them.

I just ran tests on the same configuration as specified here:
https://getfirebug.com/releases/firebug/test-bot.config

And here are my results:

Firefox 3.6:
[FAILED] commandLine/debug.js: Verify command line debug API.

Firefox 3.7:
[FAILED] script/debuggerKeyword/testDriver.js: Break on debugger keyword in various contexts.
[FAILED] console/2271/issue2271.js: Issue 2271: JS errors in AJAX callback functions are not shown
[FAILED] console/2694/issue2694.js: Issue 2694: Console output does not scroll to position on 1.6a2
[FAILED] console/spy/2285/issue2285.js: support for content-type: multipart/x-mixed-replace
[FAILED] commandLine/debug.js: Verify command line debug API.

All known problems that wait for fixes in Firefox (there are proper bugzilla reports for them).

Honza
(In reply to comment #24)
...
> This thing is going to be reporting to the Tinderbox Push Log with the other
> tests.  Buildbot has the ability to mail results to various places.  I know
> that it can mail out results to a fixed newsgroup address (see the
> mozilla.dev.tree-management newsgroup as an example).  So, it may be possible
> to conditionally mail a firebug newsgroup if a test fails so that you do not
> have to always watch tbpl which could easily become a full time job.
> 
> Chris Atlee (catlee) is the guy who set up the automated email for the
> tree-management group, and so he probably knows whether or not this is possible
> here.  I think at the very least, we could have it mail the tree-management
> newsgroup, then you and the other firebug folks could watch that group with a
> filter to throw away everything except the "Firebug" messages.  But ideally, it
> could be set up to mail your own group as well.

Ultimately we need a way to communicate between the Firebug and Firefox/Platform teams about the source of problems. So if the readers of tree-management are ok with adding a stream labeled with Firebug, I think that is a good place to start. (I think the FWG may be not the best for too many routine messages; we may need a new testing-error message group).
Attached file Firebug test results —
Honza:

Here is the output from the tests when running from both getfirebug.com and a local server.  I'm sure it's just some weird configuration I have for my server.  The local server is using everything specified in "test-bot.config"

Note: I'm checking out from the read-only version of the repository, but that shouldn't make any difference.

Thanks
(In reply to comment #29)
> Created an attachment (id=445793) [details]
> Firebug test results
> 
> Honza:
> 
> Here is the output from the tests when running from both getfirebug.com and a
> local server.  I'm sure it's just some weird configuration I have for my
> server.  The local server is using everything specified in "test-bot.config"
I see two potential problems:

1) Firefox locale is set to "chrome://global/locale/intl.properties". I don't quite understand what this means. Can you set the locale to en-US?

2) It looks like the .htaccess doesn't work properly. Let's see for example the net/176/issue176.html test.

If you click the "Execute Test" button, two HTTP requests are executed.
The first is issue176.txt. Content-Type of this request must be set to: "video/x-flv", even if the file extension is ".txt". The .htaccess is there to ensure this. Can you verify?

> Note: I'm checking out from the read-only version of the repository, but that
> shouldn't make any difference.
Yes this shouldn't be a problem.

Honza
> (I think the FWG may be not the best for too many
> routine messages; we may need a new testing-error message group).
OK, we can discuss this tomorrow at Firebug's conf call.
Honza
(In reply to comment #30)
> 1) Firefox locale is set to "chrome://global/locale/intl.properties". I don't
> quite understand what this means. Can you set the locale to en-US?
As far as I can tell my locale is en-US.  After some brief Googling, it seems that Gecko grabs the locale from this file.  Also the value of 'general.useragent.locale' is set to 'en-US' in about:config.

> 2) It looks like the .htaccess doesn't work properly. Let's see for example the
> net/176/issue176.html test.
It turns out I had 'mod_rewrite' disabled for my server.  Upon enabling this, issue176 now passes, but the rest of the tests still fail.

If there's anything else I can do, or you have any other ideas I'd love to try it out.  Otherwise, seeing as this is almost definitely a configuration issue, it might make more sense to wait until this is being run on the actual test machine before worrying about it too much.

-Ahal
> It turns out I had 'mod_rewrite' disabled for my server.  Upon enabling this,
> issue176 now passes, but the rest of the tests still fail.
Great

> If there's anything else I can do, or you have any other ideas I'd love to try
> it out.  Otherwise, seeing as this is almost definitely a configuration issue,

Lets see the "net/2209/issue2209-2.js" now. I tend to think that this failures is also related to .htacess. If you open the test page and click the button a XHR is executed. Check the response Content-Type, it must be set to "text/js" (which is done by header() function, within issue2209-2.php). I bet the content type is not set.

Next one is net/601/issue601.js. This one also sets header() but shouldn't fail even it the Content-Type is different. Please check the response in this case, I would be curious why it's different. Perhaps different line endings or page encoding? (which I think could be also the problem for other tests).

> it might make more sense to wait until this is being run on the actual test
> machine before worrying about it too much.
Yes, agree. I think we should put everything in the place and make sure that the entire automation process works. Consequently we can fix all configuration problems (or problems with tests) step by step.

This can also show how the process of fixing real Firebug failures will actually look like, which is related to John's comment #28.

Is there any way for me to load the test-page from the same server/configuration that is used by test-bot? Then I can verify manually with my Firefox+Firebug combo.


> Ultimately we need a way to communicate between the Firebug and
> Firefox/Platform teams about the source of problems. So if the readers of
> tree-management are ok with adding a stream labeled with Firebug,
> I think that is a good place to start. 
I personally agree. Chris, is this the way to go?

So, here is what I think could happen in the future:

1) A Firebug test fails.
2) There is a message sent to tree-management group labeled with Firebug.
3a) FWG team runs the test manually and finds the problems in the test itself of in Firebug. FWG fixes it.
3b) FWG team runs the test manually and finds the problem in Firefox. FWG is passing the failing test (perhaps with more info) to somebody from Firefox/Platform teams for further investigation. Firefox/Platform fixes it.
3c) FWG team runs the test manually and it works using getfirebug.com. FWG is passing the failing test to...?

Or what should be the right process here?

Honza
I've update the code at: http://github.com/ahal/Firebug-Test-Runner 

The log file now gets copied over to stdout (as a stream) and I've added a bunch of additional error checking. There are also steps to configure the web server the tests need to be hosted on in the README file.

I'm waiting for confirmation before integrating fb-run.py with build-bot and we'll need to figure out how to periodically run fb-update.py.

Let me know if you have any comments/suggestions/things you'd like to see.
(In reply to comment #34)
> I've update the code at: http://github.com/ahal/Firebug-Test-Runner 
> 
> The log file now gets copied over to stdout (as a stream) and I've added a
> bunch of additional error checking. There are also steps to configure the web
> server the tests need to be hosted on in the README file.
> 
> I'm waiting for confirmation before integrating fb-run.py with build-bot and
> we'll need to figure out how to periodically run fb-update.py.
Sounds great!

So, if I understand correctly, the problem with failing tests is fixed?

> Let me know if you have any comments/suggestions/things you'd like to see.
The only remaining question is the process (also described in comment #33) when a failing tests appears. What would be really useful for FWG is a page with information who to contact with specific problem.

Honza
> So, if I understand correctly, the problem with failing tests is fixed?
Cool, I have just launched them all and the only problem I see is with
console/2160/issue2160.js (there is apparently problem with the test).
I'll fix it.
Honza
I have updated test-bot config file:
https://getfirebug.com/releases/firebug/test-bot.config
to use Firebug 1.6a11 (released last week).

Any news on the fb-run.py & build-bot integration?

Honza
I've setup the webserver we will be using at "http://10.2.73.186/" (must VPN into the colo first).

To run these with the script see Comment 25 and the README file.  Set [run]->serverpath to "http://10.2.73.186/" in "fb-test-runner.config".
1) 

2) I recommend checking in the config files, 


(In reply to comment #38)
> I've setup the webserver we will be using at "http://10.2.73.186/" (must VPN
> into the colo first).
> 
> To run these with the script see Comment 25 and the README file.  Set
> [run]->serverpath to "http://10.2.73.186/" in "fb-test-runner.config".

Instead of adding a dependency to the new 10.2.73.186 webserver, I'd prefer this new "firebug" unittest suite use the local webserver already installed on every machine which runs unittest/talos suites. 

We have machines in multiple locations, concurrently testing different versions of firefox. It would be better if we could not have to worry about concurrent testing of different firefox branches, load across WAN pipes between colos, failover, scaling of this one new webserver to handle production load, etc, etc.


Ctalbert had details of what to use instead of 127.0.0.1 in order to make this work best with mobile too, if thats of interest.
1) I think the Firebug being tested, and the FBtest and test-bot.config being used should all be checked in. The test setup should get all of those from repo and install them on test machine. The test breakdown should remove them from the machine.

This feels important, because the tricky "what changed in that test" discussions I've seen are not about the last few days, when events are still fresh - its about what changed months ago, in the weeks between two firedrill releases. For that historical audit, checking in the files has been most robust in our work with both unittests and talos.


2) If we just think of this as a new "firebug" unittest suite, we could queue this to be run when other unittest suites are run (on checkin, on nightly, on release builds). We could also get it displayed on Tinderbox waterfalls, on TBPL, and it would be presented in a format that developers would immediately understand.


3) I'd like to suggest changing the summary of this bug. To me, this bug is about "Setup new firebug test suite so it can be automated". Once the scope of suite, and setup/breakdown steps are debugged, and its ready to run to testrun on a RelEng staging machine, please file a separate bug "automate running new firebug test suite" in mozilla.org/ReleaseEngineering.

Hope that helps.
John, is #2 is bug 532445?
(In reply to comment #39)
> 1) 
> 
> 2) I recommend checking in the config files, 
> 
> 
> (In reply to comment #38)
> > I've setup the webserver we will be using at "http://10.2.73.186/" (must VPN
> > into the colo first).
> > 
> > To run these with the script see Comment 25 and the README file.  Set
> > [run]->serverpath to "http://10.2.73.186/" in "fb-test-runner.config".
> 
> Instead of adding a dependency to the new 10.2.73.186 webserver, I'd prefer
> this new "firebug" unittest suite use the local webserver already installed on
> every machine which runs unittest/talos suites. 
> 
That server is qm-codecover02 which is already being referenced by buildbot for the code coverage builds.  There is no webserver installed with talos as far as I know: it uses file:// URLs.  Those will not work here with firebug due to security restrictions in pages served via file:// URLs (we tried).

We selected this server because it already has uptime restrictions (if it goes down, buildbot breaks).  This does put an extra restriction on the server, but it's pretty easy to put this on any server you'd like, so pick a machine.

> We have machines in multiple locations, concurrently testing different versions
> of firefox. It would be better if we could not have to worry about concurrent
> testing of different firefox branches, load across WAN pipes between colos,
> failover, scaling of this one new webserver to handle production load, etc,
> etc.
These tests are available 24x7, on getfirebug.com.  Any person on the planet can run them, and it hasn't brought down that server.  We're talking about a rather small load on a production (Apache) webserver.

Do you seriously want us to hack this into httpd.js and serve them locally?  I'd like to move away from that dependency.  While httpd.js is great when you need a webserver that can do interesting backflips for unit tests, I'd much rather us use a standard webserver for general-purpose testing that doesn't need to be local. 
> 
> Ctalbert had details of what to use instead of 127.0.0.1 in order to make this
> work best with mobile too, if that's of interest.
And for mobile as well, we've also discussed moving away from httpd.js eventually.  I think we should really think about this one more time.
(In reply to comment #40)
> 1) I think the Firebug being tested, and the FBtest and test-bot.config being
> used should all be checked in. The test setup should get all of those from repo
> and install them on test machine. The test breakdown should remove them from
> the machine.
> 
> This feels important, because the tricky "what changed in that test"
> discussions I've seen are not about the last few days, when events are still
> fresh - its about what changed months ago, in the weeks between two firedrill
> releases. For that historical audit, checking in the files has been most robust
> in our work with both unittests and talos.
> 
That's been our idea all along to check them in.  The question is where to put them.  They don't necessarily need to live in mozcentral, but they can.  My thought there was that there is no need for developers to run these tests on their boxes.  Developers can (much more easily) run the standard getfirebug.com tests.  I thought it might be easier to have the buildbot slaves download the materials they needed instead of putting them in m-c.  But, whatever, we can do whichever is easiest.
> 
> 2) If we just think of this as a new "firebug" unittest suite, we could queue
> this to be run when other unittest suites are run (on checkin, on nightly, on
> release builds). We could also get it displayed on Tinderbox waterfalls, on
> TBPL, and it would be presented in a format that developers would immediately
> understand.
> 
That's the entire point here.  Yes.
> 
> 3) I'd like to suggest changing the summary of this bug. To me, this bug is
> about "Setup new firebug test suite so it can be automated". 
That's what it was.  That has now been done.

> Once the scope of
> suite, and setup/breakdown steps are debugged, and its ready to run to testrun
> on a RelEng staging machine, please file a separate bug "automate running new
> firebug test suite" in mozilla.org/ReleaseEngineering.
>
It's ready now.  We need your help with writing the buildbot patch for this or letting us know how you'd like that patch created. 
> Hope that helps.
Yep, it does.  Thanks.
Sorry, but just FYI we can't use httpd.js with firebug: see  Bug 497799, basically sync code on one thread can't read and write itself.
(In reply to comment #43)
> (In reply to comment #40)
> > 1) I think the Firebug being tested, and the FBtest and test-bot.config being
> > used should all be checked in. The test setup should get all of those from repo
> > and install them on test machine. The test breakdown should remove them from
> > the machine.
> > 
> > This feels important, because the tricky "what changed in that test"
> > discussions I've seen are not about the last few days, when events are still
> > fresh - its about what changed months ago, in the weeks between two firedrill
> > releases. For that historical audit, checking in the files has been most robust
> > in our work with both unittests and talos.
> > 
> That's been our idea all along to check them in.  The question is where to put
> them.  They don't necessarily need to live in mozcentral, but they can.  My
> thought there was that there is no need for developers to run these tests on
> their boxes.  Developers can (much more easily) run the standard getfirebug.com
> tests.  I thought it might be easier to have the buildbot slaves download the
> materials they needed instead of putting them in m-c.  But, whatever, we can do
> whichever is easiest.
> > 
> > 2) If we just think of this as a new "firebug" unittest suite, we could queue
> > this to be run when other unittest suites are run (on checkin, on nightly, on
> > release builds). We could also get it displayed on Tinderbox waterfalls, on
> > TBPL, and it would be presented in a format that developers would immediately
> > understand.
> > 
> That's the entire point here.  Yes.
> > 
> > 3) I'd like to suggest changing the summary of this bug. To me, this bug is
> > about "Setup new firebug test suite so it can be automated". 
> That's what it was.  That has now been done.

Agree with Clint on all these points. Plus yes I would also recommend *not* to use httpd.js

> > Once the scope of
> > suite, and setup/breakdown steps are debugged, and its ready to run to testrun
> > on a RelEng staging machine, please file a separate bug "automate running new
> > firebug test suite" in mozilla.org/ReleaseEngineering.
> >
> It's ready now.  We need your help with writing the buildbot patch for this or
> letting us know how you'd like that patch created. 
> > Hope that helps.
> Yep, it does.  Thanks.

So, if I understand correctly, the buildbot patch is the only missing thing? Or is it already done?

Honza
(In reply to comment #44)
> Sorry, but just FYI we can't use httpd.js with firebug: see  Bug 497799,
> basically sync code on one thread can't read and write itself.

Why does the code have to be synchronous, again?
If you are asking "why isn't httpd.js implemented in its own thread so it does not block browser js calls?", sorry I don't know the answer.  

I believe that all of our code is 'browser-like', that is our test suite driver is a browser that calls for the test HTML page, then runs JS on the values, and that JS causes Firefox to request pages. So we are synchronous with the server just because we are a browser. Or maybe I don't understand the question.
What I didn't understand is why you couldn't 1) initiate asynchronous processing, then 2) do a little work and write a little data, set up a callback to continue work later, then explicitly yield, and repeat step 2 as needed until 3) noting completion of asynchronous processing.
At the time we were using httpd.js we had problems with several tests based on dynamically generated responses (using registerPathHandler). So, we switched to a production server and used PHP to generate the responses. which solved the problems.

While I understand that using httpd.js can represent smaller load on test servers I agree with Clint that having a real production server is better.

Anyway, I can take a look at the problem again (perhaps httpd.js is now fixed or there is proper solution we should use), but it would mean to rewrite ~20 tests to adopt httpd.js and delay the goal of this bug.

I would personally propose to go with a standard webserver so, Firebug test suite is integrated with automated tests and Firebug team notified about results. This is something, which represents big help in finding compatibility issues in Firebug-Firefox combo.

Later, if we are experiencing expensive load on test servers, we can make httpd.js higher priority and go towards it adoption.

Honza
(In reply to comment #49)
> At the time we were using httpd.js we had problems with several tests based on
> dynamically generated responses (using registerPathHandler). So, we switched to
> a production server and used PHP to generate the responses. which solved the
> problems.
> 
> While I understand that using httpd.js can represent smaller load on test
> servers I agree with Clint that having a real production server is better.
> 
> Anyway, I can take a look at the problem again (perhaps httpd.js is now fixed
> or there is proper solution we should use), but it would mean to rewrite ~20
> tests to adopt httpd.js and delay the goal of this bug.
> 
I don't think this is a good use of your time.  The idea here is to run the Firebug unit tests which are run against an Apache webserver at getfirebug.com.  Since that is the way they are run, and since over 50% of the internet uses Apache, it makes sense to use that for testing a web-developer tool like firebug.  

This is what Andrew has implemented on our side, and I strongly think this is what we should do rather than re-writing the firebug tests so that they play well with the quirks of our automation system.  The goal here was not to change firebug so that they pass here, the goal here was to run the firebug tests as they are automatically.

> 
> Later, if we are experiencing expensive load on test servers, we can make
> httpd.js higher priority and go towards it adoption.
I've been doing some behind the scenes work (with a number of folks) and we are going to bring up this system in stand-alone, staging automation early next week.  This way we can get real data on how what the webserver load looks like and whether or not it seems sustainable.  As a result, the buildbot patch is not written yet, it and this question of the webserver are the two things blocking forward movement.

I'll report back with links to the staging automation as it comes online and the metrics from it once we see what the load looks like.  I think having real data will help us make a decision on how to march forward.  And it will have the added benefit that we can get some preliminary results too, which would be awfully nice as we roll toward Firefox 4 beta and 3.6.4 ship.
(In reply to comment #50)
> (In reply to comment #49)
> > At the time we were using httpd.js we had problems with several tests based on
> > dynamically generated responses (using registerPathHandler). So, we switched to
> > a production server and used PHP to generate the responses. which solved the
> > problems.
> > 
> > While I understand that using httpd.js can represent smaller load on test
> > servers I agree with Clint that having a real production server is better.
> > 
> > Anyway, I can take a look at the problem again (perhaps httpd.js is now fixed
> > or there is proper solution we should use), but it would mean to rewrite ~20
> > tests to adopt httpd.js and delay the goal of this bug.
> > 
> I don't think this is a good use of your time.  The idea here is to run the
> Firebug unit tests which are run against an Apache webserver at getfirebug.com.
>  Since that is the way they are run, and since over 50% of the internet uses
> Apache, it makes sense to use that for testing a web-developer tool like
> firebug.  
> 
> This is what Andrew has implemented on our side, and I strongly think this is
> what we should do rather than re-writing the firebug tests so that they play
> well with the quirks of our automation system.  The goal here was not to change
> firebug so that they pass here, the goal here was to run the firebug tests as
> they are automatically.
Agree


> > 
> > Later, if we are experiencing expensive load on test servers, we can make
> > httpd.js higher priority and go towards it adoption.
> I've been doing some behind the scenes work (with a number of folks) and we are
> going to bring up this system in stand-alone, staging automation early next
> week.  This way we can get real data on how what the webserver load looks like
> and whether or not it seems sustainable.  As a result, the buildbot patch is
> not written yet, it and this question of the webserver are the two things
> blocking forward movement.
> 
> I'll report back with links to the staging automation as it comes online and
> the metrics from it once we see what the load looks like.  I think having real
> data will help us make a decision on how to march forward.  And it will have
> the added benefit that we can get some preliminary results too, which would be
> awfully nice as we roll toward Firefox 4 beta and 3.6.4 ship.
Great thanks!

Honza
Just to note that test-bot results are displayed here:
http://getfirebug.com/tests/content/testbot/results/

Honza
The current testbot/results page does not seem to report if the test run fails completely. So when the FF 4.0 numbering changed, we got no report, and when the XPCOM registration change landed again no report. But these cases are ones we want to know about!  Of course I am guessing because all I see is nothing about the tests on those days.
I agree.  I'll send a record to the database in the case that the tests fail to start at all.

Crashes will be a bit trickier, but I'll think of some way to handle those better as well.
If the tests fail to run at all there is now a header document sent to the database with a "total tests" field of 0.

This is already represented in the view, but it should probably be updated for better formatting.
Thanks!

Test result page updated. If the "total tests" field is set to 0, the appropriate entry says: "No test executed" so, it's possible to distinguish: (a) the entire test-suite failed and (b) all tests passed.

Check out: http://getfirebug.com/testresults

Honza
These tests have been running standalone for a long while and there are no longer any plans to integrate them into automation.

See https://github.com/mozilla/Firebug-Test-Runner
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.