Closed Bug 626434 Opened 14 years ago Closed 13 years ago

Firebug testbot needs to reliable

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: johnjbarton, Unassigned)

Details

(Whiteboard: [firebug-p1]) 

We've made huge progress on Firebug testing over the last year. We now have tests that prevent Firebug regressions and they are adequate to test Firefox trunk for major issues. We have an automatic test runner "testbot" that reduces the overhead of running the tests. 

The testbot could be a valuable channel for communicating status between Firefox and Firebug devs. Unfortunately it is a channel, but not a valuable one ;-).

We need to come up with a way to debug the testbot so its results match what we get on our dev machines.  As it stands the Firebug team has no access to the server and the server team cannot debug the Firebug test runs. We can't make progress this way. 

The Firebug team knows about a couple of tests that seem to have timing issues; we'll fix them. But after that we need a new approach.
I put this on [firebug-p1], but of course it does not block anything.
Whiteboard: [firebug-p1]
Please cc me on any test-bot related bugs and direct any questions towards me.

I'm not sure why the test-bot is reporting more errors than when you run the tests manually, but the tests are being run on two VM's, so it could be a performance issue.

I have two ideas to potentially improve stability.

1) Back in the summer we tried to integrate the firebug tests into build-bot and one of the requirements was that we couldn't hit any external (to mozilla) webservers.  That is why the tests are being run on 10.250.5.0 at the moment.  However, for the purposes of the test-bot, we don't have this requirement, so it really doesn't make much sense not to use www.getfirebug.com instead. I'm going to update the test runners to point to that server instead.  I don't know if this is the cause of the flakiness, but it definitely isn't helping.

2) If the firebug team wants, we could give them control over the test-bot. I can package the test-bot as a pypi package.  However this would mean having dedicated machines or vm's to run the tests.

I agree that it definitely needs to be debugged. I'll run the tests manually on the VM's to see if it's an environment issue.
> 2) If the firebug team wants, we could give them control over the test-bot. I
> can package the test-bot as a pypi package.  However this would mean having
> dedicated machines or vm's to run the tests.
Yes, this would be great. So, I guess two machines should be enough, correct?
Honza
Also we should run the testlist off "https" always just to be consistent everywhere.
Just for ease of reference the testbot URL is
http://getfirebug.com/testresults
To make the testbot appear more reliable, consider adding:
  1) A field to the testlist "bugzilla: <number>",
  2) Show the number in the property list when test results are expanded, linked to bugzilla,
  3) Color test result with bugzilla numbers orange rather than red,
  4) Color test runs orange rather than green or red if the run fails only on tests with bugzilla numbers.

This would help communications about the test results.
I just wanted to add this to the bug report so we can remember it: As I understand it from IRC, one major source of false-negative tests has been discovered by Andrew and Jan: the server runs with a browser size much smaller than any we use on the desktop.  Honza is changing the tests to account for this, though I think a minimum setting and consistent setting on the server should also be considered.
(In reply to comment #3)
> Yes, this would be great. So, I guess two machines should be enough, correct?

Ideally three machines, one for Windows, Linux and Mac.  I think the main problem here, as johnjbarton has said, is that no one really has the full picture.  I'm not really sure how the tests/fbtest framework works, and you aren't sure how the test-bot works or the vm environment is set up.

I'm going to do some refactoring and package the test-bot up in a pypi package so you guys can run it on an environment that you know works. However, we'll probably still need to keep things running on our side if we ever want to integrate into tbpl.  But we can cross that bridge when we get there.
> I'm going to do some refactoring and package the test-bot up in a pypi
> package so you guys can run it on an environment that you know works.
Great, I'll try to run the test-bot script on my machine (Windows).
Honza
(In reply to comment #6)
> To make the testbot appear more reliable, consider adding:
>   1) A field to the testlist "bugzilla: <number>",
>   2) Show the number in the property list when test results are expanded,
> linked to bugzilla,
>   3) Color test result with bugzilla numbers orange rather than red,
>   4) Color test runs orange rather than green or red if the run fails only on
> tests with bugzilla numbers.
> This would help communications about the test results.
I like the idea. So, the test number would be part of test entry in the test list, correct?

For example:

{
  group: "dom",
  uri: "dom/breakpoints/breakOnProperty.js",
  desc: "Break on property change",
  testPage: "dom/breakpoints/breakOnProperty.html",
  bugzilla: 123456
}

Honza
We seem to be down to two failures on linux with 1.7 / FF4.0.
I fixed one of those, a problem where the global state was not cleared correctly for the test.
I added some tracing to try to figure out the other one (break on errors).

The Windows results are bogus, needs work on the testbot machine.

I don't have a mac to look into the others.

Overall we are getting close, the results are much more consistent now.
I looked into the Windows failures and the problem is that the window is getting opened really really small (no content visible whatsoever).

I have no idea why it is doing this and I tried clearing profile caches and setting command line arguments to no avail.

Honza, are you still planning on making the FBTest extension set the window size?  That should fix the results once it is checked in.
(In reply to comment #12)
> Honza, are you still planning on making the FBTest extension set the window
> size?  That should fix the results once it is checked in.

FBTest 1.7b13 has the support included. 
http://getfirebug.com/releases/fbtest/1.7/fbTest-1.7b13.xpi

http://getfirebug.com/releases/firebug/test-bot.config
Is updated to use b13

I have updated test-bot docs on getfirebug.com/wiki
http://getfirebug.com/wiki/index.php/Firebug_Test_Bot#Browser_Window_Size

In a nutshell:

If the -runFBTests flag is specified on the command line for firefox.exe (which is, since the test bot is using it) the browser window (with Firebug) size & position is updated.

position: 0,0
size: 1024x768

Both, default position and size can be customized in preferences.

pref("extensions.fbtest.defaultOuterWidth", 1024);
pref("extensions.fbtest.defaultOuterHeight", 768);
pref("extensions.fbtest.defaultScreenX", 0);
pref("extensions.fbtest.defaultScreenY", 0);

Honza
Awesome, it looks like the fix worked. The test-bot is reporting 3 failures on Windows (and Linux) consistently. Is this what you see when you run the tests manually?

If you guys feel that the test-bot is reliable enough I think we can close this bug and re-open if something else goes wrong.
Yes, it looks much better now, let me just yet investigate these 3 failures and then we can close.

Honza
Honza, I fixed 3408 last week. I believe the storage.js is a real Firefox bug, the storage is not being updated somehow. I can't reliably reproduce the breakOnError case, but it acts like jsd is off or not properly calling onError.
When running testbot on my machine with latest source, I am getting quite good results (I have uploaded it to coucheone.com and redirected the test-results app to it through URL parameter):
http://legoas/firebug/tests/content/testbot/results/?db=http://firebug.couchone.com/

So, I believe we can close this bug.

Honza
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
The URL above gives 
L'Url demandée /pauline.legoas/index.html/firebug/tests/content/testbot/results/?db=http://firebug.couchone.com/ n'existe pas.
You need to log in before you can comment on or make changes to this bug.