669768 - Firefox tbpl incorrectly reports failure when SDK tests pass

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Myk Melez [:myk] [@mykmelez]]

After I updated the version of the Add-on SDK being tested on central to fix a test harness bug <http://tbpl.mozilla.org/?tree=Firefox&noignore=1&rev=ea752bcac5c5>, test runs are once again running to completion, but in three cases tbpl incorrectly reports failure although all tests were successful:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309986944.1309987204.27825.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309988186.1309988602.2640.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309988854.1309989252.5665.gz

Lukas: could this have anything to do with the Python rewrite of the script being used to trigger these test runs and collect results?

Comment 1

[Phil Ringnalda (:philor)]

In all three logs:

info: executing 'test-unload.testUnloading'
error: An exception occurred.
Traceback (most recent call last):
error
info: pass: Unloader functions are called on unload.
console: [JavaScript Error: "An exception occurred.
Traceback (most recent call last):
error
"]

and things like

4127 of 4127 tests passed.
OK
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null

which is what the buildbot harness looks for, /^Traceback/, to tell that the jetpack harness was unhappy about something other than just a failed test. Since not having that check on the Jetpack tree was why you didn't know that it was broken, I'd say tbpl is correctly reporting failure, and if those aren't failures you'll want them to stop saying "Traceback" at the start of a line.

Comment 2

[Myk Melez [:myk] [@mykmelez]]

Phil: good point.

Irakli: can you look into why these tracebacks are being reported even though all tests appear to have passed?

Comment 3

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Attachment: Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

Pointer to Github pull-request

Comment 4

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Comment on attachment 544785 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

This fixes following errors:


info: executing 'test-unload.testUnloading'
error: An exception occurred.
Traceback (most recent call last):
error
info: pass: Unloader functions are called on unload.
console: [JavaScript Error: "An exception occurred.
Traceback (most recent call last):
error
"]

Unfortunately I'm not able to reproduce:

error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 544785 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

r=myk

(In reply to comment #4)
> Unfortunately I'm not able to reproduce:
> 
> error: An exception occurred.
> Traceback (most recent call last):
>   File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
> TypeError: this._browser is null
> error: An exception occurred.

I can't reproduce it either.  But it's very important that we fix the problem.  What's the next step?

Comment 6

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

I'll land this one and let's see maybe it solves the other one as well ?

Comment 7

[Myk Melez [:myk] [@mykmelez]]

This landed 18 days ago, and it did indeed appear to resolve the third problem.

https://github.com/mozilla/addon-sdk/commit/cc2de5b6e21531a3861b7edd645aaef8a1eb0931