Firefox tbpl incorrectly reports failure when SDK tests pass

RESOLVED FIXED in 1.1

Status

P1
normal
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: myk, Assigned: irakli)

Tracking

unspecified

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
After I updated the version of the Add-on SDK being tested on central to fix a test harness bug <http://tbpl.mozilla.org/?tree=Firefox&noignore=1&rev=ea752bcac5c5>, test runs are once again running to completion, but in three cases tbpl incorrectly reports failure although all tests were successful:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309986944.1309987204.27825.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309988186.1309988602.2640.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1309988854.1309989252.5665.gz

Lukas: could this have anything to do with the Python rewrite of the script being used to trigger these test runs and collect results?
In all three logs:

info: executing 'test-unload.testUnloading'
error: An exception occurred.
Traceback (most recent call last):
error
info: pass: Unloader functions are called on unload.
console: [JavaScript Error: "An exception occurred.
Traceback (most recent call last):
error
"]

and things like

4127 of 4127 tests passed.
OK
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null

which is what the buildbot harness looks for, /^Traceback/, to tell that the jetpack harness was unhappy about something other than just a failed test. Since not having that check on the Jetpack tree was why you didn't know that it was broken, I'd say tbpl is correctly reporting failure, and if those aren't failures you'll want them to stop saying "Traceback" at the start of a line.
(Reporter)

Comment 2

7 years ago
Phil: good point.

Irakli: can you look into why these tracebacks are being reported even though all tests appear to have passed?
Assignee: nobody → rFobic
Component: Release Engineering → General
Product: mozilla.org → Add-on SDK
QA Contact: release → general
Target Milestone: --- → 1.1
Version: other → unspecified
(Reporter)

Updated

7 years ago
Priority: -- → P1
Status: NEW → ASSIGNED
Created attachment 544785 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

Pointer to Github pull-request
Comment on attachment 544785 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

This fixes following errors:


info: executing 'test-unload.testUnloading'
error: An exception occurred.
Traceback (most recent call last):
error
info: pass: Unloader functions are called on unload.
console: [JavaScript Error: "An exception occurred.
Traceback (most recent call last):
error
"]

Unfortunately I'm not able to reproduce:

error: An exception occurred.
Traceback (most recent call last):
  File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
TypeError: this._browser is null
error: An exception occurred.
Attachment #544785 - Flags: review?(myk)
(Reporter)

Comment 5

7 years ago
Comment on attachment 544785 [details]
Pointer to Github pull request: https://github.com/mozilla/addon-sdk/pull/206

r=myk

(In reply to comment #4)
> Unfortunately I'm not able to reproduce:
> 
> error: An exception occurred.
> Traceback (most recent call last):
>   File "resource://testpkgs-api-utils-lib/tabs/tab.js", line 93, in destroy
> TypeError: this._browser is null
> error: An exception occurred.

I can't reproduce it either.  But it's very important that we fix the problem.  What's the next step?
Attachment #544785 - Flags: review?(myk) → review+
I'll land this one and let's see maybe it solves the other one as well ?
(Reporter)

Comment 7

7 years ago
This landed 18 days ago, and it did indeed appear to resolve the third problem.

https://github.com/mozilla/addon-sdk/commit/cc2de5b6e21531a3861b7edd645aaef8a1eb0931
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.