669520 - "Jetpack" tbpl incorrectly reports success when tests fail because of SDK test harness bug

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

When Add-on SDK tests fail because of a bug in the test harness, the "Jetpack" tbpl incorrectly reports success, f.e. (all the green ones):

http://tbpl.mozilla.org/?tree=Jetpack&rev=7d395565fac4

However, the "mozilla-central" tbpl correctly reports failure, f.e.:

http://tbpl.mozilla.org/?noignore=1&rev=a7684eca2bb7

Both should report failure.

Comment 1

[Phil Ringnalda (:philor)]

http://hg.mozilla.org/build/buildbotcustom/file/04be3ddbb235/steps/unittest.py#l872 appears to be where the m-c builds realize that they should be orange, but everything's such a twisty maze I don't have any clue where the apparently-different place the Jetpack tree runs come from might be.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #0)
> When Add-on SDK tests fail because of a bug in the test harness, the
> "Jetpack" tbpl incorrectly reports success, f.e. (all the green ones):
> 
> http://tbpl.mozilla.org/?tree=Jetpack&rev=7d395565fac4

was testing against mozilla-central rev http://tbpl.mozilla.org/?noignore=1&rev=3f27dc203e62 which shows the same results (all but windows are green)

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #1)
> http://hg.mozilla.org/build/buildbotcustom/file/04be3ddbb235/steps/unittest.
> py#l872 appears to be where the m-c builds realize that they should be
> orange, but everything's such a twisty maze I don't have any clue where the
> apparently-different place the Jetpack tree runs come from might be.

That would be http://hg.mozilla.org/build/buildbotcustom/file/3e3ad9b317e1/process/factory.py#l7655 which doesn't do the "Traceback" scraping (hence the lack of parsed results on Jetpack tbpl output).

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

There hasn't been an update with an example of this where the same cset was used, resolving invalid.