734223 - Jetpack jobs on other trees are orange/red

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

If we look at the Firefox tree [1], we can see all Jetpacks jobs going orange while the Jetpack tree [2] shows most of them green.

Perhaps the difference is that the jetpack-location.txt [3] points to a tar ball [4] from a revision from Feb. 24th [5].

I will push the latest version to try and see what happens.

[1] https://tbpl.mozilla.org/?noignore=1&jobname=jetpack
[2] https://tbpl.mozilla.org/?tree=Jetpack
[3] hg.mozilla.org/mozilla-central/raw-file/default/testing/jetpack/jetpack-location.txt
[4] http://hg.mozilla.org/projects/addon-sdk/archive/28b4f9e190e3.tar.bz2
[5] http://hg.mozilla.org/projects/addon-sdk/rev/28b4f9e190e3

Comment 1

[Armen [:armenzg]]

Attachment: try jetpack's latest revision

Comment 2

[Armen [:armenzg]]

This whiteboard should make the patch go through the try server and report back.

Comment 3

[Mozilla RelEng Bot]

Autoland Patchset:
	Patches: 604159
	Branch: mozilla-central => try
	Destination: http://hg.mozilla.org/try/pushloghtml?changeset=e729f4e26ea0
Try run started, revision e729f4e26ea0. To cancel or monitor the job, see: https://tbpl.mozilla.org/?tree=Try&rev=e729f4e26ea0

Comment 4

[Armen [:armenzg]]

There is also the issue that Tracebacks turn the build "orange".

philor-afk: armenzg: yeah, it's "^Traceback" in the whatever it's called, evaluateCommand I think, because the way it's run on the rest of the trees predates the Jetpack suite having a pass/fail number to look at

Comment 5

[Phil Ringnalda (:philor)]

http://hg.mozilla.org/build/buildbotcustom/annotate/bdd5e90dc50f/steps/unittest.py#l876 and https://bugzilla.mozilla.org/show_bug.cgi?id=570248#c35

When we first automated it, it was hard to get a total pass/fail count, so warner was saying "just trust the exit code to tell you success or failure" and instead we trust the presence of "^Traceback" to tell us fail (for non-Jetpack-tree, for the Jetpack tree we trust the exit code, and just translate it from red to orange).

Comment 6

[Brian Warner [:warner :bwarner]]

Yeah, I highly recommend using rc=0 to mean success, rc!=0 to mean failure, and log+summarize stdout/stderr but don't parse it to decide whether the tests passed. We have lots of tests with "exception" or "tracback" or "error" in their names, and the verbose mode might emit strings with those words in random places, so parsing the output is likely to be pretty fragile.

Comment 7

[Mozilla RelEng Bot]

Try run for e729f4e26ea0 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=e729f4e26ea0
Results (out of 14 total builds):
    success: 14
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-e729f4e26ea0

Comment 8

[Phil Ringnalda (:philor)]

Attachment: buildbotcustom - trust the exit code

I don't really get why this isn't how every harness works, why we have to do the "maybe the harness gave us a reasonable exit code, or maybe it didn't and we have to grovel around looking for a magic word that means no-failure or a magic word that means failure," but you can see it working (or more accurately, working but being defeated by what buildbot is doing) in https://tbpl.mozilla.org/?noignore=1&jobname=jetpack&fromchange=9d19b56f39c5&tochange=ee56787a20fb - the first push should be green, it's exactly what the Jetpack folks want their test runs to look like, but right now we're turning it orange because of looking for ^Traceback, while the top push introduced a test failure, so the harness returns a non-zero exit code which we're failing to give the usual treatment of turning into WARNINGS and making it orange.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 606996 [details] [diff] [review]
buildbotcustom - trust the exit code

Once upon a time, there was tinderbox, and it was good...err...

Harnesses from a long time ago in a galaxy far away didn't return proper exit codes and relied on log scraping instead.

Now we know better.

Are you sure you don't want to change the script to exit with 1 only in the case of the actual test failing?

Comment 10

[Phil Ringnalda (:philor)]

I'd like to change the script, yes, but I don't know how to deal with the exit code here idiomatically. For the Jetpack tree, it's simple and obvious, it's using rc_eval_func({1: WARNINGS}) and just sticking in a 2: FAILURE does it, but here, other than ploddingly unidiomatically just saying "if cmd.rc == 1 and superResult == FAILURE: return WARNINGS" I don't actually know how to wire up rc_eval_func instead of an evaluateCommand.

Comment 11

[Phil Ringnalda (:philor)]

Comment on attachment 606996 [details] [diff] [review]
buildbotcustom - trust the exit code

And despite the horrible fit with the current summary, I think most of the fiddling with the script I want to do will fit better in bug 737656.


http://hg.mozilla.org/build/buildbotcustom/rev/df2151e42eba

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

this was deployed during a reconfig today

Comment 13

[Phil Ringnalda (:philor)]

And now https://tbpl.mozilla.org/?noignore=1&rev=7290c4f0a150&jobname=Jetpack is properly green/orange

Comment 14

[Armen [:armenzg]]

Should we unhide the green ones and notify developers?

Comment 15

[Phil Ringnalda (:philor)]

No, this wasn't about "solve every problem that causes Jetpack to not be a tier 1 suite," it was just about "make the green runs green so that when kwierso looks at them with noignore=1, he doesn't have to dig through the wrong colors to see which is orange-green and which is red-orange."