1119731 - gaia-build (Gb) and gaia-build-unit tests (Gbu) are marked as successful even though they contain test failures

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Ricky Chien [:rickychien] (inactive)]

As title, I've perceived that problem for a while. Please see https://treeherder.mozilla.org/ui/#/jobs?repo=b2g-inbound&revision=032eefa60028&exclusion_state=all

Comment 1

[Ricky Chien [:rickychien] (inactive)]

(In reply to Ricky Chien [:rickychien] from comment #0)
> As title, I've perceived that problem for a while. Please see
> https://treeherder.mozilla.org/ui/#/jobs?repo=b2g-
> inbound&revision=032eefa60028&exclusion_state=all

1. Show hide jobs
2. Click on Gbu(it's green) in B2G Desktop Linux x64 opt
3. Look at failure summary:


TEST-UNEXPECTED-FAIL | build/test/unit/webapp_manifests_test.js | webapp-manifest.js setConfig, genStageWebappJSON, fillExternalAppManifest fillExternalAppManifest

TypeError: Cannot read property 'leafName' of undefined

TEST-UNEXPECTED-FAIL | build/test/unit/webapp_optimize_test.js | webapp-optimize.js "before each" hook

SyntaxError: Unexpected token >

TEST-UNEXPECTED-FAIL | build/test/unit/webapp_zip_test.js | webapp-zip.js setOptions, getCompression, and isExcludeFromZip setOptions

TypeError: Cannot call method 'clone' of undefined 

However, our Gbu is green :(

Comment 2

[Ed Morley [:emorley]]

The colour that a job appears in Treeherder/TBPL's UI is determined by the job's result code, which is not determined by Treeherder, but instead something upstream: {Buildbot, mozharness, the test harness, ...}

The reason the log viewer doesn't display those error lines by default (until "show successful steps" is checked), is because Buildbot incorrectly called that step within the job successful.

ie:
http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-linux64_gecko/1420789651/b2g-inbound_ubuntu64_vm-b2gdt_test-gaia-build-unit-bm115-tests1-linux64-build179.txt.gz

00:53:28     INFO -  [0m[31m     TypeError: Cannot call method 'clone' of undefined[0m[90m
00:53:28     INFO -        at WebappZip.setOptions (/builds/slave/test/gaia/build/webapp-zip.js:25:36)
00:53:28     INFO -        at Context.<anonymous> (/builds/slave/test/gaia/build/test/unit/webapp_zip_test.js:97:17)
00:53:28     INFO -        at callFn (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:223:21)
00:53:28     INFO -        at Test.Runnable.run (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:216:7)
00:53:28     INFO -        at Runner.runTest (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:373:10)
00:53:28     INFO -        at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:451:12
00:53:28     INFO -        at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:298:14)
00:53:28     INFO -        at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:308:7
00:53:28     INFO -        at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:246:23)
00:53:28     INFO -        at /builds/slave/test/gaia/node_modules/mocha/lib/runner.js:270:7
00:53:28     INFO -        at done (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:185:5)
00:53:28     INFO -        at callFn (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:228:7)
00:53:28     INFO -        at Hook.Runnable.run (/builds/slave/test/gaia/node_modules/mocha/lib/runnable.js:216:7)
00:53:28     INFO -        at next (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:258:10)
00:53:28     INFO -        at Object._onImmediate (/builds/slave/test/gaia/node_modules/mocha/lib/runner.js:275:5)
00:53:28     INFO -        at processImmediate [as _immediateCallback] (timers.js:330:15)
00:53:28     INFO -  [0m
00:53:28     INFO - Return code: 0
00:53:28     INFO - TinderboxPrint: gaia-build-unit-tests: 83/<em class="testfail">3</em>/0
00:53:28     INFO - Tests exited with return code 0: success
00:53:28     INFO - # TBPL SUCCESS #
00:53:28     INFO - Running post-action listener: _resource_record_post_action
00:53:28     INFO - Running post-run listener: _resource_record_post_run
00:53:29     INFO - Total resource usage - Wall time: 98s; CPU: 94.0%; Read bytes: 26705920; Write bytes: 846815232; Read time: 7376; Write time: 1922256
00:53:29     INFO - install - Wall time: 36s; CPU: 100.0%; Read bytes: 49152; Write bytes: 57147392; Read time: 12; Write time: 90720
00:53:29     INFO - run-tests - Wall time: 63s; CPU: 91.0%; Read bytes: 25980928; Write bytes: 764567552; Read time: 7328; Write time: 1763340
00:53:29     INFO - Running post-run listener: _upload_blobber_files
00:53:29  WARNING - Blob upload gear skipped. Missing cmdline options.
00:53:29     INFO - Copying logs to upload dir...
00:53:29     INFO - mkdir: /builds/slave/test/build/upload/logs
program finished with exit code 0
elapsedTime=589.422636
========= Finished '/tools/buildbot/bin/python scripts/scripts/gaia_build_unit.py ...' (results: 0, elapsed: 9 mins, 50 secs) (at 2015-01-09 00:53:29.735866) =========

It seems like perhaps mozharness believes the exit code zero, and ignores the fact that the test failed count was "3".

This conditional doesn't seem to be working as expected:
http://mxr.mozilla.org/build-central/source/mozharness/mozharness/mozilla/testing/gaia_test.py#260

Comment 3

[Ricky Chien [:rickychien] (inactive)]

Gaia-try's Gb has same problem. https://treeherder.mozilla.org/ui/#/jobs?repo=gaia-try&revision=c45e9bdfb9d0
I think it's very crucial because it should be a permanent red in tree status now.

Comment 4

[Ed Morley [:emorley]]

(In reply to Ricky Chien [:rickychien] from comment #3)
> Gaia-try's Gb has same problem.
> https://treeherder.mozilla.org/ui/#/jobs?repo=gaia-try&revision=c45e9bdfb9d0
> I think it's very crucial because it should be a permanent red in tree
> status now.

Adjusted summary.

However this is still something that someone from B2G/releng will need to fix - picking someone who may be able to help :-)

Comment 5

[Julien Wajsberg [:julienw]]

Hey George, I think the problem here comes from how we run mocha in Gb and Gbu. In both cases we use the utils.Commander util [1], which does not forward mocha's exitValue (both in utils-xpc and utils-node) [2] [3].

I don't really understand the code that invokes "callback" in both utils-xpc and utils-node, and especially it seems to me that both codes are not equivalent.

[1] https://github.com/mozilla-b2g/gaia/blob/793773bb2944b42a85dd160049e605cbd880c4da/build/build-test.js#L68-L73
[2] https://github.com/mozilla-b2g/gaia/blob/793773bb2944b42a85dd160049e605cbd880c4da/build/utils-xpc.js#L999-L1011
[3] https://github.com/mozilla-b2g/gaia/blob/793773bb2944b42a85dd160049e605cbd880c4da/build/utils-node.js#L64-L87

Because you're the original author of Commander, can you have a look at this? Thanks!

Comment 6

[(Inactive after June) George Duan [:gduan] [:喬智]]

You're right.
push to gaia-try to verify.
https://github.com/mozilla-b2g/gaia/pull/27698

Comment 7

[(Inactive after June) George Duan [:gduan] [:喬智]]

Attachment: PR to master

Comment 8

[(Inactive after June) George Duan [:gduan] [:喬智]]

Comment on attachment 8555166 [details] [review]
PR to master

Hi Ricky,
Please kindly check this patch.
this patch has successfully turn gb to red if test fails.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Just to be sure, you're going to fix the existing failures before merging this PR, right? :)

Comment 10

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8555166 [details] [review]
PR to master

LGTM.

I leaved some comments. I prefer to use 1 instead of -1 as exit code.
Remember to put test.skip() back before keyboard tests is fixed.

Comment 11

[Ricky Chien [:rickychien] (inactive)]

Ryan, We're going to fix these keyboard integration tests in bug 1125753.

Comment 12

[John Lu [:mnjul][Please NI or I'll miss]]

Please ping me if fixing the keyboard tests (bug 1125753) needs to be prioritized.

Comment 13

[(Inactive after June) George Duan [:gduan] [:喬智]]

After re-triggered, the gb is green now. I believe there would be more intermittent failures coming from Gb ...

master: https://github.com/mozilla-b2g/gaia/commit/52e714d415f100c1e5b68453d52bcaca13691249

Comment 14

[John Lu [:mnjul][Please NI or I'll miss]]

(In reply to John Lu [:mnjul] [MoCoTPE] from comment #12)
> Please ping me if fixing the keyboard tests (bug 1125753) needs to be
> prioritized.

Side note: the patch fixing the tests has just landed onto master.