1084163 - [mozharness] When make check fails, other tests are not run

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Mike Hommey [:glandium]]

Before mozharness:
  https://treeherder.mozilla.org/ui/#/jobs?repo=elm&revision=e5c5caa646ec
make check happened after make upload, so tests could start, and make check failure turned the build orange.

With mozharness:
 https://treeherder.mozilla.org/ui/#/jobs?repo=elm&revision=e8f90a5158c5
make check happens before make upload somehow, so tests don't start.

I guess make check should have a dependency on make upload.

Comment 1

[Michael Shal [:mshal]]

Attachment: remove-check-sendchange

This just removes 'make check' from the automation/build step. We'll run it as a separate step in mozharness, due to the complications of either moving sendchange into automation/build (which is needed to trigger tests before 'mach build' returns).

Comment 2

[Michael Shal [:mshal]]

inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/ebfb08b7752d
try: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=434c88299059

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ebfb08b7752d

Comment 4

[Mike Hommey [:glandium]]

(In reply to Michael Shal [:mshal] from comment #1)
> This just removes 'make check' from the automation/build step. We'll run it
> as a separate step in mozharness

Was that ever done? Is there a bug #?

Comment 5

[Michael Shal [:mshal]]

It looks like it was added in http://hg.mozilla.org/build/mozharness/rev/42d035f84c63 as part of bug 1055918 (and a collection of other bugs).

Comment 6

[Mike Hommey [:glandium]]

(In reply to Michael Shal [:mshal] from comment #5)
> It looks like it was added in
> http://hg.mozilla.org/build/mozharness/rev/42d035f84c63 as part of bug
> 1055918 (and a collection of other bugs).

Well, make check doesn't run on elm, so it's either not true, or something broke since then.

Comment 7

[Michael Shal [:mshal]]

jlund, any idea why make check wouldn't be running on elm?

Comment 8

[Jordan Lund (:jlund)]

(In reply to Michael Shal [:mshal] from comment #7)
> jlund, any idea why make check wouldn't be running on elm?

hmm, so https://tbpl.mozilla.org/php/getParsedLog.php?id=51749740&full=1&branch=elm shows that make -k check was run. it even highlights the errors:

21:21:45  WARNING - TEST-UNEXPECTED-FAIL | GTest unit test: failed
21:21:45     INFO - Finished running GTest tests.
21:21:45  WARNING - gtest TEST-UNEXPECTED-FAIL | gtest | test failed with return code 1
21:21:45     INFO - make[1]: *** [check] Error 1
21:21:45     INFO - make[1]: Leaving directory `/builds/slave/elm-lx-00000000000000000000000/build/src/obj-firefox/testing/gtest'
21:21:45     INFO - make: *** [check] Error 2
21:21:45     INFO - PASSED
21:21:46    ERROR - Return code: 2
21:21:46     INFO - TinderboxPrint: check<br/>1810/<em class="testfail">4</em>/0

though the overall build is not orange. *sigh* that's a bug. /me files https://bugzil.la/1093911

Comment 9

[Jordan Lund (:jlund)]

adding the following to an upcoming ash run:

diff --git a/mozharness/mozilla/building/buildbase.py b/mozharness/mozilla/building/buildbase.py
index 8beecae..8888e57 100755
--- a/mozharness/mozilla/building/buildbase.py
+++ b/mozharness/mozilla/building/buildbase.py
@@ -1431,11 +1431,18 @@ or run without that action (ie: --no-{action})"

         parser = CheckTestCompleteParser(config=c,
                                          log_obj=self.log_obj)
-        self.run_command_m(command=cmd,
-                           cwd=dirs['abs_obj_dir'],
-                           env=env,
-                           output_parser=parser)
+        return_code = self.run_command_m(command=cmd,
+                                       cwd=dirs['abs_obj_dir'],
+                                       env=env,
+                                       output_parser=parser)
         parser.evaluate_parser()
+        if return_code:
+            self.return_code = self.worst_level(
+                EXIT_STATUS_DICT[TBPL_WARNING], self.return_code,
+                AUTOMATION_EXIT_CODES[::-1]
+            )
+            self.error("'make -k check' did not run successfully. Please check "
+                       "log for errors.")

Comment 10

[Michael Shal [:mshal]]

aurora uplift for bug 1093897: https://hg.mozilla.org/releases/mozilla-aurora/rev/43dcdc7f4626

Comment 11

[Michael Shal [:mshal]]

beta uplift for bug 1093897: https://hg.mozilla.org/releases/mozilla-beta/rev/5591e0a83c4d