732638 - After "make reftest" or "make crashtest" run completes w/ failures, the harness says "To rerun your failures please run 'make mochitest-plain-rerun-failures'"

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Daniel Holbert [:dholbert]]

STR:
 1. Run "make reftest" in your objdir
  1a. If by some lucky chance you don't encounter any reftest failures, then modify a reftest such that you encounter failures.

ACTUAL RESULTS: The last 2 lines printed are:
> To rerun your failures please run 'make mochitest-plain-rerun-failures'
> make: *** [reftest] Error 1
(This already makes no sense, because I was running reftests, not mochitests.  But even if I run that command just for fun, It fails with "runtests.py: error: unable to find --run-only-tests file 'makefailures.json'")

EXPECTED RESULTS: reftest harness shouldn't print out a message about mochitests.

Looks like this was introduced by bug 725112.

Comment 1

[Daniel Holbert [:dholbert]]

More trivial STR:
 1. Create a "reftest.list" in your srcdir with these contents:

!= about:blank about:blank 

 2. Run TEST_PATH=[relative_path_in_srcdir_to_the_above]/reftest.list make reftest

This will produce
> REFTEST TEST-UNEXPECTED-FAIL | about:blank | image comparison (!=)
followed by more lines, and finally the " mochitest-plain-rerun-failures " line from comment 0.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: fix the error message on make reftest (1.0)

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 613639 [details] [diff] [review]
fix the error message on make reftest (1.0)

Review of attachment 613639 [details] [diff] [review]:
-----------------------------------------------------------------

Can you try my suggestion below? If that proves too difficult to get working we can use your patch, but I'd like something nicer.

::: testing/testsuite-targets.mk
@@ +106,5 @@
> +  @errors=`grep "TEST-UNEXPECTED-" $@.log` ;\
> +  if test "$$errors" ; then \
> +	  echo "$@ failed:"; \
> +	  echo "$$errors"; \
> +          echo "To rerun your failures please run 'make $@-rerun-failures'"; \

I'm not really wild about duplicating all this shell crap twice. Can we use some Makefile-fu and make this fancier? I'm thinking something like

define check_test_error_internal
  @errors=`grep "TEST-UNEXPECTED-" $@.log` ;\
  if test "$$errors" ; then \
	  echo "$@ failed:"; \
	  echo "$$errors"; \
          $(if $(1),echo $(1)) \
	  exit 1; \
...

CHECK_TEST_ERROR = $(call,check_test_error_internal)
CHECK_TEST_ERROR_RERUN = $(call,check_test_error_internal,"To rerun your failures please run 'make $@-rerun-failures'")

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: fix the error message on make reftest (2.0)

updated with similar suggestion to previous review comment.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 614378 [details] [diff] [review]
fix the error message on make reftest (2.0)

Review of attachment 614378 [details] [diff] [review]:
-----------------------------------------------------------------

Nice tiny patch, thanks!

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/73fd80989841

Comment 7

[(no longer active)]

https://hg.mozilla.org/mozilla-central/rev/73fd80989841