1507635 - Running reftest/crashtest tests in isolation from 'mach try fuzzy' doesn't work

Status: RESOLVED FIXED

(Testing :: General, defect, P3)

Description

[Marco Castelluccio [:marco]]

Attachment: reftest.tar.xz

./mach try fuzzy layout/reftests/css-ruby/nested-ruby-1.html layout/reftests/w3c-css/submitted/counter-styles-3/disclosure-styles.html

The failure is:
> runreftest.py: error: unrecognized arguments: -- tests/reftest/tests/layout/reftests/reftest.list

Comment 1

[Marco Castelluccio [:marco]]

After bug 1507767 is fixed, crashtests can be scheduled but they fail with a similar error:
> runreftest.py: error: unrecognized arguments: -- tests/reftest/tests/testing/crashtest/crashtests.list

Comment 2

[Marco Castelluccio [:marco]]

Attachment: When running specific tests on try, don't use the 'tests' attribute from the suite

This is the first step to make this work. After applying this, reftests start but fail very early with:
> ERROR - REFTEST ERROR | Exactly one of reftest.manifests or reftest.tests must be specified.

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Log reftest.manifests and reftest.tests when there is an error with them

Comment 4

[Marco Castelluccio [:marco]]

Attachment: reftest paths should be relative to abs_reftest_dir/tests

Comment 5

[Andrew Halberstadt [:ahal]]

Comment on attachment 9026134 [details] [diff] [review]
Log reftest.manifests and reftest.tests when there is an error with them

Review of attachment 9026134 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/reftest.jsm
@@ +363,4 @@
>  
>          if ((testList && manifests) || !(testList || manifests)) {
>              logger.error("Exactly one of reftest.manifests or reftest.tests must be specified.");
> +            logger.debug("reftest.manifests is: " + manifests);

Did you mean to make this debug?

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 9026136 [details] [diff] [review]
reftest paths should be relative to abs_reftest_dir/tests

Review of attachment 9026136 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for catching and fixing this!

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab36ef2d6d00
When running specific tests on try, don't use the 'tests' attribute from the suite. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b24e89c77f1
Log reftest.manifests and reftest.tests when there is an error with them. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/da7bf8203ec1
reftest paths should be relative to abs_reftest_dir/tests. r=ahal

Comment 8

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/ab36ef2d6d00
https://hg.mozilla.org/mozilla-central/rev/8b24e89c77f1
https://hg.mozilla.org/mozilla-central/rev/da7bf8203ec1

Comment 9

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/0fb2cab11e3c