Closed Bug 1507635 Opened 10 months ago Closed 10 months ago

Running reftest/crashtest tests in isolation from 'mach try fuzzy' doesn't work

Categories

(Testing :: General, defect, P3)

defect

Tracking

(firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox65 --- fixed

People

(Reporter: marco, Assigned: marco)

References

(Blocks 1 open bug)

Details

Attachments

(4 files)

Attached file reftest.tar.xz
./mach try fuzzy layout/reftests/css-ruby/nested-ruby-1.html layout/reftests/w3c-css/submitted/counter-styles-3/disclosure-styles.html

The failure is:
> runreftest.py: error: unrecognized arguments: -- tests/reftest/tests/layout/reftests/reftest.list
After bug 1507767 is fixed, crashtests can be scheduled but they fail with a similar error:
> runreftest.py: error: unrecognized arguments: -- tests/reftest/tests/testing/crashtest/crashtests.list
Priority: -- → P3
Summary: Running reftest tests in isolation doesn't work → Running reftest tests in isolation from 'mach try fuzzy' doesn't work
Summary: Running reftest tests in isolation from 'mach try fuzzy' doesn't work → Running reftest/crashtest tests in isolation from 'mach try fuzzy' doesn't work
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
This is the first step to make this work. After applying this, reftests start but fail very early with:
> ERROR - REFTEST ERROR | Exactly one of reftest.manifests or reftest.tests must be specified.
Attachment #9026096 - Flags: review?(ahal)
Comment on attachment 9026134 [details] [diff] [review]
Log reftest.manifests and reftest.tests when there is an error with them

Review of attachment 9026134 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/reftest.jsm
@@ +363,4 @@
>  
>          if ((testList && manifests) || !(testList || manifests)) {
>              logger.error("Exactly one of reftest.manifests or reftest.tests must be specified.");
> +            logger.debug("reftest.manifests is: " + manifests);

Did you mean to make this debug?
Attachment #9026134 - Flags: review?(ahal) → review+
Comment on attachment 9026136 [details] [diff] [review]
reftest paths should be relative to abs_reftest_dir/tests

Review of attachment 9026136 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for catching and fixing this!
Attachment #9026136 - Flags: review?(ahal) → review+
Attachment #9026096 - Flags: review?(ahal) → review+
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab36ef2d6d00
When running specific tests on try, don't use the 'tests' attribute from the suite. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b24e89c77f1
Log reftest.manifests and reftest.tests when there is an error with them. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/da7bf8203ec1
reftest paths should be relative to abs_reftest_dir/tests. r=ahal
https://hg.mozilla.org/mozilla-central/rev/ab36ef2d6d00
https://hg.mozilla.org/mozilla-central/rev/8b24e89c77f1
https://hg.mozilla.org/mozilla-central/rev/da7bf8203ec1
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
You need to log in before you can comment on or make changes to this bug.