1262433 - fail-if not supported in robocop.ini

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: Testing, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1261270, there is a report that "fail-if" doesn't work in robocop.ini. We normally use skip-if in robocop.ini, but there may be times when fail-if is preferred...and it *should* be supported by nature of being based on mochitests.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Joel may be able to provide some insight here as well. I saw a comment in one of the manifests about Android using "old-style" mochitest manifests or something to that effect?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we had a bug to clean this up in all manifests in bug 958147.  

for robocop, we use manifest parser:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runrobocop.py#487

and for desktop mochitests we also use the same manifest parser:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1047

one difference is that we don't use the disabled option explicitly in the robocop version.  The other different is the filters, we only support chunk_by_slice in robocop, but in desktop mochitest we have chunk_by_dir, chunk_by_runtime and fallback to chunk_by_slice.

that is how we parse the .ini files, when this is sent to the browser and read in page from the JS harness, we use:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/manifestLibrary.js#7

this might be where the difference is- worth some additional time investigating any differences- maybe we can remove differences?

Comment 3

[Geoff Brown [:gbrown]]

There's no linkage between the robocop asserter class, mobile/android/tests/browser/robocop/src/org/mozilla/gecko/FennecMochitestAssert.java, and the test manifest. A test can explicitly call todo(), but otherwise, a failed assertion will result in an UNEXPECTED-FAIL.

There's more separation in robocop between the manifest (currently known only to the robocop harness) and the assertions (in java code), as compared to regular mochitests. So there's more work that I had expected.

There's value in supporting fail-if, certainly; but on the other hand, we've supported robocop for 4+ years using skip-if and hardly noticing that fail-if was missing. I don't think I'll get around to implementing this in the near future...maybe one day?

Comment 4

[Geoff Brown [:gbrown]]

Mass closing Firefox for Android :: Testing bugs with no progress in 2017.

If this bug is important to you, please re-open.