1836639 - Expose manifest.ini conditional annotations beyond "skip-if" like "run-if" and "fail-if" test information via searchfox test info boxes

Status: NEW

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

Searchfox currently only exposes "skip-if" via the test-info boxes but it turns out there's a bunch of other conditional directives that can be used and should be exposed in the info boxes for completeness (and because it's quite misleading when not exposed).

This list seems to be the most comprehensive:

known_conditional_annotations = [
    "skip-if",
    "fail-if",
    "run-if",
    "fails-if",
    "fuzzy-if",
    "random-if",
    "asserts-if",
]

But the filters defined in the manifestparser which are then specified in the default filters list only seems to cover "skip-if", "run-if", and "fail-if" but not include "fails-if" with an "s" which can be found in test-info-all.json. "fuzzy-if", "random-if", and "asserts-if" do not seem to exist in test-info-all.json.

The testinfo file also lists unconditional annotations but they just seem to get tallied but not otherwise processed?

known_unconditional_annotations = ["skip", "fail", "asserts", "random"]

:ahal, do you know if searchfox should be trying to expose all of the things like "random-if" or is adding "run-if" and "fail-if" all that need to be added (and maybe someone should normalize the existing uses of "fails-if" that are perhaps ignored?). (NB: I see jmaher is not available for needinfos right now.) Note that there's no rush on this as I'm currently only available to mentor someone to fix this, but not actually available to fix this.

Comment 1

[Andrew Halberstadt [:ahal]]

I think fails-if, fuzzy-if, random-if and asserts-if are all from reftest manifests which uses its own bespoke parser. So if you ignore reftest, then run-if and fail-if are the only things missing (I think?).

Reftest is a bit horrific.. the manifests are parsed within the Gecko process via manifest.jsm (so it can use context values from Gecko), but there's a separate Python-based parser here. I think the build system uses this(?) so it's probably good enough for Searchfox. I'd definitely keep reftest in a separate bug either way.

Comment 2

[Andrew Halberstadt [:ahal]]

Also fyi, there's a new effort just getting underway to convert the manifestparser-based manifests to TOML. That may or may not impact your decision to pick this up.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Andrew Halberstadt [:ahal] from comment #2)

Also fyi, there's a new effort just getting underway to convert the manifestparser-based manifests to TOML. That may or may not impact your decision to pick this up.

Oh neat! TOML does seem like a nice direction to go! Thanks for the heads up!

For the existing test info propagation, searchfox just gets the info from test-info-all.json and is not concerned about the actual format, so it's just a question of adding the fields to propagate there in per-file-info.toml and then updating the liquid template section that pulls the data out and emits it.

That said, the switch to TOML should potentially make it easier for searchfox to do syntax and semantic highlighting of the manifest files. In particular, it would be neat for someone to be able to click on, say, "xorigin" and have the context menu display an option like "go to docs" that would take the user to a firefox-source-docs page's header that explains what xorigin means or something along those lines. (I don't believe firefox-source-docs has the token "xorigin" right now, but with an affordance like that it might make sense to add one. Alternately, my experiment with having searchfox index python suggests it wouldn't be too hard to have searchfox understand how to find the right python symbol/doc-block to jump to in the authoritative python source instead.) We could also potentially provide a treeherder link that filters the m-c jobs to the "xorig" jobs, etc.