1763241 - Refactor test_check_insta to support comment lines in test file and use a tree hierarchy instead of single dir

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

Currently all the check file inputs and outputs live in a single directory each. This was a minimal first step for pragmatism. We now have enough tests that the attempt at filename-based structure has really hit its limits. Also, the lack of support for comments in the files has been limiting. While the pipeline structure of the tests is inherently very clear about what we're doing, the inability to document the why is limiting, especially as bug 1763005 will allow us to provide a more literate programming style of presentation. So we'll also add support for #-prefixed comment lines.

The general structure is going to look like:

• analysis
  • cpp
  • rust
  • xpidl
• crossref
  • crossref
  • identifiers
  • merge
• web
  • search (I'm going to rename the existing "query" tests to "search" and rename the pipeline command as well)
• fancy - This will hold tests for new capabilities that are still in-development and not yet ready to be surfaced to anyone by default.
  • diagram
    • traverse

Tests of our HTML output will continue to be co-located next to tests of JSON output that they are clones of because this locality seems beneficial for maintenance purposes and sanity-checking. In the future if we adopt something like whole-file HTML expectations, those would probably go under a new top-level "html" directory.

In terms of filenames, thus far I've tried to embed a mangled version of the underlying test source file that gives rise to the (definitions for) the symbols we're looking at. I'll replace these with directories of the same name as the file, with the exception being the "merge" "merge-analyses" test which operates on a file granularity; that will still get some kind of mangled name.

For the new "query" mechanism I've proposed in bug 1762817, that's TBD. It would either go under "fancy/query" or it may get its own sibling directories to "checks/" and "snapshots/" or something to capture that it operates in a different domain where we want to assert over both the constructed pipeline and the results, where the results themselves may include a bunch of extra metadata. For that situation we may end up with significantly more assertion output files per input for purposes of clarity and avoiding the per-file diffs harming usability as some representations may be more prone to others and github really likes to hide diffs that it thinks are too busy.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: mozsearch changes

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: mozsearch-mozilla