1783761 - Switch from ad-hoc JS templating / "output" and "dir" output in favor of rust-based liquid templating as part of indexing

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

While figuring out how to best implement the (static) settings UI in bug 1703656 without introducing new technical debt, I realized it's probably a good time to switch from the ad-hoc JS templating we're doing to rust logic using our liquid-templating (which also can be done from JS).

The general setup for output.sh right now is:

• parallel job invokes output-file.rs over all-files
• output-dir.js gets invoked to build the directory listings
• output-template.js gets invoked to build the "search.html" file that router.py uses
• output-help.js gets invoked to build the "help.html" that nginx gets pointed at to be our root /index.html

The JS logic is a combination of:

• output.js which has template string providing the HTML prolog plus a bunch of helpers like for breadcrumbs.
• app domain logic for output-dir.js

The interaction with the static settings page comes from:

• The settings page ideally wants to not introduce yet another copy of the HTML prolog that we have to update all the time.
• There's interesting potential to let any examples in the settings UI be somewhat more realistic by actually embedding real searchfox data into them.
  • This would be something that happens down the road, but could be nice to plan for.
  • This would most sanely probably depend on the searchfox "tests" repo or an "examples" repo for most trees, although it could be neat for mozilla-central to have real-world examples.

Design Considerations

Displaying directory tree listings for specific revisions

Right now searchfox will happily show you files at historical revisions, but you'll get a 404 if you try and see a directory listing at a specific revision. While the changes here probably should not require that we do the revision control magic, it would be good to architect so that this is easier to do when the time comes.

Dynamic queries returning files

Tying in with the above, for example, we already have a PipelineValues::FileMatches which is currently just a vec of FileMatch which is just a path: String, but one could imagine:

• augmenting the information to (extensibly) contain the information required for the current directory display listing.
• adding a searchfox-tool "batch" command that knows how to run an arbitrary (preconfigured?) pipeline multiple times and direct the output to disk.
  • It seems like this could be handled either by using search-files to return just a list of directories which could then drive the batch job, or alternately search-files could return everything but this could then be partitioned by grouping by directory and then running the batch against those resulting groups. Partitioning seems like it should keep things more O(num files) rather than adding any scaling factors that could be magnified if we add extra metadata.
• allowing search-files or some alternate command to basically do git ls-tree to provide the information, although noting that it would obviously not have most extra metadata like test "skip-if" info or any knowledge of __GENERATED__ files.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: Refactorings towards rust-based directory listings

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed PR: Upgrade to clap v3 from clap v2/structopt

I brought us up-to-date on clap; around the creation of searchfox-tool the clap v3 integration of structopt was still a WIP and so I stuck with clap v2/structopt to avoid complications.

This was motivated by figuring out how to implement the batch-render command where it might make sense to explicitly have ((sub-)sub-)commands for clarity rather than just having strings from a hard-coded match branch.

The other obvious candidate for batch-render is the existing "output-file" binary which also makes sense to support for the purposes of "let's dynamically ingest and pre-render some try server outputs". (An interesting question is whether we can easily have nginx do a try_files with failover to the dynamic proxy request for that.)

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch PR: nginx setup fix, progress towards unification of per-file-info derivation with crossref for templating

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch PR: Overhaul the per-file info pipeline.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch-mozilla PR: Expose CONFIG_REPO and more reliably download WPT metadata

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch PR: Switch directory listings to rust from JS

This gets us broadly to what comment 0 proposed; batch-render processes search-files. Much more detail in all the pull requests.

Next PRs are minor to switch over the help.html and search.html templating which is mainly just wrapping them in the boilerplate.

I am not going to remove the JS templating code in this bug, and will leave the invocations commented out in output.sh so that if someone wants to do any A/B testing locally with the old version, they can do so, noting that it may be appropriate to comment out the "checks" mechanism if doing that since the checks will complain about some of the HTML expectation ("insta") checks.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch PR: Move remaining JS templating to rust.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch PR: Clean up templates, footer should be part of scrolling area