1749232 - Implement searchfox-tool pipeline commands for graph building and graphviz rendering

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

With bug 1749176 landed, it's now possible to directly access crossref data from rust, putting some of the diagramming goals originally proposed in bug 1339243 and realized in the fancy branch in reach for production.

My general plan for this effort, which will be spun out into multiple bugs that will be filed as progress as made, is:

1. Basic graph support from the command line (this bug):
   • Implement calls-from and calls-to pipeline commands (better names accepted!) that produce singular petgraph graph pipeline values from following the transitive closure of the callees and uses edges, respectively, filtering for the target nodes being things that can receive/make calls.
     • PR 494 implemented a core "traverse" command for "callees".
     • PR 504 implemented a "uses" edge traversal for the "traverse" command including automatic expansion via the overrides/overriddenBy relationship, noting that I arbitrarily biased the edge direction here to favor "uses" and that will likely need more work in the future.
   • Implement a graph command that renders the output to .dot or .svg with minimal effort made to deal with the presentation of the data or expose DOM attributes that express the underlying symbol relationships to searchfox (which will come later).
     • PR 496 implemented a core "graph" command that renders to raw (un-laid out) dot, svg, or laid-out dot.
2. Bug 1761701: Create a new pipeline-server.rs tokio-based web server that effectively exposes searchfox-tool to the web at an undocumented endpoint using the local_index abstract server backend (new bug) that enables seeing SVG graphs. This is intended to provide a means for experimentation but is explicitly not intended to be supported UX; it's expected we'll have a feedback cycle that could result in something else that compiles down to underlying pipeline ops.
   • Each pipeline step will need to explicitly be opted in to being allowed to be used from the web. I don't think there are any commands that are unsafe, but it seems like a prudent architectural choice to make.
   • Path logic in the LocalIndex AbstractServer impl will be updated to use a helper for path construction that provides a hard backstop against directory traversal attacks like stuffing a ton of "../" in a path.
   • Some adversarial test cases will be added to cover the directory traversal attack case and also lexicographically minimum and maximum tokens for identifier and crossref lookup.
   • This server should probably be niced to be a lesser priority than the real web servers and its oom_score_adj potentially manipulated so that if it goes crazy it doesn't take out the real web servers either.
3. Create a calls-between command (new bug). Presumably this is just building a new graph using petgraph's all_simple_paths using the same callees traversal that powers calls-from.
4. Adapt the fancy branch's HierBuilder logic from core_diagram.js to effectively transform the symbol-level graph into a hierarchy-aware representation with clusters so that classes with multiple methods in the graph can have them clustered into a table, etc. (new bug).
   • This need not be a direct port.
5. Adapt the fancy branch's renderToNestedList logic so that we can have an accessible alternate representation of graphs (new bug).
6. Many more steps...

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Attachment: PR: Basic "traverse" command implemented.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Basic graph representation and serialization support is now in via https://github.com/mozsearch/mozsearch/pull/494.

The next step is to support rendering the graph to .dot and .svg. My current plan is to use https://github.com/besok/graphviz-rust as the graphviz generation layer with the standard native graphviz binaries being shelled out to locally. The crate seems to be very well-featured in terms of supporting parsing (which we don't really need but seems very nice to have if we would ever need it) and generation via a macro syntax that's reminiscent of serde_json's json! macro which seems far preferable to a procedural style API. An alternative to shelling out would be to use wasm-compiled versions, but I don't think our threat model necessitates that as the intent is to:

• use the macro layer (and its explicit support for escaping) to ensure that are never doing anything resembling string concatenation to build the dot files
• never serve as an arbitrary renderer of dot files. Any dot files we render will be explicitly generated by mozsearch or be something checked into the tree (in which case we'd ideally make sure those are run under sandboxing, etc.). In any case where we are trying to allow the user to do exploratory stuff to generate a patch to go into the tree or something like that, those will be rendered on the browser.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Attachment: PR: Initial graphviz rendering

Initial graphviz rendering support has now landed in https://github.com/mozsearch/mozsearch/pull/496 allowing searchfox-tool to generate laid-out .dot and .svg output. The underlying invocation of dot unfortunately does lossy conversion to a String of the Vec<u8> output which means png output is no good. I'll remove it as an option from the codebase in the future because there's no actual desire for us to produce/consume png files; we always want svg or laid-out dot.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Resolving this as fixed as step 1 in comment 0 is complete; next step is bug 1761701. This bug's comment 0 will potentially continued to be updated via editing.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Attachment: PR: uses diagramming, improved crossref lookup perf, improved tracing logging