1791329 - Consider optimizing output-file/formatting "format mixing" stage

Status: NEW

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

From a discussion in https://chat.mozilla.org/#/room/#searchfox:mozilla.org just now about the potential performance hits from supporting a tree-sitter tokenizer to get position:sticky nesting for older revisions, I gathered some stats that suggest our "Format mixing" stage probably has a fair amount of room for optimization.

Specifically, for the ~56.6kLOC (no-analysis) https://searchfox.org/llvm/source/llvm/lib/Target/X86/X86ISelLowering.cpp our (run in parallel, so not as fast as it would be in isolation) rendering had these performance stats:

File 'llvm/lib/Target/X86/X86ISelLowering.cpp'
  Analysis load duration: 16us
  Per-file info load duration: 4us
  File contents read duration: 2383us
  File described duration: 14us
  Format code duration: 258139us
  Blame lines duration: 42446us
  Commit info duration: 21us
  Format mixing duration: 729219us
  Total writing duration: 1037747us

A less dramatic example is (has analysis) https://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.cpp with stats:

File 'layout/generic/nsIFrame.cpp'
  Analysis load duration: 150571us
  Per-file info load duration: 2479us
  File contents read duration: 916us
  File described duration: 83us
  Format code duration: 242884us
  Blame lines duration: 25333us
  Commit info duration: 19us
  Format mixing duration: 308795us
  Total writing duration: 733588us

The "Format mixing" timestamp corresponds to this range of code which does a variety of things so use of a proper profiler would probably be in order.

Maybe we could even automate running perf against this stage for m-c or beta and have it stashed and dumped in such a way that we could add "mozsearch" to the relevant config and have a performance strip replace the coverage strip. (With the added irony that adding the performance strip would likely regress the performance here.) For efficiency purposes, we could:

• store the performance data in a tarball we upload to s3 using standard "only upload this if I'm a release run" semantics.
• only run (and upload) the m-c job under perf if the revision of the mozsearch checkout has changed.
• this would mean we could see perf changes on dev/try runs too, plus the indexed mozsearch run would always have accurate (if not entirely recent) run data without worrying about gratuitously burning whatever overhead is implicit in running perf.