From a discussion in https://chat.mozilla.org/#/room/#searchfox:mozilla.org just now about the potential performance hits from supporting a tree-sitter tokenizer to get position:sticky nesting for older revisions, I gathered some stats that suggest our "Format mixing" stage probably has a fair amount of room for optimization. Specifically, for the ~56.6kLOC https://searchfox.org/llvm/source/llvm/lib/Target/X86/X86ISelLowering.cpp our (run in parallel, so not as fast as it would be in isolation) rendering had these performance stats: ``` File 'llvm/lib/Target/X86/X86ISelLowering.cpp' Analysis load duration: 16us Per-file info load duration: 4us File contents read duration: 2383us File described duration: 14us Format code duration: 258139us Blame lines duration: 42446us Commit info duration: 21us Format mixing duration: 729219us Total writing duration: 1037747us ``` A less dramatic example is https://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.cpp with stats: ``` File 'layout/generic/nsIFrame.cpp' Analysis load duration: 150571us Per-file info load duration: 2479us File contents read duration: 916us File described duration: 83us Format code duration: 242884us Blame lines duration: 25333us Commit info duration: 19us Format mixing duration: 308795us Total writing duration: 733588us ``` The "Format mixing" timestamp corresponds to [this range of code](https://github.com/mozsearch/mozsearch/blob/fc0b84dfd020ae5669cc6f0b0dd11cf8c27d8706/tools/src/format.rs#L426-L618) which does a variety of things so use of a proper profiler would probably be in order. Maybe we could even automate running perf against this stage for m-c or beta and have it stashed and dumped in such a way that we could add "mozsearch" to the relevant config and have a performance strip replace the coverage strip. (With the added irony that adding the performance strip would likely regress the performance here.) For efficiency purposes, we could: - store the performance data in a tarball we upload to s3 using standard "only upload this if I'm a release run" semantics. - only run (and upload) the m-c job under perf if the revision of the mozsearch checkout has changed. - this would mean we could see perf changes on dev/try runs too, plus the indexed mozsearch run would always have accurate (if not entirely recent) run data without worrying about gratuitously burning whatever overhead is implicit in running perf.
Bug 1791329 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
From a discussion in https://chat.mozilla.org/#/room/#searchfox:mozilla.org just now about the potential performance hits from supporting a tree-sitter tokenizer to get position:sticky nesting for older revisions, I gathered some stats that suggest our "Format mixing" stage probably has a fair amount of room for optimization. Specifically, for the ~56.6kLOC (no-analysis) https://searchfox.org/llvm/source/llvm/lib/Target/X86/X86ISelLowering.cpp our (run in parallel, so not as fast as it would be in isolation) rendering had these performance stats: ``` File 'llvm/lib/Target/X86/X86ISelLowering.cpp' Analysis load duration: 16us Per-file info load duration: 4us File contents read duration: 2383us File described duration: 14us Format code duration: 258139us Blame lines duration: 42446us Commit info duration: 21us Format mixing duration: 729219us Total writing duration: 1037747us ``` A less dramatic example is https://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.cpp with stats: ``` File 'layout/generic/nsIFrame.cpp' Analysis load duration: 150571us Per-file info load duration: 2479us File contents read duration: 916us File described duration: 83us Format code duration: 242884us Blame lines duration: 25333us Commit info duration: 19us Format mixing duration: 308795us Total writing duration: 733588us ``` The "Format mixing" timestamp corresponds to [this range of code](https://github.com/mozsearch/mozsearch/blob/fc0b84dfd020ae5669cc6f0b0dd11cf8c27d8706/tools/src/format.rs#L426-L618) which does a variety of things so use of a proper profiler would probably be in order. Maybe we could even automate running perf against this stage for m-c or beta and have it stashed and dumped in such a way that we could add "mozsearch" to the relevant config and have a performance strip replace the coverage strip. (With the added irony that adding the performance strip would likely regress the performance here.) For efficiency purposes, we could: - store the performance data in a tarball we upload to s3 using standard "only upload this if I'm a release run" semantics. - only run (and upload) the m-c job under perf if the revision of the mozsearch checkout has changed. - this would mean we could see perf changes on dev/try runs too, plus the indexed mozsearch run would always have accurate (if not entirely recent) run data without worrying about gratuitously burning whatever overhead is implicit in running perf.
From a discussion in https://chat.mozilla.org/#/room/#searchfox:mozilla.org just now about the potential performance hits from supporting a tree-sitter tokenizer to get position:sticky nesting for older revisions, I gathered some stats that suggest our "Format mixing" stage probably has a fair amount of room for optimization. Specifically, for the ~56.6kLOC (no-analysis) https://searchfox.org/llvm/source/llvm/lib/Target/X86/X86ISelLowering.cpp our (run in parallel, so not as fast as it would be in isolation) rendering had these performance stats: ``` File 'llvm/lib/Target/X86/X86ISelLowering.cpp' Analysis load duration: 16us Per-file info load duration: 4us File contents read duration: 2383us File described duration: 14us Format code duration: 258139us Blame lines duration: 42446us Commit info duration: 21us Format mixing duration: 729219us Total writing duration: 1037747us ``` A less dramatic example is (has analysis) https://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.cpp with stats: ``` File 'layout/generic/nsIFrame.cpp' Analysis load duration: 150571us Per-file info load duration: 2479us File contents read duration: 916us File described duration: 83us Format code duration: 242884us Blame lines duration: 25333us Commit info duration: 19us Format mixing duration: 308795us Total writing duration: 733588us ``` The "Format mixing" timestamp corresponds to [this range of code](https://github.com/mozsearch/mozsearch/blob/fc0b84dfd020ae5669cc6f0b0dd11cf8c27d8706/tools/src/format.rs#L426-L618) which does a variety of things so use of a proper profiler would probably be in order. Maybe we could even automate running perf against this stage for m-c or beta and have it stashed and dumped in such a way that we could add "mozsearch" to the relevant config and have a performance strip replace the coverage strip. (With the added irony that adding the performance strip would likely regress the performance here.) For efficiency purposes, we could: - store the performance data in a tarball we upload to s3 using standard "only upload this if I'm a release run" semantics. - only run (and upload) the m-c job under perf if the revision of the mozsearch checkout has changed. - this would mean we could see perf changes on dev/try runs too, plus the indexed mozsearch run would always have accurate (if not entirely recent) run data without worrying about gratuitously burning whatever overhead is implicit in running perf.