2004405 - [meta] Improve backend performance for new stats technique

Status: RESOLVED FIXED

(Testing :: PerfCompare, defect, P1)

Description

[Greg Mierzwinski [:sparky]]

The backend performance is quite poor at the moment for the new stats. We need to make some improvements to it. This blocks bug 1982849 since we had to disable the KDE calculations due to performance issues.

Comment 1

[Greg Mierzwinski [:sparky]]

I recently added a URL param to the API call to let us test out the modality analysis in the frontend: https://github.com/mozilla/treeherder/pull/9301

Even with the multiprocessing, large try runs still have network timeouts. We're looking into upgrading our treeherder machines.

Comment 2

[Greg Mierzwinski [:sparky]]

Closing this as complete now. The upgrade to treeherder machines managed to improve the situation quite a bit. Previously, I could not get this comparison to complete, but now it completes in 20s: https://deploy-preview-1027--mozilla-perfcompare.netlify.app/compare-results?baseRev=eced71ab2f72037fa4155f1100d38686b78ca4f6&baseRepo=try&newRev=3e1cf1525236fd775add870cbec5ce4c11a8e636&newRepo=try&framework=13&enable_silverman_kde=true

For future context, this was about 80 unique tasks across talos and browsertime on linux retriggered 20 times each. This was across two pushes for ~3000 tasks in total.

Something to note is that it's possible for there to be situations with more unique tasks, but at least the average use case is good enough for us to stop blocking additional work on modality analysis. We can continue making improvements in other areas.