Consider reducing the number of alerts generated from pdf tests
Categories
(Testing :: Talos, task, P2)
Tracking
(Not tracked)
People
(Reporter: mayankleoboy1, Unassigned)
References
Details
(Whiteboard: [fxp])
Issue:
PDF performance is to a large extent driven by gfx/layout perf. (i.e. non-JS). Sometimes alerts are generated for regression arising from JS bugs. These alerts:
- Arise from JS bugs that usually fix far more important (benchmark suites like sp2/sp3 or JS2, fix more important user-facing slowness)
- have a small regression percent.
- May only be generated on a single platform Mac
- Mac numbers are noisy
Got into some discussion with :jandem and he raised these issues.
Some suggestions:
- Increase the threshold alert for pdf tests.
- Where the alerts are generated only on a single platform, consider not filing a regression bug but leaving comment on the regressing bug.
- Reduce the number of pdf tests
To ensure that pdf tests continue to prioritize gfx/layout performance, the above three suggestions can be implemented only for pdf tests that are JS heavy.
Updated•2 months ago
|
Comment 1•2 months ago
|
||
I think we should decrease the number of PDFs we are testing. I believe :calixte was looking into this at one point. :calixte, has there been any progress on that?
Updated•2 months ago
|
Comment 2•2 months ago
|
||
Sorry there is no progress.
That said a good first step could be to increase the threshold, what's the current one ?
Comment 3•2 months ago
|
||
It's currently at 2%.
Reporter | ||
Comment 4•2 months ago
|
||
Some recent examples of this issue:
bug 1935745
bug 1935753
bug 1933795
Comment 5•2 months ago
|
||
:sparky, could we consider increasing the threshold to 6% ?
Comment 6•2 months ago
|
||
I put up a patch for it in 1936544. We shouldn't close this bug though since we should still consider reducing the number of PDFs being tested - there are likely quite a few that have overlapping performance coverage.
Description
•