Open Bug 1935112 Opened 2 months ago Updated 2 months ago

Consider reducing the number of alerts generated from pdf tests

Categories

(Testing :: Talos, task, P2)

task

Tracking

(Not tracked)

People

(Reporter: mayankleoboy1, Unassigned)

References

Details

(Whiteboard: [fxp])

Issue:
PDF performance is to a large extent driven by gfx/layout perf. (i.e. non-JS). Sometimes alerts are generated for regression arising from JS bugs. These alerts:

  • Arise from JS bugs that usually fix far more important (benchmark suites like sp2/sp3 or JS2, fix more important user-facing slowness)
  • have a small regression percent.
  • May only be generated on a single platform Mac
    • Mac numbers are noisy

Got into some discussion with :jandem and he raised these issues.

Some suggestions:

  • Increase the threshold alert for pdf tests.
  • Where the alerts are generated only on a single platform, consider not filing a regression bug but leaving comment on the regressing bug.
  • Reduce the number of pdf tests
    To ensure that pdf tests continue to prioritize gfx/layout performance, the above three suggestions can be implemented only for pdf tests that are JS heavy.

I think we should decrease the number of PDFs we are testing. I believe :calixte was looking into this at one point. :calixte, has there been any progress on that?

Flags: needinfo?(cdenizet)
Severity: -- → S2
Type: enhancement → task
Component: Performance → Talos
Priority: -- → P2

Sorry there is no progress.
That said a good first step could be to increase the threshold, what's the current one ?

Flags: needinfo?(cdenizet)

It's currently at 2%.

Some recent examples of this issue:
bug 1935745
bug 1935753
bug 1933795

See Also: → 1885882

:sparky, could we consider increasing the threshold to 6% ?

Flags: needinfo?(gmierz2)
Depends on: 1936544

I put up a patch for it in 1936544. We shouldn't close this bug though since we should still consider reducing the number of PDFs being tested - there are likely quite a few that have overlapping performance coverage.

Flags: needinfo?(gmierz2)
You need to log in before you can comment on or make changes to this bug.