Closed Bug 985575 Opened 6 years ago Closed 6 years ago

Investigate creeping 10% regression on CART for OS X 10.6 over March

Categories

(Firefox :: Toolbars and Customization, defect)

x86_64
macOS
defect
Not set

Tracking

()

RESOLVED WONTFIX

People

(Reporter: mconley, Unassigned)

References

Details

(Keywords: perf, Whiteboard: [talos_regression][Australis:P-])

We've noticed that while other platforms tend to be improving their CART performance, OS X 10.6 actually seems to be getting worse.

Over the course of March, there's be a slow but steady creeping up of CART numbers. We're at about 10% since the beginning of March. Graph:

http://graphs.mozilla.org/graph.html#tests=[[309,63,21],[309,63,35],[309,63,25],[309,63,24]]&sel=1392644927460,1395236927460,8.181818181818173,70.9090909090909&displayrange=30&datatype=running

We should try to figure out what's going on here.
Mike is working on some breakdown numbers to have a better grasp of the issue.

Regardless, it seems that Apple is ending 10.6 support: http://www.computerworld.com.au/article/534463/apple_signals_end_os_x_snow_leopard_support/

I think this should probably be low priority unless there's an obvious and/or easy fix without much research into the issue[s].
So here's a zoomed in version of the original graph:

http://graphs.mozilla.org/graph.html#tests=[[309,63,35],[309,63,25],[309,63,24],[309,63,21]]&sel=1393480432127.0815,1395236927460,25.301670279706336,67.80259792349112&displayrange=30&datatype=running

And here's the Datazilla breakdown:

https://datazilla.mozilla.org/?start=1393692666&stop=1395251871&product=Firefox&repository=Firefox&os=mac&os_version=OS%20X%2010.6.8&test=cart&project=talos

Here are the notable blips that I see:

(1)
March 4 (Bug 980032 excuses 5.67% CART regression, Bug 979326 lands):
  5dc091b7e86f -> 6a1587283d47
    -customize-enter.all:       +10% regression (16.32 -> 17.79)
    -customize-enter.half:      +2.63% regression (8.76 -> 8.99)
    -customize-exit.all:        +2.93% regression (18.75 -> 19.30)
    -customize-exit.error:      -10.74% improvement (255.17 -> 227.76)
    -customize-exit.half:       +17.58% regression (10.24 -> 12.04)
    -customize-enter-css.all:   +11.43% regression (30.97 -> 34.51)
    -customize-enter-css.error: -7.00% improvement (43.55 -> 40.5)
    -customize-enter-css.half:  +27.48% regression (25.58 -> 32.61)

(2)
March 4 - 5 (Bug 979034 lands):
  http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e5b09585215f&tochange=ee131fc23afd
    -customize-exit.all:        +4.57% regression (19.26 -> 20.14)
    -customize-exit.error:      +9.05% regression (233.59 -> 254.72)
    -customize-exit.half:       -3.63% improvement (11.86 -> 11.43)

(3)
March 6 (Bug 909349 lands - 9.8% CART regression excused in bug 980639):
  http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b627f45f8d0b&tochange=b0e7f63c2138
    -customize-exit.all:        -22.27% improvement (20.25 -> 15.74)
    -customize-exit.error:      +15.43% regression (257.09 -> 296.76)
    -customize-enter.css.error: +9.33% regression (40.82 -> 44.63)

(4)
March 11 (Bug 980639 lands - but does a poor job of fixing regressions):
  http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9cdaf3f7c601&tochange=ea3a5092c05a
    -customize-exit.all:        +29.58% regression (15.65 -> 20.28)
    -customize-exit.error:      -11.16% improvement (292.19 -> 259.58)

(5)
March 12 (Bug 980639 backed out):
  http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6d4044f9dfff&tochange=a56837cfc67c
    -customize-exit.all:        -22.84% improvement (20.14 -> 15.54)
    -customize-exit.error:      +17.22% regression (258.79 -> 303.35)

I'm pretty sure (1) is explained by bug 980032.

So, here's an explanation for (3), (4), and (5):

Bug 909349 landed in (3), and stayed on central, where we say a 9.8% CART regression. Bug 980639 was filed to deal with that regression, and a patch was landed in bug 980639 to try to fix the regression. That fix caused our CART numbers to go down, but a detailed look showed an increase in the frame painting average, and drop in the error counts. We prefer lower frame painting averages, so we backed out bug 980639. We then decided to accept the 9.8% CART regression - please see Bug 980639 for details.

(2) is the unknown. I suspect that might have to do with bug 979034, but I can't say for certain.
OK, many thanks, Mike. It seems that patches which generally improve perf on other platforms, sometimes hurt 10.6.

I'm closing as wontfix to formalize the decision to not peruse osx 10.6 minor animation regressions, mostly due to the fact that it's hard to justify prioritizing it.

This doesn't mean that we don't care, nor that we shouldn't land a patch if we understand clearly the cause for the minor regression and can fix it relatively easily.

Please reopen if you think this should get prioritized.

Matt, any objections?
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(matt.woodrow)
Resolution: --- → WONTFIX
Nope, that sounds reasonable.
Flags: needinfo?(matt.woodrow)
Adding Bug 990233 comment 10 as reference.

TL:DR; by Andreas Gal: We should not care much about small 10.6-only regressions unless they're likely to indicate a bigger issue which could also affect other platforms.
You need to log in before you can comment on or make changes to this bug.