Closed Bug 978844 Opened 10 years ago Closed 10 years ago

Customization Animation Tests - MacOSX 10.6 (rev4) - 3.3% regression on mozilla-inbound

Categories

(Core :: Graphics, defect)

x86
macOS
defect
Not set
normal

Tracking

()

RESOLVED WONTFIX

People

(Reporter: jmaher, Unassigned)

References

Details

(Keywords: perf, regression, Whiteboard: [talos_regression])

here is a notification of the regression:
https://groups.google.com/forum/#!topic/mozilla.dev.tree-management/vcu4ObvaThU

Here is a description of the test:
https://wiki.mozilla.org/Buildbot/Talos/Tests#TART.2FCART

This is a newer test that we invested some time in to make it more reliable and useful!
The changesets are all Matt's, which makes some sense I suppose... But if it's just 10.6, maybe we should ignore it.
any thoughts on this, time is our worse enemy.
Flags: needinfo?(matt.woodrow)
I haven't had a chance to look into this properly.

Given that it's 3% and only limited to OSX 10.6, then I doubt it matters.
Flags: needinfo?(matt.woodrow)
two votes for ignoring it- anybody object to resolving this?
Flags: needinfo?(vdjeric)
Agreed that 3% on 10.6 only isn't worth the time.

However, if we look at the past month, while windows and linux improved continuously, osx 10.8 roughly stayed the same, and 10.6 creeps up.

http://graphs.mozilla.org/graph.html#tests=[[309,63,21],[309,63,35],[309,63,25],[309,63,24]]&sel=1392644927460,1395236927460,8.181818181818173,70.9090909090909&displayrange=30&datatype=running

In general, it seems that osx regressed on Mar 1st (this bug), Mar 5th, Mar 7th and Mar 12th, while it had improvements on Mar 3rd and Mar 11th.

So we can probably not look into these specific changesets, but in general, customoze animation on osx isn't getting better.

Mike?
Flags: needinfo?(vdjeric) → needinfo?(mconley)
I would tend to agree - although I just needinfo'd you on another bug that might have us take another 10.6 CART hit (bug 941309). Wondering if you wouldn't mind commenting there.

I'm not sure what else to say here. It's unfortunate that the numbers are increasing. I also don't exactly know what heuristics we're using to determine when performance regressions are worth tracking down and when we can leave them be, and it'd be nice if that was more explicit.

Not sure what else I have to add to the matter.
Flags: needinfo?(mconley)
ok, we have some form of agreement here.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
(In reply to Mike Conley (:mconley) from comment #6)
> ... I also don't exactly know what heuristics we're using to
> determine when performance regressions are worth tracking down and when we
> can leave them be, and it'd be nice if that was more explicit.

Nothing explicit, these would be judgment calls. I also don't think we can define explicit thresholds. dev.tree-management triggers an automatic notification if the change is certain enough (more than 2x stddev iirc). What happens next is up to us.

E.g. this 3% regression on osx 10.6 only probably isn't worth the time to look into, but overall creeping regressions on osx 10.6 might be worth a consideration, depending on resources, priorities, etc.
You need to log in before you can comment on or make changes to this bug.