2.3% regression in Kraken audio-beat-detection, audio-fft on Windows XP on 2013-04-25

RESOLVED WORKSFORME

Status

()

Core
JavaScript Engine
RESOLVED WORKSFORME
5 years ago
5 years ago

People

(Reporter: mbrubeck, Unassigned)

Tracking

({perf, regression})

23 Branch
x86
Windows XP
perf, regression
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox23 fixed)

Details

(Reporter)

Description

5 years ago
There was a 2.3% regression in Kraken on mozilla-inbound yesterday which showed up on Windows XP:
http://graphs.mozilla.org/graph.html#tests=[[232,131,1]]&sel=1366844366870,1366998531576&displayrange=7&datatype=running

It's not clear if other platforms showed any regression.  Windows 7, for example, is too noisy to tell clearly, though some statistical analysis might help answer the question:
http://graphs.mozilla.org/graph.html#tests=[[232,131,12],[232,131,1]]&sel=1366844366870,1366998531576&displayrange=7&datatype=running

Here's is the regression range:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=16ebd3f796c1&tochange=ac39efa583f7

The most relevant changes seem to be bug 864727 and bug 863898.

dev-tree-management thread:
https://groups.google.com/d/topic/mozilla.dev.tree-management/eYbx4xVIDzQ/discussion
The code that was changed in those bugs had better not be exercised in Kraken.  ;)

Are these pgo builds or regular builds?  It's at least _possible_ that changes in one part of the code affect pgo of other parts of the code...
Perhaps Bug 865544 - It landed on inbound 14:34:05 PDT and adds, I believe, an extra QI to all webidl object creations?
Nah, that only affects JS-implemented WebIDL, which we currently have 0 of. :)
(Reporter)

Comment 4

5 years ago
(In reply to Boris Zbarsky (:bz) from comment #1)
> Are these pgo builds or regular builds?

Regular (non-PGO) builds.

(In reply to Jan-Ivar Bruaroey [:jib] from comment #2)
> Perhaps Bug 865544 - It landed on inbound 14:34:05 PDT and adds, I believe,
> an extra QI to all webidl object creations?

No, the regression definitely happened well before bug 865544 landed.  There are 12 test runs on inbound *before* that landing that are all regressed.

The regression range seems very clear (an abrupt increase of about 7x the standard deviation), but even if I widen it by several pushes there are no /js/src changes that could plausibly be the cause.

I retriggered the Talos run on the changeset before the regression, to check for external (e.g. infrastructure) causes:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=16ebd3f796c1&jobname=5.1.*dromaeojs
(Reporter)

Comment 5

5 years ago
(In reply to Matt Brubeck (:mbrubeck) from comment #4)
> I retriggered the Talos run on the changeset before the regression, to check
> for external (e.g. infrastructure) causes:
> https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=16ebd3f796c1&jobname=5.1.
> *dromaeojs

The retrigger appears to rule out any external causes.  From the email:

    Previous: avg 3326.633 stddev 10.983 of 12 runs up to revision 16ebd3f796c1
    New     : avg 3402.575 stddev 9.847 of 12 runs since revision ac39efa583f7

The retrigger on the pre-regression changeset had a result of 3327.20.
That really makes no sense.  There should be no WebIDL anything during Kraken tests....

Matt, do you know where the source for the relevant tests lives, or who would know that?  Also, whether we have per-test data or just an overall number?
(Reporter)

Comment 7

5 years ago
(In reply to Boris Zbarsky (:bz) from comment #6)
> Matt, do you know where the source for the relevant tests lives, or who
> would know that?

The test code is here:
https://hg.mozilla.org/build/talos/file/e405acebbbf9/talos/page_load_test/kraken

The main "harness" code is here:
http://hg.mozilla.org/build/talos/file/e405acebbbf9/talos/pageloader/chrome/pageloader.js

> Also, whether we have per-test data or just an overall number?

Search for "ai-astar;" in the log to see the raw data.

Before:
https://tbpl.mozilla.org/php/getParsedLog.php?id=22244100&tree=Mozilla-Inbound

After:
https://tbpl.mozilla.org/php/getParsedLog.php?id=22245152&tree=Mozilla-Inbound

At a glance, audio-beat-detection and audio-fft look significantly slower, while the other tests look basically unchanged.
(Reporter)

Updated

5 years ago
Summary: 2.3% regression in Kraken benchmark on Windows XP on 2013-04-25 → 2.3% regression in Kraken audio-beat-detection, audio-fft on Windows XP on 2013-04-25
Thanks!

audio-beat-detection certainly regressed by a good bit there, order of 15%+.

I'm going to look into what's going on there; a regression that big ought to be easy to hunt down...
And of course the regression is completely absent for me locally on Mac...

I guess I'll try to see if I can bisect on XP builds on try or something.  :(
Well, if I believe the numbers there the problem is one of the first 4 parts of bug 864727.  I guess I'll try pushing those and see how things look.
So if my try pushes:

https://tbpl.mozilla.org/?tree=Try&rev=cf38c3a92bc3
https://tbpl.mozilla.org/?tree=Try&rev=e1db69603c0d
https://tbpl.mozilla.org/?tree=Try&rev=7237e06b977a

are to be believed, the regression comes from this changeset: http://hg.mozilla.org/integration/mozilla-inbound/rev/3011f288bfe7

I really have no idea why this would affect the Kraken script, much less only on XP.

And if I look at current inbound builds the audio-beat-detection times are back down to their "pre-regression" levels...
(Reporter)

Comment 13

5 years ago
(In reply to Boris Zbarsky (:bz) from comment #12)
> And if I look at current inbound builds the audio-beat-detection times are
> back down to their "pre-regression" levels...

Indeed.  The un-regression range is:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=1936616a7f41&tochange=11f11d9ca555
(Bug 798165 and other cycle-collection changes by mccr8)

Feel free to close this bug as WORKSFORME unless someone feels it is useful to dig deeper for understanding...
No longer blocks: 863898
status-firefox23: affected → fixed
Depends on: 798165
The un-regression range is just as nonsensical as the regression range.  :(
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WORKSFORME
Haha.  If anything, I'd expect my nsTimeout patch to make things slower.  Though maybe we're more likely to inline the nsTimeout AddRef and Release. :)
The point is, during the parts of kraken that are being timed your nsTimeout stuff should not be exercised at all.
tracking-firefox23: ? → ---
You need to log in before you can comment on or make changes to this bug.