1097376 - Poor performance in Ember app Discourse

Status: NEW

(Core :: JavaScript Engine, defect)

Description

[Paul Gregg]

Clone of the V8 Issue 2935 https://code.google.com/p/v8/issues/detail?id=2935
Linked from the Coding Horror's: The Tablet Turning Point blog post http://blog.codinghorror.com/the-tablet-turning-point/


> Chrome [and Firefox] on Android is performing poorly in the Discourse app
> Thread:  http://meta.discourse.org/t/why-is-discourse-so-slow-on-android/8823
> Video: http://www.youtube.com/watch?v=Jd7i2NeZARU 
> 
> To test:
> A running instance of Discourse is at http://meta.discourse.org - to see the slowness I suggest entering a topic with at least 20 posts in it.
> The entire source is available at https://github.com/discourse/discourse - if you want to run it in development mode, there is a VM to make that easy.


So both Chrome & Firefox on Android seem to have similar performance numbers, about x7 to x9 slower on modern Android hardware compared to modern PC & iOS devices.
 -> iPad Air 2 = 224 ms, Firefox 33.1 on Note 3 SM-N9005 (Android 4.4.2) = 1685ms, 195 std dev.

Comment 1

[Boris Zbarsky [:bzbarsky]]

The benchmark at http://samsaffron.github.io/ember-performance/?ember=1.8.0 might be a good start...

Comment 2

[Till Schneidereit [:till]]

Numbers for Nightly, Chrome Canary, Safari 8.0, all on OS X 10.10:

Nightly (2014-11-11):
.--------------------------------------------------------------------.
|            Ember Performance Suite - Ember 1.8.0-beta.5            |
|--------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean |   Mean   | Std Dev |  Max  |
|-------------------|--------|----------|----------|---------|-------|
| Object.create()   |     50 | 56.22ms  | 56.3ms   | 2.98ms  | 66ms  |
| Render List       |     20 | 741.87ms | 745.15ms | 71.38ms | 917ms |
| Template Bindings |     40 |          | 1.55ms   | 0.97ms  | 5ms   |
| HTML Bindings     |     40 |          | 0.32ms   | 0.46ms  | 1ms   |
| Complex List      |     40 | 261.09ms | 263.35ms | 37.85ms | 420ms |
'--------------------------------------------------------------------'

Chrome Canary (41.0.2216.0):
.--------------------------------------------------------------------.
|            Ember Performance Suite - Ember 1.8.0-beta.5            |
|--------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean |   Mean   | Std Dev |  Max  |
|-------------------|--------|----------|----------|---------|-------|
| Object.create()   |     50 | 55.98ms  | 57.4ms   | 13.94ms | 95ms  |
| Render List       |     20 | 465.41ms | 467.35ms | 44.31ms | 585ms |
| Template Bindings |     40 |          | 1.07ms   | 0.56ms  | 4ms   |
| HTML Bindings     |     40 |          | 0.05ms   | 0.21ms  | 1ms   |
| Complex List      |     40 | 176.21ms | 179.75ms | 39.88ms | 309ms |
'--------------------------------------------------------------------'


Safari 8.0:
.--------------------------------------------------------------------.
|            Ember Performance Suite - Ember 1.8.0-beta.5            |
|--------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean |   Mean   | Std Dev |  Max  |
|-------------------|--------|----------|----------|---------|-------|
| Object.create()   |     50 | 18.23ms  | 18.28ms  | 1.32ms  | 23ms  |
| Render List       |     20 | 373.65ms | 373.95ms | 14.7ms  | 400ms |
| Template Bindings |     40 |          | 1.3ms    | 1.43ms  | 10ms  |
| HTML Bindings     |     40 |          | 0.25ms   | 0.43ms  | 1ms   |
| Complex List      |     40 | 118.63ms | 118.8ms  | 6.53ms  | 153ms |
'--------------------------------------------------------------------'

Analysis to follow.

Comment 3

[Till Schneidereit [:till]]

Object.create():

Ember.Object.create is used to instantiate Ember classes:
http://emberjs.com/api/classes/Ember.Object.html#method_create

(All numbers in % of total time.)

Highlights:
- no time to speak of in (not under) JITted code
- 41% under obj_defineProperty
- 18% under SetPropertyIC::update
- 9% under js_Array
- 7% under jit::createThisForFunction
- 4.5% under GetPropertyIC::update
- 4.5% under obj_create
- 4.5% under ArgumentsObject::createForIon
- 2.3% under JSScript::maybeSweepTypes


Profile of a run with an unminified version of Ember that shows similar performance characteristics to the minified one: http://people.mozilla.org/~bgirard/cleopatra/#report=2b55c5c195eac2c91f6ec11d3f34f5c39643822d

I unfortunately don't get script locations for all JS frames, so the analysis of what Ember is doing and how long which part of it takes is incomplete.


88% of time is spent under makeCtor/Class[1].

Another 9.1% is spent under some JITted function, with all of js::CreateThisForFunction and JSScript::maybeSweepTypes under there.

Of the time under makeCtor/Class, 36% are under meta[2]. All the time under SetPropertyIC::update and under obj_create is in here, as is a third of the time under obj_defineProperty.

Another 11.5% is under apply[3]. I thought replacing this with direct calls to Function#apply would allow our apply optimization to kick in, but that's not the case: it causes a significant (15% - 20%) slowdown of the overall test. So something prevents that optimization from working.

The rest is spent under functions for which I don't have source location. I think it's significant that we create ArgumentsObject instances here: it looks like the Ember people went to considerable lengths in trying to prevent just that.


[1] https://github.com/emberjs/ember.js/blob/429e634f593f719571c8639572e704f4fae7ea26/packages/ember-runtime/lib/system/core_object.js#L80
[2] https://github.com/emberjs/ember.js/blob/429e634f593f719571c8639572e704f4fae7ea26/packages/ember-metal/lib/utils.js#L288
[3] https://github.com/emberjs/ember.js/blob/429e634f593f719571c8639572e704f4fae7ea26/packages/ember-metal/lib/utils.js#L842

Comment 4

[Boris Zbarsky [:bzbarsky]]

Note that once we fix those issues we should measure on Android, since apparently Chrome on Android does a lot worse here than desktop Chrome.  So just to make sure we don't have the same platform-dependence going on.

Comment 5

[Till Schneidereit [:till]]

Render List of Items:

I'm having a hard time analyzing this, as the Gecko Profiler seems to give me numbers that can't all be quite right or, at the very least, aren't all that informative.

It's also not that easy to select sample ranges from just the code under test: the test case has a cleanup part which is way more expensive than the part under test, at least in Nightly: each iteration is in danger of triggering the slow script dialog, so takes ~10s, but the part under test takes only ~1s. I thought this was maybe caused by GC/CC, but no: only GC happens for each test iteration (CC only sometimes), and both have max pause times of ~10ms and total times < 30ms.

One thing that stuck out to me is that there's a substantial perf difference between the minified and the unminified version. That's true for all browsers, but to varying degrees:

Nightly: 1550ms unminified, 904ms minified
Chrome: 1870ms unminified, 488ms minified
Safari: 567ms unminified, 383ms minified

So Chrome gets more than 3x as fast, while both Nightly and Safari gain far less. Of course, for Safari that isn't that bad, seeing as how it's really fast either way.


What I was otherwise able to piece together from profiler[1] and Instruments runs is that most time is spent in SpiderMonkey, with DOM manipulations and rendering far less relevant.

We again spend lots of time under {Get,Set}PropertyIC::update and js_Array, as for Object.create. Those taken together seem to make up close to half of the entire run time.

Additionally, SetObjectElement and GetElementIC::update show up somewhat prominently.

Comment 6

[Till Schneidereit [:till]]

(In reply to Boris Zbarsky [:bz] from comment #4)
> Note that once we fix those issues we should measure on Android, since
> apparently Chrome on Android does a lot worse here than desktop Chrome.  So
> just to make sure we don't have the same platform-dependence going on.

Agreed. It seems to me, though, that most of my findings so far should hold regardless of the platform. I might be wrong about that, of course.

Comment 7

[Till Schneidereit [:till]]

Complex List largely exercises the same code paths as Render List of Items, and the profile looks similar. It doesn't exhibit the long pauses outside the code under test, though, so that's interesting.

Profile:
http://people.mozilla.org/~bgirard/cleopatra/#report=c0a82d9af8bb42e166fd1b2a6c7577248f108e27

I didn't look into the other two tests, as they run so fast that (I hope) their performance isn't as important.

Comment 8

[Paul Gregg]

A few clarifications & jotting ideas down:

This bug was reported specifically for the Discourse/Ember performance problems on Android, understand if the investigation involves looking for bottlenecks using desktop versions, I presume there may be a fair few dependency bugs & this becomes a tracking bug.

I guessed the iPad Air 2 number was using the Ember 1.8.0b5 Complex List benchmark, that's what the Note 3 value I measured & stated.
 -> Yep, Jeff Atwood mentions it in https://meta.discourse.org/t/why-is-discourse-so-slow-on-android/8823/75


More complete Ember benchmark results for Android (on Note 3 with power saving off):
 -> Looks like we are in the same ballpark to other browsers on Android, but still very slow compared to iOS devices

Firefox for Android 33.1
 -> Slow script popup came up during Render List (clicked Don't show & reran)
.------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5              |
|------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   |
|-------------------|--------|-----------|-----------|----------|--------|
| Object.create()   | 50     | 311.87ms  | 317.2ms   | 58.75ms  | 483ms  |
| Render List       | 20     | 3401.66ms | 3407.1ms  | 196.07ms | 3958ms |
| Template Bindings | 40     | 13.42ms   | 15.92ms   | 10.21ms  | 45ms   |
| HTML Bindings     | 40     |           | 0.92ms    | 0.68ms   | 3ms    |
| Complex List      | 40     | 1530.41ms | 1534.95ms | 119.91ms | 1797ms |
'------------------------------------------------------------------------'

Chrome 38.0.2125.114 2125114
Blink 537.36 (@183147)
JavaScript V8 3.28.71.15
.------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5              |
|------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   |
|-------------------|--------|-----------|-----------|----------|--------|
| Object.create()   |     50 | 438.31ms  | 442.62ms  | 66.72ms  | 744ms  |
| Render List       |     20 | 3292.22ms | 3309.05ms | 367.66ms | 4778ms |
| Template Bindings |     40 | 37.42ms   | 40.9ms    | 16.26ms  | 66ms   |
| HTML Bindings     |     40 |           | 0.95ms    | 1.75ms   | 11ms   |
| Complex List      |     40 | 1225.84ms | 1241.9ms  | 213.54ms | 1897ms |
'------------------------------------------------------------------------'

(Chromium) Internet 1.5.28 1528
Blink 537.36 (Unknown URL@0)
JavaScript V8 webtech-jsengine:3.18.5.15:E.1.4-git:f64e311
.------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5              |
|------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   |
|-------------------|--------|-----------|-----------|----------|--------|
| Object.create()   |     50 | 378.47ms  | 381.22ms  | 47.17ms  | 524ms  |
| Render List       |     20 | 3382.55ms | 3383.65ms | 87.83ms  | 3698ms |
(Template & HTML Bindings wouldn't run)
| Complex List      |     40 | 1907.11ms | 1910.82ms | 125.21ms | 2484ms |
'------------------------------------------------------------------------'



Would be good to get a modern iOS device & Android phone/tablet that are roughly equivalent in computing power, otherwise it's comparing apples to oranges. I think the research by Jeff Atwood from Discourse showed iOS devices running Safari blew Chrome Android & Firefox for Android out of the water.
 -> Though Jeff commented that on the Nexus 7, Firefox for Android is far behind Chrome?? http://discourse.codinghorror.com/t/the-tablet-turning-point/2746/21

This is all based on the user story of viewing a Discourse forum with 20+ posts.
https://meta.discourse.org/t/why-is-discourse-so-slow-on-android/8823
 -> Seems Chrome & Firefox on Android take twice as long to load & to show posts.

I'd suggest the important thing is to close the benchmark result gap on Safari (on iOS or OSX), for Firefox & in particular on Android.
 -> Presume step 1 is to see if the Ember framework can be made faster
  -> See if there is any low hanging fruit/pathological cases using profiling.
  -> Try to speed up benchmark to get close to fastest browser on same hardware (would be Safari on OSX).
  -> Do the same for Android benchmark results, hopefully get comparable performance (based on ratio of computing power of Android/iOS devices).
 -> Hopefully the above fixes Discourse's performance problems, but more may need to be done to get Safari performance levels.

May involve some Ember/Discourse developer evangelism to suggest coding changes that will improve their performance.
 -> Looks like Mozilla just rolled out a Discourse instance for Mozilla Guides, so this could also be a dogfood issue.

Comment 9

[Till Schneidereit [:till]]

(In reply to Paul Gregg from comment #8)
> A few clarifications & jotting ideas down:

Thanks, that's helpful.

> This bug was reported specifically for the Discourse/Ember performance
> problems on Android, understand if the investigation involves looking for
> bottlenecks using desktop versions, I presume there may be a fair few
> dependency bugs & this becomes a tracking bug.

Yes, the investigation so far focused on desktop because it's far simpler to take measurements there. As bz noted in comment 4, we absolutely have to also do measurements and investigation on Android.

> Firefox for Android 33.1
>  -> Slow script popup came up during Render List (clicked Don't show & reran)

I'm pretty sure that that's the issue I described in comment 5, so the slow script dialog is shown for a part of code that's not under test. Depending on how important that part is for overall importance, it might actually be most pressing.

> Would be good to get a modern iOS device & Android phone/tablet that are
> roughly equivalent in computing power, otherwise it's comparing apples to
> oranges. I think the research by Jeff Atwood from Discourse showed iOS
> devices running Safari blew Chrome Android & Firefox for Android out of the
> water.

Given how well Safari is doing on the desktop, I would assume that not all of that but a large part is due to WebKit/JSC doing really well on these tasks.

> I'd suggest the important thing is to close the benchmark result gap on
> Safari (on iOS or OSX), for Firefox & in particular on Android.

Right.

>  -> Presume step 1 is to see if the Ember framework can be made faster
>   -> See if there is any low hanging fruit/pathological cases using
> profiling.
>   -> Try to speed up benchmark to get close to fastest browser on same
> hardware (would be Safari on OSX).
>   -> Do the same for Android benchmark results, hopefully get comparable
> performance (based on ratio of computing power of Android/iOS devices).
>  -> Hopefully the above fixes Discourse's performance problems, but more may
> need to be done to get Safari performance levels.

Sounds like a good strategy to me, yes. I started looking into optimizing Ember code, but I might not have the time to thoroughly investigate and submit patches to the project.

> May involve some Ember/Discourse developer evangelism to suggest coding
> changes that will improve their performance.
>  -> Looks like Mozilla just rolled out a Discourse instance for Mozilla
> Guides, so this could also be a dogfood issue.

We have several Discourse instances, I think, so indeed, we're at least dogfooding it.

Comment 10

[Till Schneidereit [:till]]

Numbers for today's Nightly:
.--------------------------------------------------------------------.
|            Ember Performance Suite - Ember 1.8.0-beta.5            |
|--------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean |   Mean   | Std Dev |  Max  |
|-------------------|--------|----------|----------|---------|-------|
| Object.create()   |     50 | 37.25ms  | 37.29ms  | 1.84ms  | 47ms  |
| Render List       |     20 | 730.8ms  | 732.9ms  | 55.27ms | 828ms |
| Template Bindings |     40 |          | 1.45ms   | 1.16ms  | 6ms   |
| HTML Bindings     |     40 |          | 0.25ms   | 0.43ms  | 1ms   |
| Complex List      |     40 | 249.68ms | 253.35ms | 48.7ms  | 429ms |
'--------------------------------------------------------------------'

Object.create() improved substantially, but is still half as fast as Safari. Template Bindings and HTML Bindings *might* have improved a bit, but it's not entirely clear. Render List and Complex List haven't improved meaningfully or at all.

Comment 11

[Dave Herman [:dherman]]

>  -> Presume step 1 is to see if the Ember framework can be made faster

It's nice to contribute patches back to Ember, but since it looks like this is providing us with information where competitors (esp JSC) are faster, the higher value to us is using this data to identify optimization opportunities within SpiderMonkey.

This should have positive effects across the board. Frameworks like Ember are a pretty good barometer for real-world code since they deal with so many and such varied apps, so the more we can give them automatic optimizations, the more the web at large should benefit.

(I'm sure they'd appreciate it too, since they have to go through such nasty contortions to get portable performance across engines.)

Comment 12

[Paul Gregg]

Sorry Dave, that is actually what I had meant (just wasn't worded very well). Looking back now, I think Till got the same impression as you.
What I meant was:
> I'd suggest the important thing is to close the benchmark result gap on Safari (on iOS or OSX), for Firefox & in particular on Android.
> -> Presume step 1 is to see if the Ember framework can be made faster *by optimising Firefox & Spidermonkey*

My last paragraph was trying to suggest that if there is some really bad code in Ember that performs poorly in FF but fantastic in JSC, and that optimising FF for it seems like a poor ROI, then it /might/ be worth suggesting to the Ember project some coding changes, but I'm guessing that's a last resort type thing.


As an aside, it appears this bug is turning into a "Make desktop performance competitive on Ember.js Performance Suite". Maybe there should be a few dependant bugs created? Thoughts?

Bug 1097376 Poor performance in Ember app Discourse (Android)
  |-> Poor performance on Ember Performance Suite (Android)
    |-> Poor performance on Ember Performance Suite (Desktop)

Also, "Poor performance" was chosen only because of cloning the V8 bug, maybe "Make X faster" or "Make X performance competitive" would be better & less negative...

Comment 13

[Till Schneidereit [:till]]

(In reply to Paul Gregg from comment #12)
> Sorry Dave, that is actually what I had meant (just wasn't worded very
> well). Looking back now, I think Till got the same impression as you.

Right, I had understood that indeed. To my knowledge, nobody here did anything but SpiderMonkey optimizations to make things faster, so we might as well not have misunderstood you :)

> As an aside, it appears this bug is turning into a "Make desktop performance
> competitive on Ember.js Performance Suite". Maybe there should be a few
> dependant bugs created? Thoughts?

I guess that'd make sense. It also doesn't seem to make a huge amount of difference - the work that needs doing will still be largely the same, and I still think that as long as we're not competitive with Safari on Desktop, we haven't harvested all the low-hanging fruit and don't need to move over to Android-specific analysis.

> Also, "Poor performance" was chosen only because of cloning the V8 bug,
> maybe "Make X faster" or "Make X performance competitive" would be better &
> less negative...

Well, our performance on the benchmark we've been looking at, at least, is rather poor ...

I had high hopes for some of the more recent changes making a substantial difference, but alas, not much has changed since late November:

.--------------------------------------------------------------------.
|            Ember Performance Suite - Ember 1.8.0-beta.5            |
|--------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean |   Mean   | Std Dev |  Max  |
|-------------------|--------|----------|----------|---------|-------|
| Object.create()   |     50 | 34.33ms  | 34.4ms   | 2.19ms  | 43ms  |
| Render List       |     20 | 685.45ms | 688.15ms | 60.64ms | 803ms |
| Template Bindings |     40 |          | 1.5ms    | 0.92ms  | 6ms   |
| HTML Bindings     |     40 |          | 0.35ms   | 0.47ms  | 1ms   |
| Complex List      |     40 | 225.93ms | 228.05ms | 34.22ms | 366ms |
'--------------------------------------------------------------------'

Comment 14

[Boris Zbarsky [:bzbarsky]]

OK, so I did some profiling of the "Render List" benchmark.

First, this is almost entirely DOM manipulation and JS.  The layouty bits are no more than about 5ms out of the 435ms the benchmark runs for over here.

So what takes the time?

1)  About 20% jitcode.
2)  About 12% innerHTML sets, almost all of it parsing.  A noticeable amount of the parsing
    time (over 1/4!) is the malloc/free calls in
    nsHTML5TreeBuilder::startTokenization/endTokenization.
3)  About 9% js::SetObjectElement doing a NativeSetProperty (adding a prop, in fact) and
    then JSObject::setFlags.
4)  5% str_split_string, but 45 of that is GC.
5)  4% DOM insertBefore.  This is a lot more than the comparable number for Safari (1%, on
    a smaller baseline), afaict.
6)  3% DOM cloneNode.  About half cloning, half JS wrapping.
7)  3% DOM appendChild.  Again, seems to be more than Safari.
8)  3% jit::SetPropertyIC::update, mostly doing the set.
9)  3% childNodes getter, mostly creating the JS reflector.
10) 3% js::GetElement
11) 3% js::GetProperty
12) 3% minor GC off InterruptCheck.
13) 2% jit::GetElementIC::update, mostly doing the get.
14) 2% under js_fun_apply

There's various long-tail stuff too.

Comment 15

[Boris Zbarsky [:bzbarsky]]

One other meta-comment: If we accept that insertBefore took about 4%, from a total time of 400ms, that's 16ms.  The testcase performs 5003 insertBefore calls, so that would be about 3us per insertBefore call.  That seems pretty steep, though not surprising to me given all the work insertBefore does.  :(

Comment 16

[Paul Gregg]

It appears that there has been some wins on the benchmark & a regression for "Render List" (probably the most important). I presume the benchmark &/or device environment has not changed (I followed the same procedure, on the same hardware with no major OS changes).
 -> I did notice higher numbers on first run, so used results from second runs (which were sometimes much better, but maybe first run is a good "worst case").

Repeating Comment 8 tests again:

Firefox for Android 40.0   (was v33.1)
.---------------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5                       |
|---------------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   | % diff |
|-------------------|--------|-----------|-----------|----------|--------|--------|
| Object.create()   |     50 | 236.79ms  | 238.78ms  | 35.37ms  | 442ms  | -24.7% |
| Render List       |     20 | 3471.19ms | 3494.15ms | 447.19ms | 5290ms | +2.6%  |
| Template Bindings |     40 | 8.39ms    | 10.25ms   | 8.97ms   | 48ms   | -35.6% |
| HTML Bindings     |     40 |           | 2.02ms    | 2.21ms   | 14ms   | +120%  |
| Complex List      |     40 | 1219.58ms | 1256.02ms | 318.6ms  | 1870ms | -18.2% |
'---------------------------------------------------------------------------------'

Chrome 44.0.2403.133 (Official Build) (32-bit)  (was v38.0.2125.114)
Blink 537.36 (@199588)
JavaScript V8 4.4.63.25
.---------------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5                       |
|---------------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   | % diff |
|-------------------|--------|-----------|-----------|----------|--------|--------|
| Object.create()   |     50 | 391.45ms  | 391.94ms  | 19.62ms  | 457ms  | -11.5% |
| Render List       |     20 | 2781.81ms | 2784.1ms  | 112.96ms | 3023ms | -15.9% |
| Template Bindings |     40 | 16.23ms   | 17.95ms   | 8.34ms   | 35ms   | -56.1% |
| HTML Bindings     |     40 |           | 0.6ms     | 0.58ms   | 2ms    | -36.8% |
| Complex List      |     40 | 1041.99ms | 1048.07ms | 125.31ms | 1697ms | -15.6% |
'---------------------------------------------------------------------------------'

(Chromium) Internet 1.5.28 1528
Blink 537.36 (Unknown URL@0)
JavaScript V8 webtech-jsengine:3.18.5.15:E.1.4-git:f64e311
.------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5              |
|------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   |
|-------------------|--------|-----------|-----------|----------|--------|
| Object.create()   |     50 | 375.41ms  | 376.84ms  | 33.68ms  | 482ms  |
| Render List       |     20 | 3380.64ms | 3381.7ms  | 83.29ms  | 3552ms |
(Template & HTML Bindings wouldn't run)
| Complex List      |     40 | 1864.33ms | 1866.05ms | 79.97ms  | 1996ms |
'------------------------------------------------------------------------'
 -> good to see that the default browser results are pretty much the same as last time, as the version hasn't changed.

Comment 17

[Jan de Mooij [:jandem]]

I pulled a new version of these tests + ember.js and this time I'm able to run them locally. There are a lot of new tests as well. I'll look into this the coming weeks.

Comment 18

[Jan de Mooij [:jandem]]

It's on my TODO list, but I'm trying to get an empty request queue so I'll cancel the NI.

Comment 19

[Olli Pettay [:smaug][[e-mail]]]

I profiled this a gain, and Render List (see comment 14) seems to be now pretty much all in JS or GC

Comment 20

[Olli Pettay [:smaug][[e-mail]]]

But would be good if someone could re-profile on OSX too, since malloc seems to be particularly bad there.

Comment 21

[(no longer active)]

(In reply to Olli Pettay [:smaug] (review queue closed. Please ask reviews from other DOM peers) from comment #20)
> But would be good if someone could re-profile on OSX too, since malloc seems
> to be particularly bad there.

Needinfo Sean since he asked for expensive JS stuff coming up in profiles that aren's GSuite.

Comment 22

[Jan de Mooij [:jandem]]

Bug 1352486 has some more up-to-date information. I've been profiling that one this week.

Comment 23

[Shu-yu Guo [:shu]]

It's been a while since this was reported, let's remeasure with other Ember stuff being worked on.

Comment 24

[Paul Gregg]

With the arrival of Firefox 57, I thought it was a good time to rerun the tests & see how we are going.

I followed the same procedure (killing all apps, turning off power save mode, turning off screen timeout).
Unfortunately for consistency, the OS is now Android 5.0 (unsure if this would have affected the numbers much).
I've included the first run number as well this time.

Firefox for Android 57
.-----------------------------------------------------------------------------------------.
|             Ember Performance Suite - Ember 1.8.0-beta.5                                |
|-----------------------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean   | Std Dev  |  Max   | % v40  | % v57  |
|-------------------|--------|-----------|----------|----------|--------|--------|--------|
| Object.create()   |     50 | 269.43ms  | 272.54ms | 44.65ms  | 472ms  |        |        |
| Object.create()   |     50 | 185.18ms  | 195.22ms | 63.72ms  | 387ms  | -24.1% | -40.6% |
| Render List       |     20 | 2976.92ms | 2986.5ms | 238.38ms | 3401ms |        |        |
| Render List       |     20 | 2857.13ms | 2866.3ms | 233.29ms | 3360ms |   2.0% | -16.0% |
| Template Bindings |     40 | 10.42ms   | 13.12ms  | 9.45ms   | 41ms   |        |        |
| Template Bindings |     40 | 15.66ms   | 18.89ms  | 11.88ms  | 56ms   | -37.5% |  16.7% |
| HTML Bindings     |     40 |           | 1.22ms   | 1.57ms   | 7ms    |        |        |
| HTML Bindings     |     40 |           | 0.82ms   | 0.49ms   | 2ms    |        |        |
| Complex List      |     40 | 966ms     | 980.8ms  | 198.44ms | 1766ms |        |        |
| Complex List      |     40 | 905.71ms  | 910.82ms | 106.04ms | 1288ms | -20.3% | -40.8% |
'-----------------------------------------------------------------------------------------'
% = compared to original tests using v33.1

Chrome
62.0.3202.84 (Official Build) (32-bit)
JavaScript	V8 6.2.414.37
.------------------------------------------------------------------------------------------.
|              Ember Performance Suite - Ember 1.8.0-beta.5                                |
|------------------------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean    | Std Dev  |  Max   | % v44  | % v62  |
|-------------------|--------|-----------|-----------|----------|--------|------- |--------|
| Object.create()   |     50 | 126.32ms  | 132.36ms  | 43.68ms  | 285ms  |        |        |
| Object.create()   |     50 | 179.3ms   | 186.96ms  | 52.2ms   | 352ms  | -10.7% | -59.1% |
| Render List       |     20 | 1906.3ms  | 1922.35ms | 273.01ms | 2889ms |        |        |
| Render List       |     20 | 1809.05ms | 1811.4ms  | 93.09ms  | 2011ms | -15.5% | -45.1% |
| Template Bindings |     40 | 12.1ms    | 14.62ms   | 11.15ms  | 67ms   |        |        |
| Template Bindings |     40 | 9.16ms    | 10.85ms   | 6.45ms   | 31ms   | -56.6% | -75.5% |
| HTML Bindings     |     40 |           | 0.45ms    | 0.73ms   | 3ms    |        |        |
| HTML Bindings     |     40 |           | 0.55ms    | 1.58ms   | 10ms   |        |        |
| Complex List      |     40 | 616.21ms  | 621.2ms   | 85.09ms  | 968ms  |        |        |
| Complex List      |     40 | 577.51ms  | 578.47ms  | 33.61ms  | 669ms  | -15.0% | -52.9% |
'------------------------------------------------------------------------------------------'
% = compared to original tests using v38

Samsung Internet 6.2.01.12
.--------------------------------------------------------------------------------.
|             Ember Performance Suite - Ember 1.8.0-beta.5                       |
|--------------------------------------------------------------------------------|
|       Name        | # Runs | Geo Mean  |   Mean   | Std Dev  |  Max   | % diff |
|-------------------|--------|-----------|----------|----------|--------|--------|
| Object.create()   |     50 | 245.35ms  | 251.98ms | 57.4ms   | 415ms  |        |
| Object.create()   |     50 | 270.92ms  | 274.16ms | 42.06ms  | 393ms  | -28.4% |
| Render List       |     20 | 2369.35ms | 2383.8ms | 289.23ms | 3512ms |        |
| Render List       |     20 | 2292.91ms | 2294.8ms | 94.4ms   | 2586ms | -32.2% |
| Template Bindings |     40 | 13.83ms   | 16.92ms  | 10.56ms  | 43ms   |        |
| Template Bindings |     40 | 11.15ms   | 13.6ms   | 9.21ms   | 41ms   |        |
| HTML Bindings     |     40 |           | 0.6ms    | 0.66ms   | 3ms    |        |
| HTML Bindings     |     40 |           | 0.92ms   | 1.42ms   | 7ms    |        |
| Complex List      |     40 | 771.17ms  | 780.9ms  | 155.87ms | 1706ms |        |
| Complex List      |     40 | 753.34ms  | 755.22ms | 54.14ms  | 915ms  | -60.5% |
'--------------------------------------------------------------------------------'
% = compared to original tests using v1.5


Firefox 57 has definitely gotten quicker on the benchmarks, but while we were pretty close to Chrome, they have improved much more than Firefox. So they have increased the speed difference.