1302727 - Slow new tab animation makes Firefox appear sluggish vs. Chrome

Status: RESOLVED FIXED

(Firefox :: Theme, defect)

Description

[William Lachance (:wlach)]

So I'd been meaning to open a bug about this since forever. I think we ought to speed up Firefox's "open new tab" animation, as it makes Firefox feel somewhat sluggish compared to Chrome even when it's just as fast. As nice as the animation is, a browser just feels snappier when the operation completes faster -- the difference is easy to see if you compare the two videos I'm going to attach. I believe there's an abundance of UX research confirming this finding. E.g.:

http://blog.teamtreehouse.com/perceived-performance

"0.1 seconds — Operations that are completed in 100ms or fewer will feel instantaneous to the user. This is the gold standard that you should aim for when optimising your websites.

1 second — Operations that take 1 second to finish are generally OK, but the user will feel the pause. If all of your operations take 1 second to complete, your website may feel a little sluggish."

From examining the video, it looks like we're well over the 100ms threshold for the new tab animation.

mconley -- you said you knew where the CSS dictating the animation length was. Could you point me there so I can have a try at tweaking it?

Comment 1

[William Lachance (:wlach)]

Attachment: New tab animation on Firefox

Comment 2

[William Lachance (:wlach)]

Attachment: New tab animation on Chrome

Comment 3

[Mike Conley (:mconley) (:⚙️)]

I believe that's controlled here: http://searchfox.org/mozilla-central/rev/6b94deded39a868f248a492e74e05733d6c4ed48/browser/base/content/browser.css#131

Comment 4

[William Lachance (:wlach)]

Attachment: New tab animation on Firefox

Actually let's use default Firefox (not dev edition) for the base example, as the dark theme distracts from using it as a baseline on a dev build.

Comment 5

[William Lachance (:wlach)]

Attachment: New tab animation on Firefox with animation times reduced from 200ms -> 100ms

Ok, tested changing the setting mconley linked to locally. It works. My first try, following that article I linked to, was to use a timing of 100ms (0.1s) to ease in the new tab. Still enough to see it as animating, but no longer appears sluggish to me. See attached video.

Chrome appears to use an even lower threshold (maybe 50 or 75?). We could consider that too.

Comment 6

[William Lachance (:wlach)]

Attachment: Bug 1302727 - Shorten tab opening/closing animation to 100ms

Review commit: https://reviewboard.mozilla.org/r/78768/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/78768/

Comment 7

[William Lachance (:wlach)]

Did a try build, samples here:

https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-b291690101817ce8ac3478a256bab8263e32e3c9/

Comment 8

[William Lachance (:wlach)]

I understand that we probably need UX sign off on this bug, but I have no idea who to ask. :) Hopefully :mconley or someone else can point the right person here.

Comment 9

[Mike Conley (:mconley) (:⚙️)]

Hey phlsa, what do you think about shorting this transition time? wlach has a recording of the change in comment 5. Here's a try build for OS X if you want to test it: https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-b291690101817ce8ac3478a256bab8263e32e3c9/try-macosx64/firefox-51.0a1.en-US.mac.dmg

Comment 10

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8791331 [details]
Bug 1302727 - Shorten tab opening/closing animation to 100ms

https://reviewboard.mozilla.org/r/78768/#review77390

This looks fine, but I'm going to wait until we hear back from UX before signing off.

Comment 11

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

Looks good to me!
We're currently having a wider discussion about motion in the UI, which might change some things there as well, but in the meantime I don't see a reason to block this.

Comment 12

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8791331 [details]
Bug 1302727 - Shorten tab opening/closing animation to 100ms

https://reviewboard.mozilla.org/r/78768/#review77548

UX has signed off, and the code looks good to me.

One thing we might want to do before landing though is to trigger some TART runs and compare against base to see if the numbers change any. We'll probably want to alert jmaher in case the numbers are about to hop up for some reason (maybe due to fewer frames over a shorter duration).

Comment 13

[William Lachance (:wlach)]

(In reply to Mike Conley (:mconley) - (needinfo me!) from comment #12)
> Comment on attachment 8791331 [details]
> Bug 1302727 - Shorten tab opening/closing animation to 100ms
> 
> https://reviewboard.mozilla.org/r/78768/#review77548
> 
> UX has signed off, and the code looks good to me.
> 
> One thing we might want to do before landing though is to trigger some TART
> runs and compare against base to see if the numbers change any. We'll
> probably want to alert jmaher in case the numbers are about to hop up for
> some reason (maybe due to fewer frames over a shorter duration).

Sounds good, I triggered a try run for tart and tps:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f6916f1e583

I don't think it matters if the numbers change a bit (up or down)

Comment 14

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #13)
> I don't think it matters if the numbers change a bit (up or down)

Sorry, didn't finish that thought. I don't think it matters that much if the benchmark numbers change a bit (up or down), as we've already measured the perceptual impacts of this change.

Comment 15

[William Lachance (:wlach)]

Yeah, based on the try run it looks like this will cause a TART "regression". However as mentioned above, it's the perceptual impact that counts. We just need to rebaseline the numbers.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=af1ba72b88093f7dc3557c999081ff4914c46fce&newProject=try&newRevision=0f6916f1e583cb9d355563c71591f55cef3a029e&framework=1&filter=tart

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fde4f3db7ee6
Shorten tab opening/closing animation to 100ms r=mconley

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fde4f3db7ee6

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

fast new tab == slow tart:
== Change summary for alert #3238 (as of September 16 2016 07:20 UTC) ==

Regressions:

 20%  tart summary osx-10-10 opt            9.27 -> 11.15
 14%  tart summary windowsxp opt e10s       6.57 -> 7.51
 14%  tart summary linux64 opt              6.93 -> 7.87
 13%  tart summary windows7-32 opt          6.51 -> 7.34
 10%  tart summary osx-10-10 opt e10s       10.33 -> 11.38
 10%  tart summary windows8-64 opt e10s     6.4 -> 7.04
  9%  tart summary windows8-64 opt          5.92 -> 6.44
  8%  tart summary windowsxp opt            6.48 -> 7.02

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=3238

Do let me know if this was expected

Comment 19

[William Lachance (:wlach)]

(In reply to Joel Maher ( :jmaher) from comment #18)

> Do let me know if this was expected

Yes, see above.

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

sorry, I should have been more clear- I saw we were expecting a regression, does it make sense given the code that changed that we had such a large change on osx and differences between e10s/non-e10s and platforms?

for example, why is osx change e10s half of non e10s (and linux is as well), when win8/winxp have higher e10s regressions?

Comment 21

[Jared Wein [:jaws] (please needinfo? me)]

Why does speeding up the animation result in slower TART numbers? wlachance, could it be that this is requesting too much from Gecko at one time? It doesn't make sense to me that we would blindly be OK with accepting a regression in our TART metric, especially when we are specifically shortening the animation.

Comment 22

[William Lachance (:wlach)]

(In reply to Joel Maher ( :jmaher) from comment #20)
> sorry, I should have been more clear- I saw we were expecting a regression,
> does it make sense given the code that changed that we had such a large
> change on osx and differences between e10s/non-e10s and platforms?
> 
> for example, why is osx change e10s half of non e10s (and linux is as well),
> when win8/winxp have higher e10s regressions?

I'll try to answer this based on the documentation on TART (and my slim knowledge of the platform). I could be wrong.

Compare view on windows xp (where we noticed the least difference overall):

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=14a6fa8db8fd67a4d3726672e86ed0d52dcd35b6&newProject=autoland&newRevision=fde4f3db7ee65ec33ac056a1d53c4ebe09e4dbe7&originalSignature=8203c19ab37fa72112a91624e94f48387aba4d4c&newSignature=8203c19ab37fa72112a91624e94f48387aba4d4c&framework=1

Compare view on OS X 10.10 non-e10s (where we noticed the most):

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=14a6fa8db8fd67a4d3726672e86ed0d52dcd35b6&newProject=autoland&newRevision=fde4f3db7ee65ec33ac056a1d53c4ebe09e4dbe7&originalSignature=068b77fdbe85d85b74b670620b0101f62059c39d&newSignature=068b77fdbe85d85b74b670620b0101f62059c39d&framework=1

A few things that jump out at me in both of these comparisons:

* preload tests have less of a regression that non-preload tests (filter on "preload-no" vs "preload-yes")
* fade tests show either an improvement or no observable regression (filter on "fade")

My *guess* (based on reading the documentation for the various tests) is that this is because TART isn't actually just testing animation smoothness. Implicitly, a bunch of stuff is also happening when a new tab is opened besides the animation. This operation probably slows down the animation *before*, but eventually completes (after which the animation becomes "smoother"). Since we're animating over a shorter interval now, the "non tab animation" parts of TART have a greater impact on the result. 

The fact that the "fade" tests show almost no regression also supports this conclusion. I would also expect e10s to show smaller regressions, because some of the new tab operation is now taking place in another process (which would have less of a performance impact)

:mconley, does this analysis make sense to you?

Comment 23

[William Lachance (:wlach)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #21)
> Why does speeding up the animation result in slower TART numbers? wlachance,
> could it be that this is requesting too much from Gecko at one time? It
> doesn't make sense to me that we would blindly be OK with accepting a
> regression in our TART metric, especially when we are specifically
> shortening the animation.

I don't really know what "too much" means, but I am now convinced we should at least understand what's going on (though I'm 99% sure this is fine). Did an initial investigation, NI'ing mconley to confirm.

Comment 24

[Avi Halachmi (:avih)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #21)
> Why does speeding up the animation result in slower TART numbers?

Please define "speeding up the animation".

The TART test consists of different animation use cases, and 3 metrics per animation, all of which get an equal weight in the final number.

Per animation TART measures:

- The "Error", i.e. how much longer it took to complete the animation compared to how much it should have taken. The "should have" part is calculated here and IIRC was last modified by mconley or according to instructions by him: https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tart/addon/content/tart.js#463

- "All" - average frame interval - overall duration (from the trigger) divided by the number of frames.

- "Half" - average frame interval over half of the "should have" duration - at the end of the animation. E.g. if the animation should take 400ms, then the average interval over the last 200ms of the animation. This intended to show the throughput after the "heavy lifting" which can happen at the beginning of an animation.


All those 3 metrics are measures for each animation scenario, e.g. opening/closing a new tab with and without newtab preload, triggering the css tab animation without actually opening a new tab, new tab with different DPIs, different text at the tab title, with and without an icon, etc.

Here's the win7 comparison from the alert link at comment 18: https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=14a6fa8db8fd67a4d3726672e86ed0d52dcd35b6&newProject=autoland&newRevision=fde4f3db7ee65ec33ac056a1d53c4ebe09e4dbe7&originalSignature=a7ea2dec670ad95527de39e37c5fea01a2393694&newSignature=a7ea2dec670ad95527de39e37c5fea01a2393694&framework=1


For instance we see that the animation without preload seems to have regressed more than the animation with preload.

We see that the pure tab open/close CSS animation (without actually opening/closeg a tab) seems to have improved.

So we'll have to assess whether or not those numbers are real, and whether or not we should do anything about them, including modifying TART, if required.

Comment 25

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Avi Halachmi (:avih) from comment #24)
> (In reply to Jared Wein [:jaws] (please needinfo? me) from comment #21)
> > Why does speeding up the animation result in slower TART numbers?
> 
> Please define "speeding up the animation".

The transition duration was reduced to less than half of the previous value.

Comment 26

[Avi Halachmi (:avih)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #25)
> The transition duration was reduced to less than half of the previous value.

Well.. this would indeed typically result in less than half of the frames - which arguably subjectively looks worse as an animation.

As noted earlier, typically when you trigger an animation it takes some time for "heavy lifting" of whatever happens right after the trigger, and that stuff typically interferes with the animation itself.

So yes, overall, on average, it makes sense that there are less frame per second of animation since this initial duration is unaffected by the shortening of the animation duration.

It would seem to me that this is mostly what the numbers represent.

Knowing this tradeoff, the question which we have to answer is: did we know this is how it's going to end up, and regardless, do we still prefer it over a longer duration despite this tradeoff?

Comment 27

[William Lachance (:wlach)]

(In reply to Avi Halachmi (:avih) from comment #26)

> Knowing this tradeoff, the question which we have to answer is: did we know
> this is how it's going to end up, and regardless, do we still prefer it over
> a longer duration despite this tradeoff?

UX signed off on the animation as portrayed in the attachments, which is what we landed.

At such a short interval, you're not going to notice a few dropped frames -- the mind fills in the blanks. If you break Chrome's animation down, it has even less frames than we do in our shortened version (I think just 2 intermediate frames before it displays the end results). The important thing is how it feels from the perspective of someone who wants to *use* the product. No matter how slick an animation might be, it gets pretty old after the 10th time.

Comment 28

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #27)
> UX signed off on the animation as portrayed in the attachments, which is
> what we landed.
> 
> ... The important thing
> is how it feels from the perspective of someone who wants to *use* the
> product.

Absolutely, the numbers only told us there's a measurable difference, but ultimately this needs a human to evaluate and make a decision.

That being said, did UX sign off after observing and comparing the animation side by side on different platforms and with different types of systems?

For instance, this is likely to have affected slower systems relatively much more, possibly to a degree where it's possible that there won't be any visible animation at all.

> No matter how slick an animation might be, it gets pretty old after
> the 10th time.

Dunno, this is really off topic, but if it's slick and short enough to not annoy, personally I still enjoy it more than without animation.

Comment 29

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Avi Halachmi (:avih) from comment #26)
> (In reply to Jared Wein [:jaws] (please needinfo? me) from comment #25)
> > The transition duration was reduced to less than half of the previous value.
> 
> Well.. this would indeed typically result in less than half of the frames -
> which arguably subjectively looks worse as an animation.

Yes, that makes sense. Sorry for my confusion earlier.

Comment 30

[William Lachance (:wlach)]

(In reply to Avi Halachmi (:avih) from comment #28)
> (In reply to William Lachance (:wlach) from comment #27)
> > UX signed off on the animation as portrayed in the attachments, which is
> > what we landed.
> > 
> > ... The important thing
> > is how it feels from the perspective of someone who wants to *use* the
> > product.
> 
> Absolutely, the numbers only told us there's a measurable difference, but
> ultimately this needs a human to evaluate and make a decision.
> 
> That being said, did UX sign off after observing and comparing the animation
> side by side on different platforms and with different types of systems?

I assume they just looked at the screen recordings from MacOS X, so no. Just for testing, I compared before/after on a windows vm with its cpu turned way down (to 30%), but I don't know how representative this would be of a system in the real world.

Comment 31

[William Lachance (:wlach)]

Attachment: Old (200ms+) new tab animation on Firefox on a slow windows vm

Comment 32

[William Lachance (:wlach)]

Attachment: New tab animation on a slow windows vm with timings reduced to 100ms

Comment 33

[Avi Halachmi (:avih)]

Re the screen captures at comment 31/32, both look far from 60fps to me, thus probably not representing what many people observe.

This could be due to the fact that it runs in a VM, possibly without proper HW layers acceleration, or due to the recording itself which misses frames.

Bottom line, I'd avoid using these specific captures for the purpose of decision making on this.

Comment 34

[William Lachance (:wlach)]

Generated some videos on :mconley's old netbook (Asus Aspire One - Atom 1.66ghz, 1gb memory):

Old behaviour (200ms+ tab open): https://drive.google.com/open?id=0B1JDLRVWauyebFVxR3QwQ3dKT3M
New behaviour (100ms tab open): https://drive.google.com/file/d/0B1JDLRVWauyeVkdIS0RrTXVKMms

Comment 35

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #34)
> Generated some videos on :mconley's old netbook (Asus Aspire One - Atom
> 1.66ghz, 1gb memory):
> 
> Old behaviour (200ms+ tab open):
> https://drive.google.com/open?id=0B1JDLRVWauyebFVxR3QwQ3dKT3M
> New behaviour (100ms tab open):
> https://drive.google.com/file/d/0B1JDLRVWauyeVkdIS0RrTXVKMms

To me both look equally terrible, i.e. there's barely any animation visible at all. Is this also what you observed on screen and at the video?

If that's the case, then on this system the new duration doesn't make it worse because it's already the worst possible scenario also before the change.

100ms animation, especially where some/significant duration of it doesn't animate at all, makes one think whether no animation at all would be better than this.

I'm not saying that you should, but IMO if you really want to understand what people think about such change, you should find few different platforms/systems which are neither top end nor low end, have a before/after build, and let several people play with it themselves and tell you which they like more.

Also, videos don't tell everything. For instance they don't show the lag from the trigger until it actually starts animating, and it could have some weight as well in the overall satisfaction, and possibly different weights with different overall duration.

I'm not against this change, and I won't object to taking the regression, but at least from whet I've seen here, it seems to me that there could be more cases to assess.

Comment 36

[William Lachance (:wlach)]

(In reply to Avi Halachmi (:avih) from comment #35)
> (In reply to William Lachance (:wlach) from comment #34)
> > Generated some videos on :mconley's old netbook (Asus Aspire One - Atom
> > 1.66ghz, 1gb memory):
> > 
> > Old behaviour (200ms+ tab open):
> > https://drive.google.com/open?id=0B1JDLRVWauyebFVxR3QwQ3dKT3M
> > New behaviour (100ms tab open):
> > https://drive.google.com/file/d/0B1JDLRVWauyeVkdIS0RrTXVKMms
> 
> To me both look equally terrible, i.e. there's barely any animation visible
> at all. Is this also what you observed on screen and at the video?

Yeah, they are both pretty awful.

> If that's the case, then on this system the new duration doesn't make it
> worse because it's already the worst possible scenario also before the
> change.
> 
> 100ms animation, especially where some/significant duration of it doesn't
> animate at all, makes one think whether no animation at all would be better
> than this.

I agree, though I'm not sure if it would make a ton of difference. Firefox is pretty much borderline unusable on this platform in any case. :( Navigating, scrolling pages, etc. is just incredibly janky.

Above it said that UX was going to reconsider animations in the product in general, I'm sure they will look at this area as well at that point.

Anecdotally I am finding that the change makes Firefox feel a fair bit snappier to me, even (maybe especially?) on my old 2011 laptop.

Comment 37

[Mike Conley (:mconley) (:⚙️)]

Ideally, we would have done some kind of SHIELD study, and instrumented the CSS using some prefs to get a sense of whether or not users noticed the change, and if it improved perceived performance.

We're not really in the state where these sort of light-weight experiments can be done without spinning up a bunch of machinery. Maybe soon.

In the meantime, I think we should stick the course with the 100ms change, and see what happens. It's trivial to revert if we change our minds.

Comment 38

[Chris H-C :chutten]

So... would large changes in FX_TAB_ANIM_OPEN_PREVIEW_FRAME_INTERVAL_MS [1] and FX_TAB_ANIM_ANY_FRAME_INTERVAL_MS [2] be expected and explained by this change, too? 

[1]: http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/608/alerts/?from=2016-09-17&to=2016-09-17
[2]: http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/847/alerts/?from=2016-09-17&to=2016-09-17

Comment 39

[Avi Halachmi (:avih)]

(In reply to Chris H-C :chutten from comment #38)
> So... would large changes in FX_TAB_ANIM_OPEN_PREVIEW_FRAME_INTERVAL_MS [1]
> and FX_TAB_ANIM_ANY_FRAME_INTERVAL_MS [2] be expected and explained by this
> change, too? 

Yes, it measures a similar thing to what TART measures. The remaining question is whether or not most people would prefer it over the previous duration.

Comment 40

[Mike Conley (:mconley) (:⚙️)]

(In reply to Chris H-C :chutten from comment #38)
> So... would large changes in FX_TAB_ANIM_OPEN_PREVIEW_FRAME_INTERVAL_MS [1]
> and FX_TAB_ANIM_ANY_FRAME_INTERVAL_MS [2] be expected and explained by this
> change, too? 
> 
> [1]:
> http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/608/
> alerts/?from=2016-09-17&to=2016-09-17
> [2]:
> http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/847/
> alerts/?from=2016-09-17&to=2016-09-17

Yes, I believe so. By reducing the duration of the animation, we've reduced the number of frames that will be produced.

There's work that goes one when a tab is opened, and it occurs while the tab open animation is underway. Because there are fewer frames, it means we have a higher likelihood of the animation intersecting the work, which is my front-runner theory on why we're seeing this difference here.

Comment 41

[Avi Halachmi (:avih)]

(In reply to Mike Conley (:mconley) - (needinfo me!) from comment #40)
> Yes, I believe so. By reducing the duration of the animation, we've reduced
> the number of frames that will be produced.

It doesn't report frame count, but rather frames per second, similar to TART. But because a bigger percentage of the duration is now without animation (despite probably being the same in absolute terms), then the reported FPS drops, rightfully.