987532 - (Silk) [meta] Project Silk on B2G.

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[Vincent Lin[:vilin]]

The idea of Project Butter is to make display smoother. It includes vsync monitor/observer manager/observers, input event resampling, vsync-triggered RefreshDriver timer, vsync-triggered compositor and triple-buffering.

This meta bug is used to trace these modifications belongs to difference modules.

Bug 980241 - Vsync-triggered RefreshDriver
Bug 987523 - Vsync-triggered CompositorParent
Bug 970751 - Deliver input events more consistently per composite to make scrolling smoother
Bug 987527 - Register Vsync monitor to HWComposer and notify to gecko/hal 
Bug 987529 - Implement an observer manager for vsync in gecko/hal

Comment 1

[Andreas Gal :gal]

Very nice. Please make sure to move all the gfx work into the Graphics component and the input events stuff should be in Widget:Gonk.

Comment 2

[Vincent Lin[:vilin]]

design document
https://etherpad.mozilla.org/vsync-butter

Comment 3

[u459114]

Add bug 987529 into dependency list.

All the following work of SiLK is based on implementation of bug 987529

Comment 4

[u459114]

Add bug 930939 in dependency list.

In attachment 8437591 [details], I try to describe why Input handling OOMT is important for SiLK.

Comment 5

[Avi Halachmi (:avih)]

This bug has B2G at its title, but desktop would have a lot to gain from such system as well (windows specifically, because OS X already blocks on glSwap even without perfect timing, and I'm not sure about Linux).

How much does the work here apply for desktop Firefox? If reasonably but not fully, is there a desktop equivalent of this bug/plan?

Comment 6

[u459114]

(In reply to Avi Halachmi (:avih) from comment #5)
> This bug has B2G at its title, but desktop would have a lot to gain from
> such system as well (windows specifically, because OS X already blocks on
> glSwap even without perfect timing, and I'm not sure about Linux).
> 
> How much does the work here apply for desktop Firefox? If reasonably but not
> fully, is there a desktop equivalent of this bug/plan?

Hi Avi,
Here is vsync dispatch flow attachment 8429109 [details].
There are two platform dependent objects
1. (HW vsync)The HWComposer rectangle. at the bottom of this diagram, is B2G only. On B2G, HWComposer receive vsync hw event and dispatch to the client - VsyncDispatcher.
To porting this mechanism on different platforms, we need to construct another back-end to receive HW vsync event and poke VsyncDispatcher at right time. 
2. (Vsync observer) InputDispatcher - each nsAppShell has its own way of input event handling. 

Current plan is to make it work and fine tune on B2G first. After that, then we will move forward to the other platforms.

Comment 7

[Avi Halachmi (:avih)]

Thanks for the info. Any rough plan/hope for the timelines of this? E.g. are we hoping to start looking at desktops in weeks? months? years?

Comment 8

[u459114]

https://wiki.mozilla.org/B2G/Roadmap
1. B2G 2.1 land project silk with preference off.
2. B2G 2.2 turn on by default.

Ideally, most engineer work should be done in milestone #1(due: Oct 13, 2014). Depend on the status of project silk in the end of 2.1, we can make a decision whether kick off platform support at that moment. Is this make sense to you?

Comment 9

[Avi Halachmi (:avih)]

I was just trying to get a rough idea of the roadmap, which you provided (and I like that it seems at the block diagram that many components are shared between platforms - at least at this level).

If it makes sense or not has a lot to do with resources and priorities, and I'm probably unaware of enough of those to answer that. If you were asking for my opinion about focusing on one platform first, then it sounds like a good plan, and it's probably also worth bumping it against the desktop gfx teams to make sure it will fit nicely, which I'm sure you've done as well.

Comment 10

[Sandip Kamat]

Hi Peter/CJ: Can we define some benchmarks/metrics for Project Silk for 2.1?

Comment 11

[Peter Chang[:pchang]]

ni myself to provide the measurement of project silk.

Comment 12

[Peter Chang[:pchang]]

(In reply to Sandip Kamat from comment #10)
> Hi Peter/CJ: Can we define some benchmarks/metrics for Project Silk for 2.1?

Since Project Silk is off by default for 2.1, I think we should meet the following conditions.
a. Green on try server
b. No performance impact, like app launch time and scrolling
c. Green on try server when SW vsync(bug 1043822) turns on

In 2.2, Project Silk is on by default, we not only need to meet above conditions but also the following items.

d. Frame uniformity

Mason and Jerry, any items need to be considered for Project Silk?

Comment 13

[Mason Chang [Inactive] [:mchang]]

Hi Peter,

When you say (c), do we have any requirements for hardware vsync or can it be landed by 2.1?

Other than that, it looks fine by me. We have some difficulty in measuring (d), but as long as we have improved uniformity, I'm ok with that requirement. I don't have a good answer yet on a number on exactly how to measure uniformity.

Thanks!

Comment 14

[Peter Chang[:pchang]]

(In reply to Mason Chang [:mchang] from comment #13)
> Hi Peter,
> 
> When you say (c), do we have any requirements for hardware vsync or can it
> be landed by 2.1?
> 
> Other than that, it looks fine by me. We have some difficulty in measuring
> (d), but as long as we have improved uniformity, I'm ok with that
> requirement. I don't have a good answer yet on a number on exactly how to
> measure uniformity.
> 
> Thanks!

For the item c, it tries to verify the Project Silk with SW vsync on try server, like b2g emulator. So far we don't have real device with KK on try server.

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

Jerry, let's set the design document/wiki page to this bug as attachment, and then ask for a review from :jrmuizel, :bas and :roc for it.  That way we can track the r+ for the design in a single place.

Comment 16

[Sandip Kamat]

(In reply to peter chang[:pchang][:peter] from comment #12)
> (In reply to Sandip Kamat from comment #10)
> > Hi Peter/CJ: Can we define some benchmarks/metrics for Project Silk for 2.1?
> 
> Since Project Silk is off by default for 2.1, I think we should meet the
> following conditions.
> a. Green on try server
> b. No performance impact, like app launch time and scrolling
> c. Green on try server when SW vsync(bug 1043822) turns on
> 
> In 2.2, Project Silk is on by default, we not only need to meet above
> conditions but also the following items.
> 
> d. Frame uniformity
> 
> Mason and Jerry, any items need to be considered for Project Silk?


Given that and b need to happen anyways, and c adds value, Does this effectively mean "phase 1" scope in 2.1 is mostly around bug 1043822?

Comment 17

[Milan Sreckovic [:milan] (needinfo for best results)]

I like the 2.1 definition in comment 12, just would like to be a bit more explicit, and check if one more thing is possible.  So, for 2.1:

a) with sync pref'd off, there is no performance regression on any platform/configuration (including those with hardware composer).  There are also no try regressions (or it couldn't land.)
b) with sync pref'd on, and SW Vsync option, try is green, and there are no performance regressions on platforms without hardware composer.

Now, we would need to find room for the hardware Vsync (is it landed, but with some performance regressions, or not landed at all?), as you mention actual frame uniformity (which we will define the goals hopefully as something that can be assigned a number or two), and whatever else is needed.

Comment 18

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Attachment: silk design wiki

This the silk project design wiki.
Please review this design wiki.
Then we can go to review the vsync dispatcher framework in bug 1048667.

Comment 19

[Sandip Kamat]

(In reply to Milan Sreckovic [:milan] (PTO 8/11 - 8/15) from comment #17)
> I like the 2.1 definition in comment 12, just would like to be a bit more
> explicit, and check if one more thing is possible.  So, for 2.1:
> 
> a) with sync pref'd off, there is no performance regression on any
> platform/configuration (including those with hardware composer).  There are
> also no try regressions (or it couldn't land.)
> b) with sync pref'd on, and SW Vsync option, try is green, and there are no
> performance regressions on platforms without hardware composer.
> 
> Now, we would need to find room for the hardware Vsync (is it landed, but
> with some performance regressions, or not landed at all?), as you mention
> actual frame uniformity (which we will define the goals hopefully as
> something that can be assigned a number or two), and whatever else is needed.

This makes sense.  Peter, Can you help list the phase 1 items and benchmarks for this? Let's open a separate bug for this and phase 2. possibly we need to group the bugs listed on wiki for 2 meta bugs.

Comment 20

[Peter Chang[:pchang]]

(In reply to Sandip Kamat from comment #19)
> (In reply to Milan Sreckovic [:milan] (PTO 8/11 - 8/15) from comment #17)
> > I like the 2.1 definition in comment 12, just would like to be a bit more
> > explicit, and check if one more thing is possible.  So, for 2.1:
> > 
> > a) with sync pref'd off, there is no performance regression on any
> > platform/configuration (including those with hardware composer).  There are
> > also no try regressions (or it couldn't land.)
> > b) with sync pref'd on, and SW Vsync option, try is green, and there are no
> > performance regressions on platforms without hardware composer.
> > 
> > Now, we would need to find room for the hardware Vsync (is it landed, but
> > with some performance regressions, or not landed at all?), as you mention
> > actual frame uniformity (which we will define the goals hopefully as
> > something that can be assigned a number or two), and whatever else is needed.
> 
> This makes sense.  Peter, Can you help list the phase 1 items and benchmarks
> for this? Let's open a separate bug for this and phase 2. possibly we need
> to group the bugs listed on wiki for 2 meta bugs.

Sandip, please check the Project Silk wiki page and you will see bugs with target milestone. I think it is enough for us to track Project Silk in 2.1.

About the benchmarks, Project Silk tries to align Vsync for every rendering operation to provide better user experience, like smoother scrolling, but not tries to boost the scrolling performance. Project Silk is on for 2.2, therefore, I would not expect to see the benchmark number for 2.1 although we already have better frame uniformity now and we will continue improving it after 2.1.

Besides frame uniformity, we also try to create a app to draw a small ball which follows with your finger. We can measure the distance between fingers(touch) and the position of the small ball for 2.2. The detail of benchmark will be listed at the benchmark section of Project Silk wiki page.

Comment 21

[Mason Chang [Inactive] [:mchang]]

 > Besides frame uniformity, we also try to create a app to draw a small ball
> which follows with your finger. We can measure the distance between
> fingers(touch) and the position of the small ball for 2.2. The detail of
> benchmark will be listed at the benchmark section of Project Silk wiki page.

Hi Peter, sorry I'm confused,  how would this work? From touch interpolation, we'll always be behind the actual touch event, so we should be smoother while following the touch but increase in distance. Could we also have an animation benchmark? If vsync and composites are within 16 ms, we should see smoother OMTA as well right? Thanks!

Comment 22

[No-Jun Park [:njpark]]

Testing Perspective:

minused in-moztrap flag because there is no manual test case to verify animation is 'smoother.'  This feature will be verified with following methods:
For 2.1:
- Manual graphics smoke test suite, checking for possible regression when the feature is preffed off
For 2.2:
- Manual graphics smoke test suite, checking for possible regression when the feature is preffed on/off
- Frame uniformity tool (bug 990832)
- Check for consistent composite interval of approximately 16.6 ms ( https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler)

Comment 23

[Robert O'Callahan (:roc) (email my personal email if necessary)]

The current design has VsyncThread making a sync call to the MainThread to run the InputDispatcher task. It's not clear to me why InputDispatcher has to run on the main thread, and it seems to me things would be a lot better if it didn't have to. So please document somewhere why InputDispatcher has to run on the MainThread.

Assuming that has to stay the same, what's the point of having the VsyncThread? If I understand the diagrams correctly, HWcomposer sends an async message to the VsyncThread to start the vsync process. If it can send an async message to the MainThread instead, we could go directly into the InputDispatcher and avoid a context switch.

Also, it looks like we don't need to return to the VsyncThread after the InputDispatcher has finished. The MainThread could itself send async messages to the CompositorThread and the NotifyVsyncEvent listeners when InputDispatcher finishes. This would eliminate another context switch.

Comment 24

[Matt Woodrow (:mattwoodrow)]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #23)
> The current design has VsyncThread making a sync call to the MainThread to
> run the InputDispatcher task. It's not clear to me why InputDispatcher has
> to run on the main thread, and it seems to me things would be a lot better
> if it didn't have to. So please document somewhere why InputDispatcher has
> to run on the MainThread.

iirc, the APZ event processing code all currently runs on the main thread. We need that to run to completion, and forward any required input events to the child processes before we send the vsync notification that will trigger the refresh driver.

I assume the VsyncThread exists for the future where this processing will happen on that thread, without MainThread involvement. Then we'd only have async notifications coming from the VsyncThread.

Assuming that's all correct, it would be nice if it were more clear that the synchronous call to the MainThread is a temporary thing, explaining why it exists and how it will be fixed.

Comment 25

[Mason Chang [Inactive] [:mchang]]

(In reply to Matt Woodrow (:mattwoodrow) from comment #24)
> (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #23)
> > The current design has VsyncThread making a sync call to the MainThread to
> > run the InputDispatcher task. It's not clear to me why InputDispatcher has
> > to run on the main thread, and it seems to me things would be a lot better
> > if it didn't have to. So please document somewhere why InputDispatcher has
> > to run on the MainThread.
> 
> iirc, the APZ event processing code all currently runs on the main thread.
> We need that to run to completion, and forward any required input events to
> the child processes before we send the vsync notification that will trigger
> the refresh driver.
> 
> I assume the VsyncThread exists for the future where this processing will
> happen on that thread, without MainThread involvement. Then we'd only have
> async notifications coming from the VsyncThread.
> 
> Assuming that's all correct, it would be nice if it were more clear that the
> synchronous call to the MainThread is a temporary thing, explaining why it
> exists and how it will be fixed.

Correct, we currently have a plan to get APZ off the main thread, this is in bug 930939. We currently want to process all touch events on the VsyncThread as we did not want to make a new input thread, which android already has.

The ideal architecture would be that we first sync notify APZ, then notify the Compositor. Thus, when we composite, we will have the most up to date touch coordinates rather than have a racy condition where we might process the previous touch event versus the latest touch events when we composite. This should reduce touch latency for the user. Please let me know if you have any questions. Thanks!

Comment 26

[Jerry Shih[:jerry] (UTC+8) (inactive)]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #23)
> The current design has VsyncThread making a sync call to the MainThread to
> run the InputDispatcher task. It's not clear to me why InputDispatcher has
> to run on the main thread, and it seems to me things would be a lot better
> if it didn't have to. So please document somewhere why InputDispatcher has
> to run on the MainThread.
bug 930939 will do the dispatch off main thread.
> 
> Assuming that has to stay the same, what's the point of having the
> VsyncThread? If I understand the diagrams correctly, HWcomposer sends an
> async message to the VsyncThread to start the vsync process. If it can send
> an async message to the MainThread instead, we could go directly into the
> InputDispatcher and avoid a context switch.

The latency between using HWC or VsyncThread to dispatch the input is quite small.
If we dispatch in VsyncDispatcher, the code is easy to read.

> 
> Also, it looks like we don't need to return to the VsyncThread after the
> InputDispatcher has finished. The MainThread could itself send async
> messages to the CompositorThread and the NotifyVsyncEvent listeners when
> InputDispatcher finishes. This would eliminate another context switch.

If chrome main thread is very busy, the main thread can't handle each vsync event.
The compositor or content can't receive vsync event anymore.
In current implementation, I use a monitor with 2ms timeout to wait the main thread input processing.
If the monitor timeout, we continue to notify compositor and content process.

Comment 27

[Robert O'Callahan (:roc) (email my personal email if necessary)]

If bug 930939 will eliminate the main thread from the event processing path and process those events on the VsyncThread instead, that sounds great. But as Matt said, please adjust your documentation to explain what the final state is going to be, with a diagram. Thanks!

Comment 28

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Thanks, roc and matt.

Please check:
https://wiki.mozilla.org/Project_Silk#Order_to_dispatch_Vsync_events

We have the diagram for current design and the ideal model with bug 930939.

Comment 29

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Looks good thanks!

Comment 30

[Kai-Chih Hu [:khu]]

Use feature-b2g:2.2? rather than just 2.2.

Comment 31

[Mason Chang [Inactive] [:mchang]]

Just a note, we require Kit-Kat for silk. This has to be a technical requirement. On JellyBean on both Nexus 4 and Flame, vsync intervals coming from the drivers were inconsistent. e.g., sometimes a vsync would come in at 32 ms or 4 ms instead of standard 16.6ms intervals, which would, by design, create janky and worse behavior than we currently have. So far from our testing, Kit-Kat, v184 Flame has fixed this.

Comment 32

[jerryj]

Referred here by avihpit in #894128 regarding a vsync issue.  Please validate and verify any Silk vsync implementation against this testing tool:

   http://www.duckware.com/test/chrome/jerky3.html

Comment 33

[Kai-Chih Hu [:khu]]

Bobby, can you follow up this topic? Thanks.

Comment 34

[Bobby Chien]

manage in feature-b2g for future release

Comment 36

[Mason Chang [Inactive] [:mchang]]

Silk on b2g is on by default in master and now uplifted to 2.2.