1222972 - Compare e10s vs non-e10s hang stacks from Telemetry pings

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P3)

Description

[poiru]

For the initial version, we should find the top 10 e10s and non-e10s hang stacks.

Comment 1

[poiru]

Results: http://nbviewer.ipython.org/github/vitillo/e10s_analyses/blob/master/aurora/e10s_top_hang_stacks.ipynb

Some findings from Roberto:
- IPDL::PCookieService::RecvGetCookieString is present in 7% of the e10s stacks (and 0% in non-e10s). The seems like something we could and should fix.
- Startup::XRE_Main is the topmost frame for 33% of e10s stacks (and 18% for non-e10s). Perhaps we need to add more pseudostack markers?
- (content script) is the topmost frame in 13% of non-e10s and 0.1% of e10s stacks. That seems like a big win for e10s, but is this an expected result?

Any thoughts?

Comment 2

[Jim Mathies [:jimm]]

This doesn't compare well with what I see here - 

https://telemetry.mozilla.org/chromehangs/

This data is coming from nightly and doesn't account for e10s processes. It doesn't match to your non-e10s data though, which has me wondering if this analysis is missing something.

For example the top stack for non-e10s with a ping count of 2570 is listed below. I would expect that to show up in our aurora data as well. 

    PR_WaitCondVar (in nss3.pdb)
    mozilla::CondVar::Wait(unsigned int) (in xul.pdb)
    mozilla::ipc::GeckoChildProcessHost::WaitUntilConnected(int) (in xul.pdb)
    mozilla::plugins::PluginProcessParent::WaitUntilConnected(int) (in xul.pdb)
    mozilla::plugins::PluginModuleChromeParent::LoadModule(char const *,unsigned int,nsPluginTag *) (in xul.pdb)
    GetNewPluginLibrary(nsPluginTag *) (in xul.pdb)
    nsPluginHost::EnsurePluginLoaded(nsPluginTag *) (in xul.pdb)
    nsPluginHost::GetPluginForContentProcess(unsigned int,nsNPAPIPlugin * *) (in xul.pdb)
    mozilla::plugins::SetupBridge(unsigned int,mozilla::dom::ContentParent *,bool,nsresult *,unsigned int *) (in xul.pdb)
    mozilla::dom::ContentParent::RecvLoadPlugin(unsigned int const &,nsresult *,unsigned int *) (in xul.pdb)
    mozilla::dom::PContentParent::OnMessageReceived(IPC::Message const &,IPC::Message * &) (in xul.pdb)
    mozilla::ipc::MessageChannel::DispatchSyncMessage(IPC::Message const &,IPC::Message * &) (in xul.pdb)
    mozilla::ipc::MessageChannel::DispatchMessageW(IPC::Message const &) (in xul.pdb)
    mozilla::ipc::MessageChannel::OnMaybeDequeueOne() (in xul.pdb)
    MessageLoop::DoWork() (in xul.pdb)

Comment 3

[Jim Mathies [:jimm]]

If this is what we have, this data isn't actionable, for example - 

- 43.2056% (43.2310%):
  Timer::Fire
  Startup::XRE_Main

- 33.5110% (18.4398%):
  Startup::XRE_Main

- 7.3378% (0.0041%):
  IPDL::PCookieService::RecvGetCookieString
  Startup::XRE_Main

You might chat with vladan or bsmedberg, I think he wrote the original telemetry stack collection code.

Comment 4

[poiru]

(In reply to Jim Mathies [:jimm] from comment #2)
> This doesn't compare well with what I see here - 
> 
> https://telemetry.mozilla.org/chromehangs/
> 
> This data is coming from nightly and doesn't account for e10s processes. It
> doesn't match to your non-e10s data though, which has me wondering if this
> analysis is missing something.

This analysis is based on pseudostacks. It seems like the chromehangs analysis uses actual stacks.

Comment 5

[Vladan Djeric (:vladan)]

BHR uses pseudostack frames, chrome hangs use real C++ stacks. We can do a chromehangs e10s comparison as well, but I think the bigger issue here is that BHR lacks coverage for non-gfx code. I filed bug 1224374 to expand its coverage (based on top chromehang stack frames)

Comment 6

[Vladan Djeric (:vladan)]

(In reply to Birunthan Mohanathas [:poiru] from comment #1)
> - (content script) is the topmost frame in 13% of non-e10s and 0.1% of e10s
> stacks. That seems like a big win for e10s, but is this an expected result?

There's an issue with missing child-process BHR stacks: bug 1228437

Comment 7

[Chris H-C :chutten]

This was done as part of the e10s analyses done by dzeber and bmiroglio.