571691 - farmville-related spike in plugin hang reports on 2010-06-10, around 18:30 (PDT? or maybe UTC?)

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Plug-ins, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Among the Firefox 3.6.4 (OOPP beta) users, there was a major spike in plugin hang reports around 19:00 (I think the times are PDT, although they might be UTC) on June 10 (Thursday).

This looks like something that is a real change among our user population, since there were significant changes among the domain composition for plugin hangs before and after this point.  If I compare the top URLs in the domains for the URLs in the plugin crash reports for: (a) the 2-day period before the spike and (b) the 28 hour period after the speak (omitting the data between 18:00 and 19:00 on Thursday), I see the top 10 domains as:

> hang_browser_top_bydom_before

      apps.facebook.com        www.orkut.com.br              my.mail.ru 
                   4486                    2472                     830 
       www.facebook.com         www.youtube.com fb-0.treasure.zynga.com 
                    709                     601                     287 
            about:blank        www.habbo.com.br       www.farmville.com 
                    277                     242                     210 
           vkontakte.ru 
                    154 

> hang_browser_top_bydom_after

      apps.facebook.com        www.orkut.com.br       www.farmville.com 
                   8604                    2262                     681 
       www.facebook.com         www.youtube.com              my.mail.ru 
                    625                     537                     488 
       www.habbo.com.br fb-0.treasure.zynga.com  globoesporte.globo.com 
                    267                     260                     213 
            about:blank 
                    210 

If I look at the URLs that spiked within apps.facebook.com (without query strings), I see that http://apps.facebook.com/onthefarm/index.php and http://apps.facebook.com/onthefarm/ had particularly strong spikes.

In particular, if I look at number of plugin hang reports per day from Firefox 3.6.4 users with onthefarm in the URL, I see:
20100607  586
20100608  709
20100609  712
20100610 2465
20100611 4380
For hangs with farmville.com in the URL, I see instead
20100607  258
20100608  354
20100609  387
20100610  581
20100611  852

The actual hourly hang numbers around the spike like this:

processed_rounded     Plugin hang: browser side Plugin hang: plugin side
  2010-06-10 13:00:00                       481                      488
  2010-06-10 14:00:00                       435                      438
  2010-06-10 15:00:00                       513                      512
  2010-06-10 16:00:00                       541                      544
  2010-06-10 17:00:00                       492                      499
  2010-06-10 18:00:00                       670                      668
  2010-06-10 19:00:00                      1004                     1013
  2010-06-10 20:00:00                       834                      831
  2010-06-10 21:00:00                       724                      724
  2010-06-10 22:00:00                       655                      660
  2010-06-10 23:00:00                       683                      688
  2010-06-11 00:00:00                       607                      613

note that this is a time of day when we're normally decreasing; compare the previous day:

processed_rounded     Plugin hang: browser side Plugin hang: plugin side
  2010-06-09 13:00:00                       452                      459
  2010-06-09 14:00:00                       446                      450
  2010-06-09 15:00:00                       472                      480
  2010-06-09 16:00:00                       459                      460
  2010-06-09 17:00:00                       487                      491
  2010-06-09 18:00:00                       453                      457
  2010-06-09 19:00:00                       446                      450
  2010-06-09 20:00:00                       353                      350
  2010-06-09 21:00:00                       417                      421
  2010-06-09 22:00:00                       338                      340
  2010-06-09 23:00:00                       308                      312
  2010-06-10 00:00:00                       346                      346

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: graph showing hang spike

This bug is about the spike in the purple+blue lines on June 10.  The spike in the red line on June 7 is bug 571118.

Comment 2

[chris hofmann]

did we ever get any contacts for the farmville development team?

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The browser-side signature that increased most was:
hang | mozilla::plugins::PPluginInstanceParent::CallPBrowserStreamConstructor(mozilla::plugins::PBrowserStreamParent*, nsCString const&, unsigned int const&, unsigned int const&, mozilla::plugins::PStreamNotifyParent*, nsCString const&, nsCString const...
which appeared from nowhere (precisely 0 hangs (searching all hang reports) on the June 9th, then 1720 on the 10th, 4392 on the 11th).  It's now the #2 hang signature on URLs with "onthefarm".

But there were also significant increases in the signatures:
hang | mozilla::plugins::PPluginInstanceParent::CallNPP_HandleEvent(mozilla::plugins::NPRemoteEvent const&, short*)
hang | mozilla::plugins::PPluginInstanceParent::CallNPP_SetWindow(mozilla::plugins::NPRemoteWindow const&)
among urls with "onthefarm".

Comment 4

[Carsten Book [:Tomcat]]

(In reply to comment #2)
> did we ever get any contacts for the farmville development team?

i'm working on this!

Comment 5

[Ryan VanderMeulen [:RyanVM]]

What are the odds that this is related to the release of Flash 10.1?

Comment 6

[chris hofmann]

adoption of flash versions is generally pretty slow.  We are hoping that 10.1 goes faster, but its looks like it's a low pct. of these hangs.


flash version breakdown for that first signature on 06/11 is

$ awk -F\t '$1 ~ /hang...mozilla::plugins::PPluginInstanceParent::CallPBrowserStreamConstructor.mozilla::plugins::PBrowserStreamParent./ {print $22}' 20100611* | sort | uniq -c 
   4 10.0.1.218
  60 10.0.12.36
   3 10.0.2.54
 175 10.0.22.87
 457 10.0.32.18
 391 10.0.42.34
2377 10.0.45.2
   4 10.1.50.426
  35 10.1.51.45
  22 10.1.51.66
   8 10.1.51.95
   5 10.1.52.14
   9 10.1.53.21
  31 10.1.53.38
  15 10.1.53.55
  58 10.1.53.60
 362 10.1.53.64
   9 10.1.53.7
   2 7.0.14.0
   7 9.0.115.0
  23 9.0.124.0
   3 9.0.28.0
   7 9.0.45.0
  15 9.0.47.0
 310 [blank]

$ awk -F\t '$1 ~ /hang...mozilla::plugins::PPluginInstanceParent::CallNPP_SetWindow.mozilla::plugins::NPRemoteWindow.const../ {print $22}' 20100611* | sort | uniq -c 
   1 10.0.0.525
   3 10.0.1.218
  56 10.0.12.36
   3 10.0.2.54
 145 10.0.22.87
 336 10.0.32.18
 350 10.0.42.34
1758 10.0.45.2
   3 10.1.50.426
  36 10.1.51.45
  20 10.1.51.66
   4 10.1.51.95
   2 10.1.52.14
   6 10.1.53.21
  25 10.1.53.38
   9 10.1.53.55
  33 10.1.53.60
 251 10.1.53.64
   4 10.1.53.7
   1 8.0.22.0
  10 9.0.115.0
  15 9.0.124.0
   1 9.0.16.0
   7 9.0.45.0
   5 9.0.47.0

Comment 7

[Mike Beltzner [:beltzner, not reading bugmail]]

Kev: do we know anyone at Zynga?

Comment 8

[[not reading bugmail]]

Well, farmville and facebook flash crashes are popping up all over sumo and elsewhere with 3.6.4 with OOPP on, though not many users are reporting which flash versions they are using.  They just are seeing it crash.

Comment 9

[Benjamin Smedberg]

Resolving old bugs which are likely not relevant any more, since NPAPI plugins are deprecated.