Closed Bug 1608028 Opened 4 years ago Closed 4 years ago

FarmVille Flash game content is blank with gfx.direct3d11.use-double-buffering

Categories

(Core Graveyard :: Plug-ins, defect, P1)

72 Branch
Unspecified
Windows
defect

Tracking

(firefox-esr68 unaffected, firefox72+ wontfix, firefox73+ disabled, firefox74+ disabled, firefox75 disabled)

RESOLVED WORKSFORME
Tracking Status
firefox-esr68 --- unaffected
firefox72 + wontfix
firefox73 + disabled
firefox74 + disabled
firefox75 --- disabled

People

(Reporter: pankaj17n, Assigned: handyman)

References

(Regression)

Details

(Keywords: regression)

Attachments

(4 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

  1. Load https://apps.facebook.com/onthefarm on Firefox 72 (Windows)
  2. Click Allow Flash

Actual results:

Flash content did not load

Expected results:

Flash content should have loaded

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Plug-ins
Product: Firefox → Core

Same problem here.
Windows 7 32 bits, firefox 72.0.1

about:plugins doesn't show the flash plugin. VLC plugin is missing as well.

Firefox does not even ask to activate, nothing.

Actually if I reinstall flash player, it works right after. But as soon as I close firefox and reopen it, it's broken again.

Downgraded to firefox 71, the issue is gone.

With Flash 32.0.0.303 on Windows 10, after I click to run Flash for FarmVille, I just see a blank white area where the game should be. If I right click on the blank white area, I see Flash's context menu and then Firefox hangs. I can't scroll the web page or switch to other Firefox tabs. Other Flash websites (like zombo.com) load correctly before I try to load FarmVille.

@ Miko, I tried bisecting this Flash bug with mozregression and landed on this pushlog with your fix for nsDisplayWrapList bug 1529698 (in Firefox 68). Do you think your fix might cause this Flash bug?

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=e013f1f17109a8c22cbc7abf6f78db55bd2a8efb&tochange=e37407909f92b48af964ac41514a1b58c2ba1ff4

@ Jim, someone on the plugin team should probably investigate this Flash bug. FarmVille is a high profile Flash game. Does QA test FarmVille in their Firefox smoke tests?

Flags: needinfo?(mikokm)
Flags: needinfo?(jmathies)
Keywords: regression
OS: Unspecified → Windows
Priority: -- → P1
Regressed by: 1529698

I am curious why I see a blank white area starting in Firefox 68, but people commenting in the bug report say the problem started in Firefox 72 and the game works correctly for them in Firefox 71.

I was testing with mozregression-gui. Perhaps the way mozregression-gui launches Python which launch Firefox which launches the NPAPI plugin process which launches Adobe's Flash Protected Mode process is causing some inter-process hangs that are are not reproducible when launching Firefox on its own?

Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: flash content is not visible on firefox 72 windows → FarmVille Flash game content is blank on Firefox 72 (or 68?) Windows

Actually I just realized my issue is maybe different from the one reported by the original poster. Sorry for the confusion.

My problem is not only with farmville or similar games but with all pages using flash that I have tested.
Even https://get.adobe.com/fr/flashplayer/about/ doesn't work work with FF 72.0.1

Firefox does not even ask me to activate flash player.
Actually it looks it does not detect the plugin is installed. When I type about:plugins, it only shows openh264 and Widewine, while shockwave flash should be there. My VLC Web Plugin is missing as well from the about:plugins page.

The first time I launch FF after installing the flash plugin, it works as expected. But when I close FF and relaunch it, it doesn't work anymore. Then I reinstall the plugin and it works again. And so on.

I downgraded to FF 71, with the same profile. The issue is gone.

See Also: → 1609257

Steph, since the bug you're seeing affects more than just FarmVille, I filed a new bug report to avoid confusing these two problems. We can continue our discussion in the new bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1609257

(In reply to Chris Peterson [:cpeterson] from comment #4)

I am curious why I see a blank white area starting in Firefox 68, but people commenting in the bug report say the problem started in Firefox 72 and the game works correctly for them in Firefox 71.

Perhaps there's some kind of pref that held the new behaviour implemented in 68 to nightly or early beta or something, and then we shipped it to release in 72?

Series issue (maybe?) with flash on release. Can we get some help reproducing this to see how common it is?

Flags: needinfo?(jmathies) → needinfo?(andrei.vaida)

Managed to confirm the issue with ease Windows 10 - Flash 32.0.0.314 - 73.0b5/74.0a1(20200114214307).

Flash versions verified - acquired from the Adobe archive:

  • 31.0.0.153 - Released 11/20/2018 - 74.0a1 (bad)
  • 27.0.0.130 - Released 09/12/2017 - 73.0b5 (bad)
  • 26.0.0.126 - Released 06/13/2017 - 73.0b5 (bad)
  • 23.0.0.162 - Released 09/13/2016 - 73.0b5 (bad - closing browser displays game for a brief second)
  • 18.0.0.375 - Released 09/13/2016 - 73.0b5 (works after install, refresh, browser restart)
  • 22.0.0.209 - Released 07/12/2016 - 73.0b5/72.0.1 (works after install, refresh, browser restart)
  • 22.0.0.192 - Released 06/16/2016 - 73.0b5 (works after install, refresh, browser restart)
  • 21.0.0.213 - Released 04/7/2016 - 73.0b5/74.0a1 (works after install, refresh, browser restart)

So, apparently releases on 09/13/2016 from Adobe, had some changes that might cause this.

Flags: needinfo?(andrei.vaida)

68.4.1esr is not affected by this issue (with Adobe Flash Player 31.0.0.153).

(In reply to Chris Peterson [:cpeterson] from comment #3)

@ Miko, I tried bisecting this Flash bug with mozregression and landed on this pushlog with your fix for nsDisplayWrapList bug 1529698 (in Firefox 68). Do you think your fix might cause this Flash bug?

The patch did change some boundary calculation logic so it is definitely possible, although a bit surprising that it would surface so long after landing. I managed to reproduce this bug on Windows (did not reproduce on macOS), and I will check if backing out the changes locally fixes this.

Flags: needinfo?(mikokm)

Hey David, could you see if you can reproduce on a device? If so maybe you can find a cause.

Flags: needinfo?(davidp99)

I can reproduce on win10, 64-bit with latest flash in Nightly.I think flash is running and might even be painting sometimes but usually I'm getting a blank white space in the page.

Not likely we'll be able to do anything for Fx72 at this point from a dot release standpoint, but it would be good if we could figure something out for Beta73 still.

Severity: normal → critical

(In reply to Miko Mynttinen [:miko] from comment #11)

(In reply to Chris Peterson [:cpeterson] from comment #3)

@ Miko, I tried bisecting this Flash bug with mozregression and landed on this pushlog with your fix for nsDisplayWrapList bug 1529698 (in Firefox 68). Do you think your fix might cause this Flash bug?

The patch did change some boundary calculation logic so it is definitely possible, although a bit surprising that it would surface so long after landing. I managed to reproduce this bug on Windows (did not reproduce on macOS), and I will check if backing out the changes locally fixes this.

System: Windows 10 (64bit), Flash 32.0.0.314.
Backing out these changesets locally does not help. This makes me believe that the regression range is incorrect.

Using the flash version above, I ran mozregression, which gave me a very different regression range: every build I tried after 2018-09-12 was not working. Pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2a59b432d2bd9b15ceec6b9435f60c785a820ef2&tochange=e923330d5bd3aef1f687eddf96a06ad5ec3860ed

It is possible that this regression is also incorrect, because this site seems rather flaky and broken (console is filled with error messages, elements disappear randomly).

No longer regressed by: 1529698

This sounds pretty broken, but the regression range also makes me wonder if there was something external to Firefox contributing to this? Either way, Fx73 goes to RC in just under 2 weeks, so we need to make progress on this soon if there's going to be any chance of fixing it for the next release.

Flags: needinfo?(jmathies)

I'm digging into this -- I had to get a Facebook account approved to debug. I'll post what I can figure out today.

Flags: needinfo?(jmathies)

I don't know the cause or have a patch yet but I have confirmed that Flash is requesting and we are dutifully creating and properly maintaining a surface for this... a 2x2 pixel surface as per Flash's bad request. Since older versions of Flash are said to work here, the incorrect sizing isn't due (at least primarily) to anything we are doing. So I can't say for sure that we will be able to get around this but I'm going to try.

I have some strong leads on this but no fix yet. I think it's probable that we'll have a fix by end of week so lets reassess the bug then.

The core issue appears to be us -- that Firefox appears to occasionally get confused about the "wmode" of the plugin because Facebook has chosen an odd setting wmode = "default" (this is unusual because its supposed to be the same thing as if you don't put it). Our sanitizing logic is poor and we end up, in some parts of code, treating the plugin as a "windowed" plugin, which we haven't supported (even unofficially) for years. However, changing the value to "opaque" (the new default) seems to avoid the windowed plugin code, I'm still getting bad dimensions and no display. I'd be surprised if what I'm seeing is too complex, which is why I'm holding out hope for a patch this week.

Unfortunately, next week will be terrible for all this so, if I can't get something working soon, we may have to ... ... figure out an emergency plan? I don't know what that would be.

Assignee: nobody → davidp99
Flags: needinfo?(davidp99)

(In reply to David Parks (dparks) [:handyman] from comment #18)

Since older versions of Flash are said to work here, the incorrect sizing isn't due (at least primarily) to anything we are doing.

btw, if you'd like to test older Flash versions, Adobe has an archive of old Flash installers all the way back to Flash 9 (2010):

https://helpx.adobe.com/flash-player/kb/archived-flash-player-versions.html

Thanks Chris. I've learned a few more things about the bug but I don't have a fix. First, I had the cause wrong, although the issues I mentioned above are real -- they aren't too difficult to fix but they aren't the main issue (I now have a big black rect instead of a big white one -- this is actually progress). What I am now seeing is some DOM gymnastics in the FB page that appear to break either our reflow calculations, or Flash. IOW, the page does some things with an invisible plugin, then does some JS manipulation to add the plugin to the DOM. (With my "fixes") I see 3 plugin instances -- an invisible short-lived one (that doesn't really get killed), then one that has a Xx600 pixel surface, and another invisible one. One of the invisible surfaces gets render commands from Flash (which may not be wrong, despite being invisible). But the large one that presumably is supposed to be the game instance is not updated. Since these updates come from Flash, it makes sense that this is Flash's issue, but I'm not 100% sure of that. The way multi-plugin-instance pages work is that they all share the same Flash instance, so maybe it is confusing them internally.

I'll keep looking at this from our end but I won't have much time for it. If Adobe can check the surface issue, that would help. It also may be possible for Facebook to do something less complex in adding the plugin to the DOM. I'll post a patch/build with my current changes (that aren't ready for primetime) today.

(In reply to David Parks (dparks) [:handyman] from comment #21)

I'll keep looking at this from our end but I won't have much time for it. If Adobe can check the surface issue, that would help. It also may be possible for Facebook to do something less complex in adding the plugin to the DOM. I'll post a patch/build with my current changes (that aren't ready for primetime) today.

I emailed our Adobe contacts and CC'd you.

From the Flash Player 23 release notes:

Mozilla NPAPI AsyncDrawing Support

Async Drawing refers to the method that the browser and Flash Player use to exchange a bitmap surface where Flash Player draws the SWF content. It is used only when the stage is composited with rest of the content in the browser window. This feature allows wmode “direct” (wmode opaque and transparent) to behave as “windowless” in hardware accelerated async drawing. It is not used in fullscreen mode, or in windowed mode where the plugin draws directly to its own window. If asynchronous drawing is unavailable for any reason, the plugin falls back to using the existing synchronous drawing model.

AsyncDrawing is supported in NPAPI Plugin on Windows desktop platforms only. It is currently available from FP version 23.0 in Firefox Nightly 51.0a1, the Firefox versions supporting the feature is yet to be announced. The choice of which Async Drawing path is used (hardware or software) depends on whether the browser supports hardware or software Async Drawing modes.

To disable AsynchronousDrawing support in Firefox, go to “about:config” in the search bar of the browser and set “dom.ipc.plugins.asyncdrawing.enabled” to false.

I've opened FLASH-4191674 for further investigation.

Given that previous attempts to bisect this are yielding confusing results, having someone walk through it with both Firefox and Flash symbols is the most direct path to an answer.

I believe that the regression range is limited to the enablement of Async Drawing, which you can confirm by toggling the flag above. My sense is that we haven't really touched anything likely to break async drawing in several months, and that the customer issue being reported here has started much more recently than 2016, when Flash Player 23 shipped.

Need to disable AsynchronousDrawing support in Firefox, go to “about:config” in the search bar of the browser and set “dom.ipc.plugins.asyncdrawing.enabled” to false.
After that flash runs successfully.

This is still in the queue for someone to debug, but we've been triaging it and are pretty sure it's related to async drawing changes in Firefox. We did confirm that disabling async drawing in Firefox "resolves" the issue.

I'm also wondering if the specific version of Win10 under test bears on the test results. I'm definitely able to reproduce this under the latest moz-regression-gui on Windows, but I'm on Win10 10.0.18363 with Flash Player 32.0.0.321.

I was able to narrow this down to a reasonably small range, but I couldn't pin it down to a specific changeset. I did observe that there are some intermediate variations of "brokenness" to how the Flash content renders, where in some instances the SWF doesn't render at all, and in some instances, there's a HTML layer with troubleshooting instructions that overlays the top half of the SWF. I called those "good" in the regression because they were drawing something, but I'm wondering if there are multiple intersecting issues in this Firefox build range that led to my inconclusive bisection attempt.

I've included the full log output from mozregression and an overview screenshot to help you reproduce my results.

Attached image mozregression.PNG

Checking on my side, Windows version installed is 1903(OS Build 18362.592).
Indeed, setting dom.ipc.plugins.asyncdrawing.enabled to false did appear to allow the game to load and become somewhat playable.

Per attachment 9123224 [details] this was regressed by bug 1549674.

Has Regression Range: --- → yes
Regressed by: 1549674

Thanks all. Every bit helps.

There are two options of surface used for direct Async drawing -- an async bitmap surface and an async dxgi surface. There is new code from bug 1577336 that affects async dxgi drawing (only). Disabling DXGI surfaces (dom.ipc.plugins.allow_dxgi_surface) forces async bitmap surfaces. But this does not change anything.

The only other recent work that I know of in this corner of the code base is :Gijs code optimizing the plugin dll search. That seems even less relevant.

I've been studying logs and traces to try to narrow down the issue. Fundamentally, the bug starts early in the run (long before any of the ideas discussed here). One of the earliest indications that it is misbehaving is when Firefox windowed plugin code starts e.g. generating temporary DIB surfaces to give the plugin something to show before the first render -- a behavior that is no longer valid. From there is sets up a (windowed) WindowProc, which is very wrong, and everything quickly breaks down. My earlier efforts to avoid windowed behavior altogether seemed valid (traces showed windowed behavior as removed in the Firefox code before loading) but also did not fix the bug. At this point, I am comparing the behavior to what is happening in working plugins to see where there is a substantive difference. It's slow going but I will eventually isolate the discrepancy. So far, I think I'm seeing differing behavior in Flash's newp around the windowless property but this could easily be due to something we passed in earlier that I haven't found yet. I'm also not convinced that this isn't a confluence of issues since, again, the windowed behavior is all wrong but avoiding it didn't help.

To clarify my last post (which I didn't realize went through since I got a bugzilla error):

I agree with :gijs in comment 30, although "regressed" is a bit strong since that was 9 months and many releases ago. This is probably a combination of buggy new layout behavior in the html/js and bug 1549674, with the possibility that Flash changes are also playing a part. It is true that the behavior started by newp should not have been permitted by our code but I think the fix I had for that earlier was good and just lead to a clearer view of this issue. Those changes that make it initialize async drawing properly so I can start to see why async surfaces (both HW and SW) are relevant to bug 1549674. So I think a fix from us or Facebook (Zynga?) is most likely -- a fix in Flash is less likely.

Confirmed that Farmville runs in windowed plugin mode (sigh) when gfx.direct3d11.use-double-buffering is false. It behaves mysteriously in non-windowed mode, probably because it's not been tested/debugged in that mode. Safe transition to non-windowed is supposed to be essentially automatic but it's not because of the "wmode=default" stuff explained above. Facebook and Zynga will need to switch this to a supported mode. Our windowed code is a mine field that hasn't been debugged in years but I'm looking for a contained fix for now.

I've closed this bug on my end. If this need additional attention from us, just shoot me an email or flag me in a needsinfo comment. Thanks!

(In reply to David Parks (dparks) [:handyman] from comment #33)

Confirmed that Farmville runs in windowed plugin mode (sigh) when gfx.direct3d11.use-double-buffering is false. It behaves mysteriously in non-windowed mode, probably because it's not been tested/debugged in that mode. Safe transition to non-windowed is supposed to be essentially automatic but it's not because of the "wmode=default" stuff explained above. Facebook and Zynga will need to switch this to a supported mode. Our windowed code is a mine field that hasn't been debugged in years but I'm looking for a contained fix for now.

David, what specifically should I ask Zynga (or Facebook) to change? Zynga reported this bug to us, so they will probably be receptive to making code changes to fix FarmVille.

(In reply to Jeromie Clark from comment #34)

I've closed this bug on my end. If this need additional attention from us, just shoot me an email or flag me in a needsinfo comment. Thanks!

Jeromie, thanks for your help tracking down this bug!

Flags: needinfo?(davidp99)

Thanks Jeromie!

I'm not at a point where I can answer that yet. I'm trying things from a bunch of angles but its still to premature to rope in others. I'll get back to you with an update soon.

Flags: needinfo?(davidp99)

Sorry, that's probably not what you meant.

WRT the bigger issue, you can tell them that wmode="default" (which is an alias for "window" [1]) is invalid and they need to migrate to a supported mode. Acceptable values are:
"direct" (most common)
"window" or "gpu" (which are internally just switched to "direct" so there is no reason to use them)
"transparent"
"opqaue"
Notes on how we should have treated wmode are here [2].

This is actually important for them to do soon because the win32k-lockdown sandbox project will completely break the unsupported windowed mode. It will forbid using win32k functions in the content process, which windowed mode does a lot of. The other modes have already been refactored to support win32k-lockdown. It is the primary goal of our sandboxing work and won't be held up by a mode we don't actually support. If they need info on the timeline for this, I can get more info.


[1] Firefox should have treated "default" the way it treats "window" -- i.e. internally just replaced it with "direct" -- but we aren't doing that correctly, so "default" is a backdoor to the retired "window" mode.
[2] https://wiki.mozilla.org/Plugins/Async_Drawing

I haven't read through this thoroughly yet but to confirm DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL (gfx.direct3d11.use-double-buffering=true) first shipped in 72. It was reverted in 72.0.2 so the regression should be gone.

(In reply to David Parks (dparks) [:handyman] from comment #37)

WRT the bigger issue, you can tell them that wmode="default" (which is an alias for "window" [1]) is invalid and they need to migrate to a supported mode.

I emailed Zynga. They said they will make the changes in their next FarmVille release.

(In reply to Chris Peterson [:cpeterson] from comment #39)

I emailed Zynga. They said they will make the changes in their next FarmVille release.

If Zynga makes the wmode change, is there any Firefox work still needed in this bug? Or can we close this bug?

Flags: needinfo?(davidp99)

Once we confirm that comment 38 works as a fix then yeah, we can close this. But I guess we should try to find a way to make sure it gets tested when/if DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL gets restored.

Flags: needinfo?(davidp99)

Chris,

gfx.direct3d11.use-double-buffering=true is blocked outside of nightly by bug 1610912 / bug 1608579. My version confusion was caused by the pref change uplift. Its being blocked for video stuttering -- that likely won't be related to the issue in this bug so we still need a special fix for windowed plugins to avoid this returning when the setting is restored. Also, since double-buffering is still on in nightly, this bug is still there.

Windowed plugins use their own weird setup to maintain window parentage. I'm looking for a way to make them work together. So we should not close this bug. We will still need Zynga to address this for the reasons mentioned in comment 37.

So I'm clear, Fx73 shouldn't be affected at this point due to double buffering being disabled, right?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #43)

So I'm clear, Fx73 shouldn't be affected at this point due to double buffering being disabled, right?

Cristian can you double check this on 73?

Flags: needinfo?(cristian.fogel)

That is my understanding -- yes. The bug should only be seen in nightly. But double-checking can't hurt.

Indeed; game appears to work with 73.0, 73.0b11, 73.0b12 and Adobe Flash Player 32.0.0.321.
74.0a1 (2020-02-04) not working.

However, noticed another issue here; while scrolling the page up/down the game_area goes blank-white
(confirmed this on 2 workstations, AMD & Intel+Nvidia).

Flags: needinfo?(cristian.fogel)

The issue is verified as fixed using Windows 10 x64 with Firefox Release 72.0.2 and Beta 73.0 versions. Flags will be updated accordingly.

Hi,
I was able to reproduce the bug on Windows 10, on Firefox 72.0.2, Beta 73.0a1 (2019-12-12) (64-bit) and Nightly version 74.0a1.
I've updated flags accordingly.

Regards,
Jerónimo.

Attached image FarmVille.gif

I tried again on Windows 10 x64 with Firefox Release 72.0.2 and 73.0 and Beta 73.0 and 74.0b1 and I can't reproduce the issue.

Attached file FarmVille screencast

(In reply to David Parks (dparks) [:handyman] from comment #42)

Chris,

gfx.direct3d11.use-double-buffering=true is blocked outside of nightly by bug 1610912 / bug 1608579.

Is this still true also for 74? And is the priority setting of this bug still appropriate?

Flags: needinfo?(davidp99)

Just confirmed that the bug does not appear in 74.0b1.
The bug will continue to appear in all nightly builds as gfx has set gfx.direct3d11.use-double-buffering to true only on nightly (whatever it is at the time).

The bug is still P1 -- I am actively working on a fix for when double-buffering rides the trains.

The bug is caused by the FLIP_SEQUENTIAL swap chain not playing nice with sibling windows. A bunch of window parentage/z-order experiments to work around it all failed but I am seeing some success in experiments that use the dirty rects to block out the plugin as not-dirty. So far, this seems to keep the swap chain from destroying the sibling.

Flags: needinfo?(davidp99)

Pascal, should we then change the tracking flags? Beta and release for 74 and lower seem not to be affected.

Flags: needinfo?(pascalc)

Flags updated to reflect the fact that we have disabled double-buffering on beta/release.

David, do we see the same bug with WebRender on?

Flags: needinfo?(davidp99)

Yes, it looks the same with gfx.webrender.all=true

Flags: needinfo?(davidp99)

Jeff, do we have plans to ship 'gfx.direct3d11.use-double-buffering=true' at some point? Also, is the webrender regression tied to that pref as well, or do we have a webrender issue here that might be shipping separately?

Flags: needinfo?(jmuizelaar)

The future for gfx.direct3d11.use-double-buffering=true is unclear. The WebRender regression is separate from that pref and has perhaps been shipping since 67?

Flags: needinfo?(jmuizelaar)

I'm going to spin the webrender issue off into another bug.

https://bugzilla.mozilla.org/show_bug.cgi?id=1616294

Summary: FarmVille Flash game content is blank on Firefox 72 (or 68?) Windows → FarmVille Flash game content is blank with gfx.direct3d11.use-double-buffering
See Also: → 1616294
Flags: needinfo?(jmathies)

I've created the new bug 1620453 to address wmode cleanup. This works with Zynga's change so I'm closing WFM.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME
See Also: → 1620453
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: