1261620 - plugin-container.exe renamed; causes GPU driver issues

Status: RESOLVED INVALID

(Core :: Graphics: Layers, defect)

Description

[Kyle Repinski]

Attachment: GraphicalArtifacts.png

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160402030226

Steps to reproduce:

Updated to the latest (2016/04/02) Windows Nightly build, plugin-container.exe has now been renamed to firefox-webcontent.exe


Actual results:

Major memory leaks and graphical artifacts were caused by the AMD GPU drivers. After forcing firefox-webcontent.exe to use the Power Saving GPU instead of the default High Performance GPU, the problem was resolved. See attachments.


Expected results:

The GPU drivers should have kept firefox-webcontent.exe restricted to the internal GPU to avoid this problem, as it does for plugin-container.exe

While this IS an issue with the drivers, driver support for some of these systems is going to be dropped very soon. If AMD cannot be made aware of this, many people may be SOL if it's not fixable on Firefox's side.

This particular system uses a Radeon HG 6620G as the "Power Saving" GPU, and a Radeon HD 6750M as the "High Performance" GPU.

Comment 1

[Kyle Repinski]

Attachment: MemoryUsage.png

Comment 2

[Kyle Repinski]

Attachment: CCCFixAndInfo.png

Comment 3

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/message/reply/question/etc.)]

[Tracking Requested - why for this release]: Regression

Comment 4

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Kyle Repinski from comment #0)
> Created attachment 8737527 [details]
> GraphicalArtifacts.png
> 
> User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:48.0) Gecko/20100101
> Firefox/48.0
> Build ID: 20160402030226
> 
> Steps to reproduce:
> 
> Updated to the latest (2016/04/02) Windows Nightly build,
> plugin-container.exe has now been renamed to firefox-webcontent.exe
> 
> 
> Actual results:
> 
> Major memory leaks and graphical artifacts were caused by the AMD GPU
> drivers. After forcing firefox-webcontent.exe to use the Power Saving GPU
> instead of the default High Performance GPU, the problem was resolved. See
> attachments.
> 
> 
> Expected results:
> 
> The GPU drivers should have kept firefox-webcontent.exe restricted to the
> internal GPU to avoid this problem, as it does for plugin-container.exe
> 
> While this IS an issue with the drivers, driver support for some of these
> systems is going to be dropped very soon. If AMD cannot be made aware of
> this, many people may be SOL if it's not fixable on Firefox's side.
> 
> This particular system uses a Radeon HG 6620G as the "Power Saving" GPU, and
> a Radeon HD 6750M as the "High Performance" GPU.

In both the broken and non-broken modes, can you attach the graphics section of about:support?

Comment 5

[Kyle Repinski]

Attachment: AboutSupportWorking.png

Comment 6

[Kyle Repinski]

Attachment: AboutSupportNotWorking.png

Comment 7

[Kyle Repinski]

(In reply to David Anderson [:dvander] from comment #4)
> In both the broken and non-broken modes, can you attach the graphics section
> of about:support?

Yes, I have attached them. You'll notice that they're exactly the same.

I suspect the crux of the issue is that the GPU drivers are running firefox.exe and firefox-webcontent.exe on different GPUs, and I can't imagine that working nicely. Unfortunately I can't force firefox.exe to run on the High Performance GPU, as it locks it (and the old plugin-container.exe) for "system stability purposes". Now I understand why they did that.

Just my 2 cents here: if Firefox used a single process name (firefox.exe+firefox.exe instead of firefox.exe+firefox-webcontent.exe) like other browsers this would be a non-issue. While that might look confusing in Task Manager to some people, it seems to be more trouble than expected having these separate names and files. At the very least I think opinions need to be set aside for a minute and look at the technical advantages and disadvantages to both.

Comment 8

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Kyle Repinski from comment #7)
> (In reply to David Anderson [:dvander] from comment #4)
> > In both the broken and non-broken modes, can you attach the graphics section
> > of about:support?
> 
> Yes, I have attached them. You'll notice that they're exactly the same.
> 
> I suspect the crux of the issue is that the GPU drivers are running
> firefox.exe and firefox-webcontent.exe on different GPUs, and I can't
> imagine that working nicely. Unfortunately I can't force firefox.exe to run
> on the High Performance GPU, as it locks it (and the old
> plugin-container.exe) for "system stability purposes". Now I understand why
> they did that.

We have code to prevent this, but we've run into problems with Windows 7 before: Optimus drivers report the wrong adapter in the child process. We get around this by detecting whether the vendor mismatches, but if ATI has similar problems and both GPUs have the same vendor - that workaround won't work.

Whether or not the processes renaming sticks, we should probably look into this. Jeff, any ideas? Do we have any machines in the lab with this hardware?

Comment 9

[Kyle Repinski]

(In reply to David Anderson [:dvander] from comment #8)
> (In reply to Kyle Repinski from comment #7)
> > (In reply to David Anderson [:dvander] from comment #4)
> > > In both the broken and non-broken modes, can you attach the graphics section
> > > of about:support?
> > 
> > Yes, I have attached them. You'll notice that they're exactly the same.
> > 
> > I suspect the crux of the issue is that the GPU drivers are running
> > firefox.exe and firefox-webcontent.exe on different GPUs, and I can't
> > imagine that working nicely. Unfortunately I can't force firefox.exe to run
> > on the High Performance GPU, as it locks it (and the old
> > plugin-container.exe) for "system stability purposes". Now I understand why
> > they did that.
> 
> We have code to prevent this, but we've run into problems with Windows 7
> before: Optimus drivers report the wrong adapter in the child process. We
> get around this by detecting whether the vendor mismatches, but if ATI has
> similar problems and both GPUs have the same vendor - that workaround won't
> work.
I'm not familiar with the code in question, but would it be possible to extend it to also look at the device ID rather than just the vendor ID?

Comment 10

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Kyle Repinski from comment #9)
> I'm not familiar with the code in question, but would it be possible to
> extend it to also look at the device ID rather than just the vendor ID?

Yes - we do do that, actually [1], but drivers return buggy data on Windows 7. Optimus goes as far as to return the same unique ID for both adapters. So if the check is failing ATI might be doing something even worse :(

We could try including even more fields in the check. I'll send something to try tonight.

[1] hg.mozilla.org/releases/mozilla-release/file/tip/gfx/thebes/gfxWindowsPlatform.cpp#l1673

Comment 11

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to David Anderson [:dvander] from comment #8)
> 
> Whether or not the processes renaming sticks, we should probably look into
> this. Jeff, any ideas? Do we have any machines in the lab with this hardware?

Not exactly a 6620G, but the 6450 that Benoit has in "Bulldozer" system should be close enough.

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

Jeff, maybe we can find out from our local AMD contacts if they do "magic" (like Nvidia did) by checking the executable name and doing different things if they detect "firefox"?

Comment 13

[David Anderson [:dvander] - inactive, e-mail if emergency]

Kyle, can you try one of the nightly builds here?

http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-4c022d94cd1d66bf746816a8f87c4a97449750ca/try-win32/

Comment 14

[Jeff Muizelaar [:jrmuizel]]

I don't think we have any dual AMD GPU systems in the office. Kyle, what's the make and model of the machine you're using?

Comment 15

[Benoit Girard (:BenWa)]

The bulldozer has a single GPU. Let me know if you need me to run a test and/or get a second GPU for it (if it's supported).

Comment 16

[Jeff Muizelaar [:jrmuizel]]

I've purchased a laptop that has a AMD Radeon HD 7470M/Radeon 6520G combo. Hopefully, that will be similar enough to reproduce the problem.

Comment 17

[Kyle Repinski]

(In reply to David Anderson [:dvander] from comment #13)
> Kyle, can you try one of the nightly builds here?
> 
> http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-
> 4c022d94cd1d66bf746816a8f87c4a97449750ca/try-win32/

Same issue with this build.


(In reply to Jeff Muizelaar [:jrmuizel] from comment #16)
> I've purchased a laptop that has a AMD Radeon HD 7470M/Radeon 6520G combo.
> Hopefully, that will be similar enough to reproduce the problem.

That should be close enough, this is a (AMD+AMD) Switchable Graphics thing in general I think.


(In reply to Milan Sreckovic [:milan] from comment #12)
> Jeff, maybe we can find out from our local AMD contacts if they do "magic"
> (like Nvidia did) by checking the executable name and doing different things
> if they detect "firefox"?

I believe the 'magic' they do for Firefox is simply locking "firefox" and "plugin-container" to the same GPU in the Switchable Graphics settings. With the subprocess being renamed this magic is obviously no longer in place.

I have a bad feeling that the way Switchable Graphics profile settings work is something along the lines of "point ALL adapters to the profile's GPU setting". :(

Comment 18

[George Wright (:gw280) (needinfo me!)]

OK, we can reproduce the issue here. Paul, I understand you might be able to help us out here. Is there a way for us to add code to Firefox to state a preference for which GPU to run on? Or is there at least a way for us to detect which GPU we're running on? Our existing dual GPU detection code doesn't seem to see both GPUs here.

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

Let's back this change out while we're waiting for the updates.

Comment 20

[Kyle Repinski]

In regards to stating which GPU Firefox would like to run on, there is this:

extern "C"
{
  __declspec(dllexport) int AmdPowerXpressRequestHighPerformance = 1;
}

This would request that an application use the dGPU, albeit with the caveat that it's only supposed to work if a Switchable Graphics profile is not already assigned. firefox.exe is already locked to the iGPU... so does this flag by chance work in reverse (set to 0 for iGPU)? Or is there another flag similar to this that could be useful?

Comment 21

[Paul Blinzer]

For such Dual-GPU systems there is indeed some filename detection going on, causing the corruption if "the wrong" GPU is used (plugins are directed to integrated GPU). We will address this in the profile data.
As workaround the end user can also set up the firefox-webcontent.exe through the Catalyst Control Center (CCC) to integrated, which should address the problem. 
To do so, go to CCC, select "Power", "Switchable Graphics Application Settings", "Add Application" and assign integrated GPU to firefox-webcontent.exe

Comment 22

[Kyle Repinski]

(In reply to Paul Blinzer from comment #21)
> For such Dual-GPU systems there is indeed some filename detection going on,
> causing the corruption if "the wrong" GPU is used (plugins are directed to
> integrated GPU). We will address this in the profile data.
> As workaround the end user can also set up the firefox-webcontent.exe
> through the Catalyst Control Center (CCC) to integrated, which should
> address the problem. 
> To do so, go to CCC, select "Power", "Switchable Graphics Application
> Settings", "Add Application" and assign integrated GPU to
> firefox-webcontent.exe

Rather than be dependent on the profiles getting updated (new drivers for these GPUs aren't going to be around much longer if at all), or users being intelligent enough to figure this out, could you just add something similar to, but the opposite of, the AmdPowerXpressRequestHighPerformance symbol check? E.g. AmdPowerXpressRequestPowerSaving? Just something that can avoid these problems coming from something as simple as changing the name of an executable file. :/

Comment 23

[Gian-Carlo Pascutto [:gcp]]

(In reply to Kyle Repinski from comment #22)
> Rather than be dependent on the profiles getting updated (new drivers for
> these GPUs aren't going to be around much longer if at all), or users being
> intelligent enough to figure this out, could you just add something similar
> to, but the opposite of, the AmdPowerXpressRequestHighPerformance symbol
> check? E.g. AmdPowerXpressRequestPowerSaving? Just something that can avoid
> these problems coming from something as simple as changing the name of an
> executable file. :/

Even if AMD adds this, then you're still requiring people to update their drivers to get access to this new flag. Which means they'll get the new profiles anyway. So I don't think this helps unless they had sufficient foresight and that flag already exists.

Comment 24

[Paul Blinzer]

There is actually such a setting to select a default GPU in the Control application. But that typically defeats the purpose for a lot of users that expect the discrete GPU to be used for the majority of the applications (= perceived performance issue). The issue is typically caused by plugins like Flash, which do not have an understanding of multi-GPU systems. Native HTML5 playback should operate better here.

Comment 25

[Liz Henry (:lizzard)]

Milan, gcp, can you help find an owner for this bug? 
What is it that we may want to back out, mentioned in comment 19?

Comment 26

[George Wright (:gw280) (needinfo me!)]

The change that caused this was backed out a few days after it landed, and it's not going to be landed again. This bug should be closed as invalid.