1166637 - Content sandboxing blocks Nvidia shader cache

Status: RESOLVED WORKSFORME

(Core :: Security: Process Sandboxing, defect, P1)

Description

[Bob Owen (:bobowen)]

As part of the fix for bug 1162327, I removed the setting of the TEMP and TMP environment variables to point to the low integrity temp.

This means that we are now blocking the Nvidia shader cache [1] again.

I did this for two reasons:
* I'd seen some locking issues that meant that the low integrity temp directory wasn't getting deleted (although that probably won't be a problem now, with the fix in bug 1162327).
* I decided that fooling things like the Nvidia cache into writing to a new location, possibly increasing their own enforced quota (256MB) by multiple times (albeit temporarily), was a bad idea.

If we think that the shader cache will be beneficial for us, I think the best way of giving access to it will probably be via sandbox policy rules to the original location.

Also, as far as I can tell Chrome currently blocks access to the cache.
Certainly, I can see some access denied logs in procmon as the Chrome GPU process starts up.
In addition, I don't see any rules for it in their sandboxing policy.

[1] Discussed about half way down this article:
    http://www.geforce.com/whats-new/articles/nvidia-geforce-337-88-whql-watch-dog-drivers

Comment 1

[Bob Owen (:bobowen)]

roc, jrmuizel - I'm not sure if you're the right people to answer this, but I'm guessing you know who is if not.

Do you think that the Nvidua Shader Cache is something that will be beneficial to the content processes (and possibly other processes)?
Whether we'd want to give access from a security point of view is of course a different question.

As I point out in the description Chrome seems to be blocking it at the moment, but that's just the default for the sandbox.

Just found [1], so it looks like they are at least aware of it.
It also looks like there is an AMD equivalent.

[1] https://code.google.com/p/chromium/issues/detail?id=483673

Comment 2

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I have no idea really, but presumably it has some benefit...

Comment 3

[Jeff Muizelaar [:jrmuizel]]

Do you have more info on what the shader cache does? We compile shaders in both the parent and the child processes, so it's conceivable that the cache provides some benefit.

Comment 4

[Bob Owen (:bobowen)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #3)
> Do you have more info on what the shader cache does? We compile shaders in
> both the parent and the child processes, so it's conceivable that the cache
> provides some benefit.

Above what is says in that article, i.e. that it caches the compiled shaders, no.

I understand very little about shaders, although the cache looks like it might be storing application specific binaries.
Are shaders generally application specific?

It sounds like there's at least a reasonable chance it could be beneficial, so I think there are two choices to stop blocking it.

* Put the code back in that changes the TEMP and TMP variables to point to our low integrity temp.
This then means a new cache will be created there, but it will be deleted when Firefox is shutdown.
If these compiled shaders are held in memory and the cache is only really useful at start-up or at least wouldn't be re-read during a Firefox session, then I guess this provides no benefit.

* Add a sandbox policy rule to allow read-write access to the shader cache under %TEMP%\NVIDIA Corporation\NV_Cache.
This would allow it to use the normal cache, but has the slight disadvantage that the child process will have to ask the broker (chrome) process to create any files for it, passing the handle over IPC.
If it is just opening a file for read-access, it shouldn't need to ask the broker.

I'll also try and keep an eye on what they decide in the Chromium bug.

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

At a total guess, the shader cache might be beneficial for repeated restarts of a WebGL application.

Comment 6

[Bob Owen (:bobowen)]

There's been some more discussion on the Chromium issue for the AMD version of this cache.

It looks like AMD might be making some changes to make their cache low integrity friendly.

I've asked in the bug if any similar discussions have been had with Nvidia.

As an aside, they also mention that they do some of their own caching in Chromium using ARB_get_program_binary.
It looks like we have some stubbed out wrapper for this.

We seem to have an implemented wrapper for OES_get_program_binary.
I'm not sure what the difference is or whether we're even using this.

Comment 7

[Trevor Rowbotham [:rowbot]]

This is probably still benficial even if the sandbox were to delete the files every time Firefox was closed.  If the sandbox were to delete the files on every close, we wouldn't see any benefits in terms of browser startup, but this is probably a performance boost for games in the browser since Firefox would not always need to recompile shaders during gameplay resulting in a smoother experience.  I guess this might also benefit opening things like google maps multiple times in a session and other webgl stuff like roc mentioned.

There doesn't seem to be too much detail about the shader cache available on the web apart from the article in comment #1. This feature can be controlled on a per application basis through the nVidia control panel, but the default setting is On.  From what I've gathered it seems as though the shader cache can both help and hurt performance based on different factors.  For example, someone using a laptop with a slow harddrive may have better performance with the shader cache off since the cpu may be able to compile a non-complex shader faster than reading a precompiled one from the harddrive.  In the end though, I guess the whole point of the cache is about being able to cache the complex shaders even if you potentially take a small hit on simple ones.

Comment 8

[Jim Mathies [:jimm]]

We're going to reach out to nvidia to try and determine what impact this might have.

Comment 9

[(No longer employed by Mozilla) Aaron Klotz]

I emailed my NVIDIA contact, awaiting a response...

Comment 10

[Bob Owen (:bobowen)]

One thing gcp pointed out on IRC is that NVIDIA's own release notes about this talk about it potentially reducing load times, which makes sense.

So, this shouldn't affect runtime performance.

Comment 11

[Jim Mathies [:jimm]]

not blocking, but we want to keep track of this.

Comment 12

[Jukka Jylänki]

Hey, I am routinely seeing shader compilation (via glLinkShader) take up a large portion of time at startup of asm.js/WebGL games on Windows. One recent case was a report from Unreal Engine 4 developer at https://answers.unrealengine.com/questions/396404/html5-loading-times.html.

Comment https://bugzilla.mozilla.org/show_bug.cgi?id=918941#c11 suggests that NVidia's own shader compilation cache is extremely efficient to reduce shader compilation times, in Unity3D Dead Trigger 2 demo case, going down from 1697 msecs to 64 msecs on second run.

Walter did not profile Windows there, only Linux. Briefly testing on Windows+NVidia GPU on a slow page, I think I am seeing that the shader cache is not functioning, or at least the first run and subsequent runs seem to take equal amount of time in linking, when looking at geckoprofiler runs.

Above there are some comments that environment variables TEMP and/or TMP would be controlling this. I wonder if this is something we could have a relatively low hanging easy fix for? Is there a way to "hot-fix" locally for experimentation on what the impact would be?

Comment 13

[Bob Owen (:bobowen)]

(In reply to Jukka Jylänki from comment #12)

> Above there are some comments that environment variables TEMP and/or TMP
> would be controlling this. I wonder if this is something we could have a
> relatively low hanging easy fix for? Is there a way to "hot-fix" locally for
> experimentation on what the impact would be?

Yes we thought this might be partially resolved by another bug that changes TEMP and TMP to point to a temp directory that is writeable from the sandbox, but it looks like the shader cache is picking up the variable before this.

Bug 1257276 has been filed to allow setting the environment variables from the start.

It would probably be better all round if the graphics drivers detected when they were at low integrity and then used a its own directory under %USERPROFILE%\AppData\LocalLow.

For testing you could disable the content sandbox on Nightly by setting env var MOZ_DISABLE_CONTENT_SANDBOX=1.

Comment 14

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Yes, this is definitely going to be a big issue for games (asm.js/webassembly especially) -- they often have lots of shaders that they expect to spend time compiling up front once and then have subsequent startups be fast.

Comment 15

[Jukka Jylänki]

Thanks Bob, the MOZ_DISABLE_CONTENT_SANDBOX=1 environment variable is an interesting one to try out.

Testing on my HASWELL computer with current Firefox Nightly 64-bit:

Windows 10 Home 64-bit
3.0 GHz Intel 8-Core i7-5960X
32GB of RAM
3840x2160 pixels display

SLI 2x ASUS NVIDIA GeForce GTX 980 Ti, 4+4GB of VRAM, driver version 361.91
	- nvd3dumx.dll, nvwgf2umx.dll:
		- version 10.18.13.6191
		- date 2/8/2016
		- D3D Version 12.1, WDDM 2.0, WHQL approved

I see that if I nuke the directory C:\Users\Jukka\AppData\Local\Temp\NVIDIA Corporation\NV_Cache and launch Firefox with empty tabs, I get always 5 files populated in the cache folder, independent of whether MOZ_DISABLE_CONTENT_SANDBOX is not defined or 1. With this as a "baseline", and using the deterministic run of 

https://s3.amazonaws.com/mozilla-games/tmp/2016-04-23-PlatformerGame/PlatformerGame.html?cpuprofiler&playback

as the test sequence, I get the following:

1. Undefine MOZ_DISABLE_CONTENT_SANDBOX.
2. Nuke shader cache folder.
3. Launch Firefox. Shader cache folder gets 5 files.
4. Run PlatformerGame first run. Results:
    - 134 shaders compiled in 411 msecs total. (glCompileShader)
    - 102 shader programs linked in 6044 msecs total. (glLinkShader)
5. The same 5 files still living in shader cache folder.
6. Close and restart browser to run PlatformerGame second time. Results:
    - 134 shaders compiled in 434 msecs total.
    - 102 shader programs linked in 6127 msecs total.
7. The same 5 files still living in shader cache folder.

and with content sandboxing disabled:

1. Set MOZ_DISABLE_CONTENT_SANDBOX=1.
2. Nuke shader cache folder.
3. Launch Firefox. Shader cache folder gets 5 files.
4. Run PlatformerGame first run. Results:
    - 134 shaders compiled in 381 msecs total.
    - 102 shader programs linked in 5628 msecs total.
5. Shader cache folder now has 13 files.
6. Close and restart browser to run PlatformerGame second time. Results:
    - 134 shaders compiled in 346 msecs total.
    - 102 shader programs linked in 5370 msecs total.

This suggests that the NVidia shader cache is working with MOZ_DISABLE_CONTENT_SANDBOX=1 (and isn't working without). It's interesting that the performance improves immediately a bit on the first run, perhaps the game code compiles identical shader programs a few times, or it can reuse results if the programs have some same shaders, or similar. However the overall performance improvements are quite modest and only in the range of -10% reduced time spent in glLinkShader, and -18% for glCompileShader.

The critical bit is that shader compile times are insignificant compared to link times (though this was known already from https://bugzilla.mozilla.org/show_bug.cgi?id=918941#c11), but curiously I'm not getting at all the same ballpark of improvements as https://bugzilla.mozilla.org/show_bug.cgi?id=918941#c11. Not sure if this is a Linux vs Windows, or perhaps a D3D vs GL thing, or if my test method had a flaw. Need to poke a bit more on the shader cache enabled vs disabled bits with geckoprofiler, and try out the native GL backend to see how it compares.

Comment 16

[Jukka Jylänki]

Newer test case URL since comment 15:

https://s3.amazonaws.com/mozilla-games/ZenGarden/EpicZenGarden.html?playback&novsync&cpuprofiler

Requires Firefox 52 or newer. Visit the page, and observe the "shader compilation" loading bar at startup. Wait for a minute for the automated run to finish, At the end, the line like

>    "shaderCompilation": 7631.919999999999,

shows the time it took to compile shaders overall (all glCompileShader() + glLinkProgram() calls on the page aggregated)

Other test case URLs for similar workload:

https://s3.amazonaws.com/mozilla-games/tmp/2017-02-21-SunTemple/SunTemple.html?playback&novsync&cpuprofiler
https://s3.amazonaws.com/mozilla-games/tmp/2017-02-21-StylizedRendering/StylizedRendering.html?playback&novsync&cpuprofiler
https://s3.amazonaws.com/mozilla-games/tmp/2017-02-21-PlatformerGame/PlatformerGame.html?playback&novsync&cpuprofiler

Comment 17

[Jukka Jylänki]

An update to the STR above, the shader compilation step has been since upgraded from being only a page console log message to being explicitly shown in the loading screen. That is, instead of printing a log message, in the beginning there is a running counter that looks like

> Building shaders (23/119)

and when it finishes it briefly shows the overall time that all glCompileShader() + glLinkProgram() calls took.

In addition, after the playback has finished, the same timing is shown in the final results print:

> var results = {
>    ...
>    "shaderCompilation": 7631.919999999999,
>    ...
> };

Comment 18

[Jim Mathies [:jimm]]

Hey Jukka, is this still an issue? I see shader related cache files showing up in:

C:\Users\(username)\AppData\LocalLow\Mozilla\Temp-{f81f666a-de59-4173-bff4-71aa46918929}\NVIDIA Corporation\NV_Cache

when loading these demos.

Comment 19

[Jukka Jylänki]

(In reply to Jim Mathies [:jimm] from comment #18)
> Hey Jukka, is this still an issue? I see shader related cache files showing
> up in:
> 
> C:\Users\(username)\AppData\LocalLow\Mozilla\Temp-{f81f666a-de59-4173-bff4-
> 71aa46918929}\NVIDIA Corporation\NV_Cache
> 
> when loading these demos.

I can observe the same. However when I close the browser, the cached shaders are deleted. Is this intentional to be a session-specific cache? Then it will work only between page revisits on the same run, but not on page revisits across browser restarts.

Comment 20

[Jim Mathies [:jimm]]

(In reply to Jukka Jylänki from comment #19)
> (In reply to Jim Mathies [:jimm] from comment #18)
> > Hey Jukka, is this still an issue? I see shader related cache files showing
> > up in:
> > 
> > C:\Users\(username)\AppData\LocalLow\Mozilla\Temp-{f81f666a-de59-4173-bff4-
> > 71aa46918929}\NVIDIA Corporation\NV_Cache
> > 
> > when loading these demos.
> 
> I can observe the same. However when I close the browser, the cached shaders
> are deleted. Is this intentional to be a session-specific cache? Then it
> will work only between page revisits on the same run, but not on page
> revisits across browser restarts.

Yeah this is currently by design. I'll bring this up at our next team meeting.

Please reopen if you find additional issues.