1272851 - Use object directory for temporary directory to decrease chances of indexing and scanning

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect)

Description

[Gregory Szorc [:gps]]

On Windows, each xpcshell tests appear to set up a dummy profile in ~\appdata\local\temp. I have Windows Defender running and MsMpEng.exe is eating up to 1 full CPU core at times during the xpcshell tests, slowing down `mach xpcshell-test` considerably.

This wouldn't be happening if we put temporary files in the object directory (we instruct people to exclude the object directory from virus scanners and other indexers for performance reasons).

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1272851 - Allow temporary directory to be specified to xpcshell test harness;

A subsequent commit will change where the temporary directory is located
in certain test invocations. To do this, we need to teach the xpcshell
harness to use an alternate temporary directory instead of using the system
default (likely specified from TEMP* environment variables).

Review commit: https://reviewboard.mozilla.org/r/55504/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/55504/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1272851 - Use objdir for temporary directory for `mach xpcshell-test`;

When running xpcshell tests on my i7-6700K, the Windows Defender
MsMpEng.exe process regularly consumes up to 50% CPU - 2 whole cores.
This is because it is canning the thousands of files we create in the
temporary directory. (We create a profile for every xpcshell.exe process.)

This commit changes the behavior of `mach xpcshell-test` to use a temporary
directory under the topobjdir. We encourage people to disable file indexing
and A/V scanning in the source and object directories because this can add
overhead. So by putting temporary files in the object directory, we should
minimize the chances of file scanning slowing down test execution.

On my machine (which has Windows Defender disabled in my source and objdirs),
`mach xpcshell-test` execution time drops from ~13:40 to ~7:30. Seriously.

I'm told we have Windows Defender disabled in automation, so this hack
shouldn't need to be investigated there. i.e. this is a pure local development
win.

Review commit: https://reviewboard.mozilla.org/r/55506/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/55506/

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 8756971 [details]
Bug 1272851 - Allow temporary directory to be specified to xpcshell test harness;

https://reviewboard.mozilla.org/r/55504/#review55496

::: testing/xpcshell/runxpcshelltests.py:1165
(Diff revision 1)
>  
>          self.xpcshell = xpcshell
>          self.xrePath = xrePath
>          self.appPath = appPath
>          self.symbolsPath = symbolsPath
> +        self.tempDir = os.path.normpath(tempDir)

Wouldn't this break with the default value of `None`? Seems like you should set the default value to the system temp dir.

Comment 4

[(not currently active) Ted Mielczarek]

Comment on attachment 8756972 [details]
Bug 1272851 - Use objdir for temporary directory for `mach xpcshell-test`;

https://reviewboard.mozilla.org/r/55506/#review55498

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8756971 [details]
Bug 1272851 - Allow temporary directory to be specified to xpcshell test harness;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/55504/diff/1-2/

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8756972 [details]
Bug 1272851 - Use objdir for temporary directory for `mach xpcshell-test`;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/55506/diff/1-2/

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e80e15a2462
Allow temporary directory to be specified to xpcshell test harness; r=ted
https://hg.mozilla.org/integration/mozilla-inbound/rev/4cbcf4d53eb8
Use objdir for temporary directory for `mach xpcshell-test`; r=ted

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4e80e15a2462
https://hg.mozilla.org/mozilla-central/rev/4cbcf4d53eb8