895225 - Investigate why there are multiple gigabytes of write I/O during xpcshell tests

Status: NEW

(Testing :: XPCShell Harness, defect)

Description

[Gregory Szorc [:gps]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=25391634&tree=Mozilla-Central&full=1 reports nearly 10GB of write I/O during xpcshell test runs on this OS X 10.8 opt build. That seems excessive to me. I think someone should investigate where that I/O is coming from so it can be reduced if possible.

Since the resource monitoring is at the system level, it could be a background service incurring all the I/O.  However, other logs for this job incurred 8.5 and 9.5 GB, so it's likely either something always running or its the tests.

Comment 1

[Gregory Szorc [:gps]]

I applied the parallel xpcshell test patch and my SSD capped out at 268MB/s somewhere in Toolkit (I think it was during Places tests).

Comment 2

[Gregory Szorc [:gps]]

Mihnea: You said you had instrumented the xpcshell test runner to collect process counters. Do you have any results worth publishing?

It was speculated offline that much of this I/O could come from profile first-run activity. For this to be true, I'd expect a graph of write bytes per tests to consist of a long tail of similar height.

Also, Mihnea's try pushes with excessive amounts of parallel execution of xpcshell tests seem to indicate that the xpcshell test suite is I/O bound, not CPU bound, as we received no wall time or CPU % improvements at higher parallel levels while I/O wait times increased. This is surprising to me as I thought most tests wouldn't be doing that much I/O! Depending on the findings, we should consider optimizations to decrease I/O or automation improvements such as running tests from a RAM disk. But let's collect data on where the I/O is coming from first.

Comment 3

[Mihnea Dobrescu-Balaur (:mihneadb)]

I added support for psutil to the new parallel harness. Now that it has landed and things have cooled off a bit, I'll add per-test data collecting and start studying that data.

Should this be enabled all the time or should we keep it behind a flag (--profile/--instrument) ?

Comment 4

[Gregory Szorc [:gps]]

I'm going to move this into Testing :: xpcshell for now. Will open a new bug to track the RelEng side of things.