673310 - Investigate speeding up /tests/content/media/test/test_progress.html

Status: RESOLVED WONTFIX

(Core :: Audio/Video: Playback, defect, P2)

Description

[Jonathan Griffin (:jgriffin)]

As part of the BuildFaster project (see https://wiki.mozilla.org/ReleaseEngineering/BuildFaster), we are investigating whether certain slow mochitests can be sped up.

test_progress.html is a slow test, average execution time on a recent run (in ms):

	win32-debug, 26033
	linux-opt, 18029
	macosx64-debug, 26663
	win32-opt, 18457
	linux64-debug, 23020
	linux64-opt, 18042
	macosx-debug, 25295
	linux-debug, 25497

Comment 1

[Jonathan Griffin (:jgriffin)]

Chris, as the original author of this test, can you help investigate whether it could be sped up, e.g., by using shorter media files?

Comment 2

[Chris Pearce [:cpearce (Not reading bugmail)]]

Sure, I'll take a look. Do you have a list of other long running media tests? I always find that test_playback and test_seek are the longest running, test_progress usually only takes about 10s on my machine.

It should be safe to increase the parallelism of the media tests without increasing our random orange, now that bug 592833 has landed.

How many cores do the releng test machines have?

Comment 3

[Jonathan Griffin (:jgriffin)]

A list of all the slow tests is at http://people.mozilla.org/~jgriffin/mochitest-more-than-10s.txt

> How many cores do the releng test machines have?

I'm cc'ing catlee who should be able to answer this.

Comment 4

[Chris AtLee [:catlee]]

(In reply to comment #3)
> A list of all the slow tests is at
> http://people.mozilla.org/~jgriffin/mochitest-more-than-10s.txt
> 
> > How many cores do the releng test machines have?
> 
> I'm cc'ing catlee who should be able to answer this.

/proc/cpuinfo reports two Intel(R) Core(TM)2 Duo CPU P7550  @ 2.26GHz.

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

I've investigated speeding up the media mochitests by increasing the parallelism. With bug 664341 applied, we can safely dial-up PARALLEL_TESTS in content/media/test/manifest.js to increase the number of cases each media test runs concurrently.

I instrumented the media mochitests, and recorded how long the mochitests took on TryServer with varying values of PARALLEL_TESTS. Results tabulated here:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Ahq_Zda4jJ9WdERrakoweFJGYVZBeTUxSE1JVnZ1bEE&hl=en_US

Conclusion: If we dial up PARALLEL_TESTS to 20, we get an average speedup of the media mochitests of about 35% on opt builds and 18% on debug builds.

Note it appears that the media mochitests aren't running on Android yet. Setting PARALLEL_TESTS=20 may hurt performance there once tests start running there.

I think we should increase PARALLEL_TESTS to 20 to speed up test runs. We may need to special case it in future for android.

Comment 6

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Patch: make MediaTestManager less "batchy"

The MediaTestManager currently runs tests in batches of size PARALLEL_TESTS. It waits until all tests in the batch are finished before starting a new batch of the same size. We'd be quicker if we just started the next test when we dropped below PARALLEL_TESTS running. This patch makes that change.

This patch also adds recording of start and end time of tests, so we can track that if need be.

Comment 7

[Matthew Gregan [:kinetik]]

Comment on attachment 548925 [details] [diff] [review]
Patch: make MediaTestManager less "batchy"

Review of attachment 548925 [details] [diff] [review]:
-----------------------------------------------------------------

Comment 8

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 548925 [details] [diff] [review]
Patch: make MediaTestManager less "batchy"

I will land this separately in bug 683421.