924944 - Long delay when running tests due to installing/updating test files

Status: NEW

(Firefox Build System :: General, defect)

Description

[Honza Bambas (:mayhemer)]

One of the spawn subprocesses eats 100% of a core:

"c:\Mozilla\mozilla-build\python\python.exe" "-c" "from multiprocessing.forking import main; main()" "--multiprocessing-fork" "548"


When running a test in a loop with |while ./mach mochitest-plain single/test/to/run; do true; done| it delays repeated execution a lot.  I'm right now trying to reproduce a rare crash (ones succeeded bug log was incomplete).  Now trying again, the test it self is done in a second, but each run waits some 10s to start... 

Very annoying and contra-productive.

Comment 1

[Gregory Szorc [:gps]]

I wonder where multiprocessing is entering the equation. Can you isolate this to mach or does the issue persist while running mochitests through make targets?

Comment 2

[Honza Bambas (:mayhemer)]

while TEST_PATH=content/events/test/test_bug409604.html pymake -C _obj-browser-opt/ mochitest-plain | grep 'exit 0'; do true; done

it way faster.  So it's a mach issue.

(sorry for the confusion on IRC previously, I had a typo.)

Comment 3

[Gregory Szorc [:gps]]

Please run Python under a profiler and submit the raw results for us to analyze.

>>> python -m cProfile -o mach_profile mach mochitest-plain content/events/test/test_bug409604.html

This will produce a "mach_profile" file.

You may want to ctrl-c after the delay to limit the amount of "noise" in the results.

Comment 4

[Honza Bambas (:mayhemer)]

Attachment: mach_profile.zip

Ctrl-C prevents creation of the profile log.  So I here is a complete log only.

Comment 5

[Gregory Szorc [:gps]]

The profile shows an initial delay of ~9s installing tests. This action is performed every time you run tests to ensure test files are up to date.

There are a number of ways to fix this:

1) Buy a faster machine, notably a better SSD. Although, stats on Windows are slow, so this may have limited gains.
2) We can look into making test installation faster. While we just rewrote this code in bug 918392 to make it faster, there might be gains left on the table. We also could look at separate manifests for each _tests subdirectory to reduce the total amount of work needed when running tests of a certain flavor.
3) We can add modes to the test runners that don't reinstall tests. I would prefer #2 over this, however.

Comment 6

[Honza Bambas (:mayhemer)]

(In reply to Gregory Szorc [:gps] from comment #5)
> The profile shows an initial delay of ~9s installing tests. This action is
> performed every time you run tests to ensure test files are up to date.
> 
> There are a number of ways to fix this:
> 
> 1) Buy a faster machine, notably a better SSD. Although, stats on Windows
> are slow, so this may have limited gains.

I'm on i7@3.9 w/ SSD

> 2) We can look into making test installation faster. While we just rewrote
> this code in bug 918392 to make it faster, there might be gains left on the
> table. We also could look at separate manifests for each _tests subdirectory
> to reduce the total amount of work needed when running tests of a certain
> flavor.

Why is this actually happening during test RUN and not just during test BUILD?

> 3) We can add modes to the test runners that don't reinstall tests. I would
> prefer #2 over this, however.

Comment 7

[Gregory Szorc [:gps]]

(In reply to Honza Bambas (:mayhemer) from comment #6)
> Why is this actually happening during test RUN and not just during test
> BUILD?
> 
> > 3) We can add modes to the test runners that don't reinstall tests. I would
> > prefer #2 over this, however.

Because when developers modify a test file and then go to run the tests, they want want those test changes to "just work." Having a separate step to update tests before running them is a horrible and unintuitive pattern, especially for new contributors.

We'll make things faster.

Comment 8

[Honza Bambas (:mayhemer)]

Thanks.  Yeah, that is cool to have this feature (I often had hacks like |mach build /test && mach mochi /test|.  But it should run only on the one directory you run the test in.  Or even on the single test file, not on the whole tree or module.