Open Bug 961234 Opened 6 years ago Updated 2 years ago

"REFTEST TEST-UNEXPECTED-FAIL | | EXCEPTION: No tests to run" when running parallel reftests in a directory with few tests

Categories

(Testing :: Reftest, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

People

(Reporter: dholbert, Unassigned)

References

(Depends on 1 open bug)

Details

STR:
Run reftests with --parallel in a directory with not-too-many tests, like so:
 ./mach reftest --parallel layout/reftests/svg/moz-only/

ACTUAL RESULTS: The output ends with something like:
{
 0:46.88 REFTEST INFO | runreftest.py | Running tests: end.
 0:46.88 REFTEST INFO | Result summary:
 0:46.88 REFTEST INFO | Successful: 14 (14 pass, 0 load only)
 0:46.88 REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected
pass, 0 unexpected asserts, 0 unexpected fixed asserts, 0 failed load, 1
exception)
 0:46.88 REFTEST INFO | Known problems: 3 (3 known fail, 0 known
asserts, 0 random, 0 skipped, 0 slow)
 0:46.89 reftest failed:
 0:46.89 REFTEST TEST-UNEXPECTED-FAIL | | EXCEPTION: No tests to run
 0:46.89 REFTEST TEST-UNEXPECTED-FAIL | | EXCEPTION: No tests to run
 0:46.89 make: *** [reftest] Error 1
}

EXPECTED RESULTS: No errors.

I'm not 100% sure, but I'm guessing the problem is that there aren't enough tests in that directory, and so some of our processes don't have any tests to run. From running...
  grep "==" layout/reftests/svg/moz-only/reftest.list  | wc -l
... it looks like there are 18 tests. I think that's more than the number of reftest-processes that --parallel gives me, but maybe each process claims several tests, and so the last process is left with none, or something? (Or this guess could just be wrong.)
How many CPUs does Python think you have? Can you paste the output of:
python -c "import multiprocessing; print multiprocessing.cpu_count()"
...and this was on Linux, right? We will wind up with 2*cpu_count + 1 jobs on non-Windows:
http://hg.mozilla.org/mozilla-central/annotate/bf0e76f2a7d4/layout/tools/reftest/runreftest.py#l243
This was on linux, yes.(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> How many CPUs does Python think you have? Can you paste the output of:
> python -c "import multiprocessing; print multiprocessing.cpu_count()"

8

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> ...and this was on Linux, right?

Yes.
Okay, I'd have to do some digging here but I suspect there are two bugs:
1) You'll note we add a special job there for "needs-focus" tests. I don't think that directory you mentioned in comment 0 has any needs-focus tests, so one of those errors is probably from trying to run that nonexistent set of tests.
2) We're spawning 2*cpu_count jobs, so 16 in your case, for the remaining tests. That means we're splitting 18 tests over 16 jobs, so I could imagine a rounding error causing us to have a chunk with no tests.
You need to log in before you can comment on or make changes to this bug.