973946 - Jit-tests run slowly on linux test machines

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Dan Minor [:dminor]]

We're up over 100 minutes for debug and ASAN on linux and opt isn't great either, at around 50 minutes.

Comment 1

[Dan Minor [:dminor]]

I inserted some timing into make check on the build machines:

debug jit-test on build machine: 1120 seconds (-> ~18 minutes)
debug jit-test on build machine (parallel execution disabled): 2372 seconds (-> ~39 minutes)

The linux test machines are single core so the parallel execution disabled case is the fairest comparison. There is also the overhead associated with setting up mozharness, pulling down the test package, etc. but I would say we're still running twice as slow on the test machines.

For comparison, running the tests on my not particularly fast laptop (opt build) takes under a minute.

Comment 2

[Chris AtLee [:catlee]]

(In reply to Dan Minor [:dminor] from comment #1)
> I inserted some timing into make check on the build machines:
> 
> debug jit-test on build machine: 1120 seconds (-> ~18 minutes)
> debug jit-test on build machine (parallel execution disabled): 2372 seconds
> (-> ~39 minutes)
> 
> The linux test machines are single core so the parallel execution disabled
> case is the fairest comparison. There is also the overhead associated with
> setting up mozharness, pulling down the test package, etc. but I would say
> we're still running twice as slow on the test machines.
> 
> For comparison, running the tests on my not particularly fast laptop (opt
> build) takes under a minute.

That's not really a fair comparison...How long do debug tests take on your laptop?

Comment 3

[Chris AtLee [:catlee]]

So https://tbpl.mozilla.org/php/getParsedLog.php?id=34884740&tree=Cedar&full=1 took 44 minutes in total, 41.5 of which were inside the jit tests themselves. So we're not spending too much time on overhead here.

Comment 4

[Dan Minor [:dminor]]

(In reply to Chris AtLee [:catlee] from comment #2)
> (In reply to Dan Minor [:dminor] from comment #1)
> > I inserted some timing into make check on the build machines:
> > 
> > debug jit-test on build machine: 1120 seconds (-> ~18 minutes)
> > debug jit-test on build machine (parallel execution disabled): 2372 seconds
> > (-> ~39 minutes)
> > 
> > The linux test machines are single core so the parallel execution disabled
> > case is the fairest comparison. There is also the overhead associated with
> > setting up mozharness, pulling down the test package, etc. but I would say
> > we're still running twice as slow on the test machines.
> > 
> > For comparison, running the tests on my not particularly fast laptop (opt
> > build) takes under a minute.
> 
> That's not really a fair comparison...How long do debug tests take on your
> laptop?

Sorry I muddied the waters, the point here is that the debug tests are currently twice as slow on test machines as on build machines.

Comment 5

[Dan Minor [:dminor]]

So, assuming faster test machines aren't in the cards, I guess we're left with implementing chunking in the harness and running jit-tests in two chunks on the linux test machines.

Comment 6

[Dan Minor [:dminor]]

I was thinking of running jit-tests in two chunks on linux machines. That should get debug and ASAN runs completing in under an hour and the opt run times comparable to win and os x.

Philor, does this sound ok to you?

Comment 7

[Phil Ringnalda (:philor)]

That will give you a delightful opportunity to learn about the innards of the parser behind trychooser as you copy the stuff that makes -u mochitests run all of mochitest-* (there might be similar, and more directly corresponding, stuff that thinks it makes -u reftest run both reftest and reftest-1...reftest-99, but I think if so, it doesn't actually work).

Comment 8

[Dan Minor [:dminor]]

Running in two chunks allows everything to complete in under an hour. ASAN is still the slowest, each chunk takes around 52 minutes. I think this sufficient to be able to schedule the jit-tests in bug 973900, so I removing this bug a blocker.

I'll leave this bug open as the jit-tests are still the slowest test suite on Linux. I think the jit-tests would benefit from a different AWS instance type since they have the capability to run in parallel and the current test machines are single core.