980499 - Switch to gaia-marionette-parallel on ci to speed up builds

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: JSMarionette, defect)

Description

[Gareth Aye [:gaye] (back from PTO)]

in bug 930784, we added a gaia-marionette-parallel binary that mimics the api of the current gaia-marionette and uses parallel-mocha to distribute our js ui test cases over several concurrent b2g-desktop instances. This produces a significant speedup (4X speed with 4 concurrent b2g-desktop instances on my computer). We should migrate our ci builds on travis and tbpl to use the parallel test runner once it's stable.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Based on https://github.com/mozilla-b2g/gaia/pull/16868 and the code in mocha-parallel, it looks like the same profile (directory) is used for all parallel instances.

If that is the case, I'm not sure how this actually works since every b2g-desktop instance after the first one should see the ".parentlock" file in the profile and then fail to start.  Unless we're clobbering the .parentlock file, in which case I think I'd still be worried about weird interaction between different runs since SQLite will absolutely let multiple processes use the same database (assuming we aren't being protected by having opened the file before it was replaced by a different one, so subsequent readers/writers see a different file.)

The solution is fairly straightforward in that we just need to use and generate separate profiles for each parallel task.  (And can reuse the profile within that task.)

Comment 2

[Gareth Aye [:gaye] (back from PTO)]

Hey :asuth,

It's my understanding that we use the same baseProfile, but not the same profile.

https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/profile_builder.js#L98
https://github.com/mozilla-b2g/marionette-profile-builder/blob/master/index.js#L78
https://github.com/mozilla-b2g/mozilla-profile-builder/blob/master/lib/createprofile.js#L27

roughly trace the calls that get made to make the profile and then copy it for use by various b2g-desktop instances.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Aha!  Thanks for clarifying and providing explicit links to help me follow along/trace.  That would be great to have in the uber-detailed how things work page, I think.

Then everything seems awesome.  Fast parallel builds!  Wooooooo!

Comment 4

[Kevin Grandon :kgrandon]

Due to the current state of the bottlenecked builds on travis, wouldn't this slow down the queue overall? I would assume that the aggregate suite time on travis would be increased due to additional downloading and installing of the environment/packages. I think this is a great effort and something we want in the future, I just have my concerns about turning it on today while we are limited in capcacity.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

It could slow things down if our tests are written so that they are saturating/wasting the VM CPU.  If the Travis VMs are oversubscribed to the extent that each VM gets less than a single cpu core/thread allocated to it, then yes, contention would likely make things worse in general, especially as I/O overhead with reduced free memory would impact caching so that churn could increase.

The upside is if we do have spare CPU and/or tests are badly written so that they are doing things like waiting for timeouts, etc.

This definitely is useful on developer machines where the dev has a quad-core, hyper-threaded machine, which should be all MoCo employees who aren't using ultraportables.

Comment 6

[Firefox Bug Husbandry Bot]

Bulk closed as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1422750