We think we can run multiple Android x86 emulators on each test slave, allowing more tests to be run per unit of hardware. Before we order that hardware, we should determine how many emulators can be run simultaneously without causing trouble.
I am testing on talos-linux64-ix-001.test.releng.scl3.mozilla.com with emulator64-x86 from the android 17 sdk. Emulator startup is intense: Each emulator instance will take as much cpu as it can get for 60 to 200 seconds at startup. top reports 100% cpu and about 6% mem when starting a single emulator, and that is sustained for the 60 - 200 seconds duration of startup. Starting multiple emulators at once increases startup time to 300 - 600 seconds. More troubling, starting 4 or more emulators at once generally results in one or more emulator instances "spinning" at 100% cpu indefinitely. I assume that we can stagger emulator startup; otherwise, I recommend only running 1 emulator at a time. With staggered startup, I can run 4 *unloaded* (no tests running) instances at once without any sign of trouble. top shows each instance using about 10% cpu and 6% memory. On to testing multiple emulators, with each running tests....
That is discouraging. Launching two emulators concurrently on my laptop takes around two minutes from start to the watcher launching SUTAgent.
I have tried to run up to 8 emulators simultaneously, with staggered launch times. Sometimes all 8 emulators launch and are functional, but more than 50% of the time, the 7th or 8th emulator crashes within 30 seconds of launch. I see no error messages or hint of the cause of these crashes. ==> Up to 6 emulators can be run at one time.
...but see also https://bugzilla.mozilla.org/show_bug.cgi?id=892123#c7 -- test run times increase significantly if more than 4 tests are run at one time. ==> Up to 4 emulators should be run at one time.
Is this then a no-go? If we went this way, mozharness would have to trigger the N number of jobs, somehow report each clearly separately and know when all N jobs have terminated so we can ask the machine to reboot itself at the end of it.
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #5) > Is this then a no-go? > > If we went this way, mozharness would have to trigger the N number of jobs, > somehow report each clearly separately and know when all N jobs have > terminated so we can ask the machine to reboot itself at the end of it. It is possible to run 4 emulators at the same time and run tests concurrently, reliably, as far as I can tell from my experiments on the one loaner. I have heard rumors that mozpool/mozharness has support for assigning jobs to the emulators appropriately, but I do not know any details.
Mozharness has not ever run 4 jobs at once. We will have to see how it fairs. Do you have an ETA on when I could get some of those scripts so I could try them on staging? Just a clarificaiton: - mozpool is *only* used for Pandas and Tegras. For this project, we're *only* going to be running on emulators that run on iX test machines.
Would it be difficult to extend mozpool to support managing the emulators? I know next to nothing about mozpool, but it seems like the problem of managing multiple emulators is similar to managing a bunch of pandas or tegras.
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #7) > Do you have an ETA on when I could get some of those scripts so I could try > them on staging? Available now in bug 895186 -- pardon the delay!
I think we're just getting a bunch of concepts mixed in the same convo. * Buildbot instance runs on foopy and requests a device from mozpool * A Foopy (which is an iX machine) has N number of buildbot instances named like one of the pandas they manage * A Linux 64 ix machine will be running only one buildbot instance named after its own hostname ** The iX machine will be running a mozharness script ** The mozharness script (if written to do so) could spawn 4 emulator instances on the iX machine ** The mozharness script would need to know when those 4 different jobs will end In other words, have any of you been able to write a mozharness script that triggers 4 different emulator jobs? IIUC, the emulator job runs on the iX machine and not on an external device; Am I assuming correctly? I can have something running on staging by the end of the week.
Sorry, I had been thinking the ubuntu ix machine would be like a foopy running multiple buildbot instances corresponding to the emulator instances rather than trying to handle this from the same mozharness script. Because of that, I never tried multiple jobs from one script.
Not needed. This got clarified already.
gbrown, I don't think we need to keep this bug open if we have bug 895186. Is that OK?
Yes - fine with me.