1023595 - Schedule Mnw on cedar on emulator-jb and emulator-kk

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We'd like to try running emulator tests on emulator-jb and/or emulator-kk so that we can, among other reasons, see if those emulators experience fewer timeouts than emulator-ics:  bug 906716.

We'll start by scheduling one test suite on each of these emulators on cedar, so we can see how well they perform; I've chosen Mnw since they are a little more susceptible to timeouts than the others.  There's some concern that the newer emulators may need more RAM or CPU power, in which case the current AWS VM's that we're using may not produce good results.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule Mnw on cedar on emulator-jb and -kk,

Hopefully I've added the emulator-kk platform correctly here.

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

so I think for cedar to have emulator-kk in platforms, we need to add that key to BRANCH_UNITTEST_VARS -> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/b2g_config.py#154

But I'm not sure, still parsing what this changes.

::: mozilla-tests/b2g_config.py
@@ +1519,4 @@
>          'builds_before_reboot': 1,
>          'unittest-env': {'DISPLAY': ':0'},
>          'enable_opt_unittests': True,
> +        'enable_debug_unittests': True,

I'm confused what flipping this gets us?

It looks like on the cedar lines at the bottom we are adding opt_unittests.

Comment 3

[Jordan Lund (:jlund)]

hmm, so adding it to BRANCH_UNITTEST_VARS I think will at least not make buildbot barf but it will also add builders outside of Cedar.

So I think something is still not right here

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Jordan Lund (:jlund) from comment #2)
> Comment on attachment 8438026 [details] [diff] [review]
> Schedule Mnw on cedar on emulator-jb and -kk,
> 
> Review of attachment 8438026 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> so I think for cedar to have emulator-kk in platforms, we need to add that
> key to BRANCH_UNITTEST_VARS ->
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/
> b2g_config.py#154
> 
> But I'm not sure, still parsing what this changes.
> 
> ::: mozilla-tests/b2g_config.py
> @@ +1519,4 @@
> >          'builds_before_reboot': 1,
> >          'unittest-env': {'DISPLAY': ':0'},
> >          'enable_opt_unittests': True,
> > +        'enable_debug_unittests': True,
> 
> I'm confused what flipping this gets us?
> 
> It looks like on the cedar lines at the bottom we are adding opt_unittests.

You're right, since we're not specifically scheduling debug tests, the change on this line isn't needed yet.

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

on irc, you said you would have a look at this again and the last comment on this bug was about not needing the debug boolean flipped.

r- until next patch just so I can keep my bugzilla request queue organized.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule Mnw on cedar on emulator-jb and -kk,

Removed the unnecesary change to enable_debug_unittests, and added emulator-kk to BRANCH_UNITTEST_VARS.  This may add no-op builders to the other branches, but theoretically shouldn't cause anything to actually be scheduled, due to the opt_unittest_suites: [].  If that doesn't seem to be the case, we could add some logic here:

http://hg.mozilla.org/build/buildbot-configs/file/e3dd7657da2b/mozilla-tests/b2g_config.py#l1476

to skip adding anything for any branch but cedar, but since we haven't done that for emulator-jb, I don't know if it's needed.

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

ah, yes, it should be a no-op for other branches.

But, IIUC now, I think we may have it backwards: outside of cedar, we won't be adding builders, we will be adding schedulers. Those schedulers just won't have any builders attached to them.

something like:
   <buildbot.schedulers.basic.Scheduler> {'builderNames': [],
   'change_filter': <ChangeFilter on branch == fx-team-emulator-kk-opt-unittest>, etc ... }

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551

Comment 9

[Jordan Lund (:jlund)]

buildbot-config patch live in production :)

Comment 10

[Jonathan Griffin (:jgriffin)]

For some reason, the emulator-kk test is stuck in a pending state.  I must have missed something somewhere!

The emulator-jb job is running but unhappy; the emulator starts up and then dies a horrible death shortly thereafter.  I'm guessing the VM is underpowered, but it could also be a lib problem.  I'll probably end up needing to check out a slave to investigate.

Comment 11

[Jordan Lund (:jlund)]

I *think* we need to patch production_masters.py in a similar fashion as emulator-jb. Taking a look at this file, it seems emulator-kk might be out of our master's scope: http://mxr.mozilla.org/build/source/tools/buildfarm/maintenance/production-masters.json#331

Comment 12

[Jonathan Griffin (:jgriffin)]

Attachment: Add emulator-kk to build masters,

Comment 13

[Jordan Lund (:jlund)]

(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Created attachment 8441726 [details] [diff] [review]
> Add emulator-kk to build masters,

So I think this is right.

However since we don't appear to have emulator-kk tests anywhere yet, this is un-tested, and I have not written any patches for this file before, I am going to poll the experts.

catlee -

1) How do I verify we are ready for emu-kk tests? I see we have builds[1]
2) is this how we should go about adding? We have already enabled it on Cedar in buildbot-configs[2] but it looks like we have not added it to the limit platforms list like we did for emulator-jb on our test masters.
3) what about testing this patch? test-masters.sh should test our masters against this file. Is that enough for checking against spurious results?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=979450
[2] https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551

Comment 14

[Chris AtLee [:catlee]]

As long as these tests are only enabled on cedar, that's fine.

As for testing the patch, yeah, look at test-masters output. Perhaps use http://hg.mozilla.org/build/braindump/diff/055f2b128436/buildbot-related/dump_allthethings.sh to see what builders are added/removed across all our master types.

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,

Review of attachment 8441726 [details] [diff] [review]:
-----------------------------------------------------------------

Wow, dump_allthethings is pretty neat. ty catlee.

IIUC this looks right, this will only create the builder on cedar and this patch will result in a master picking it up and running it.

Comment 16

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,

https://hg.mozilla.org/build/tools/rev/5920f59b243e

Comment 17

[Jordan Lund (:jlund)]

huh, something is still not right here. I had to back it out: http://hg.mozilla.org/build/tools/rev/8da583a956b2

as it blew up when I put it in production. Not sure. Will have to test again. I probably won't be able to look at it until tomorrow unless my buildduty queue clears up a bit.

Comment 18

[Jordan Lund (:jlund)]

I had some time today to play with this again.

I don't know what went wrong with the reconfig to be honest. I ran this again off test-masters.sh and everything passed: https://pastebin.mozilla.org/5510612

This might be a situation where production-masters.json can break a reconfig and a rolling restart is required on our linux test machines. Kmoir/callek were forced into this situation last week from a similar change (reconfig failed but a stop/start worked).

I am going to ask coop as he is on buildduty this week. Coop -- jgriffin would like to land this but unless you can think of an alternative, a graceful restart will probably be the best route. I suppose at an off peak time would be best for pending count?

Comment 19

[Jonathan Griffin (:jgriffin)]

Thanks, I was just looking at this again myself and not seeing anything obvious.

Comment 20

[Chris Cooper [:coop] (he/him)]

jgriffin: I'm fine with you re-landing this. I think there were unresolved issues from an earlier reconfig last week that prevented this from sticking the first time.

Comment 21

[Jonathan Griffin (:jgriffin)]

(In reply to Chris Cooper [:coop] from comment #20)
> jgriffin: I'm fine with you re-landing this. I think there were unresolved
> issues from an earlier reconfig last week that prevented this from sticking
> the first time.

Thanks.  Two two:  https://hg.mozilla.org/build/tools/rev/90fd49290a90

Comment 22

[Jordan Lund (:jlund)]

Attachment: 140703_bug_1023595_emu-kk.patch

r=coop

Comment 23

[Jordan Lund (:jlund)]

Comment on attachment 8450466 [details] [diff] [review]
140703_bug_1023595_emu-kk.patch

https://hg.mozilla.org/build/puppet/rev/83c23b6da439

merged and live in prod