Closed Bug 1023595 Opened 6 years ago Closed 6 years ago

Schedule Mnw on cedar on emulator-jb and emulator-kk

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

Attachments

(3 files, 1 obsolete file)

We'd like to try running emulator tests on emulator-jb and/or emulator-kk so that we can, among other reasons, see if those emulators experience fewer timeouts than emulator-ics:  bug 906716.

We'll start by scheduling one test suite on each of these emulators on cedar, so we can see how well they perform; I've chosen Mnw since they are a little more susceptible to timeouts than the others.  There's some concern that the newer emulators may need more RAM or CPU power, in which case the current AWS VM's that we're using may not produce good results.
Hopefully I've added the emulator-kk platform correctly here.
Attachment #8438026 - Flags: review?(jlund)
Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

so I think for cedar to have emulator-kk in platforms, we need to add that key to BRANCH_UNITTEST_VARS -> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/b2g_config.py#154

But I'm not sure, still parsing what this changes.

::: mozilla-tests/b2g_config.py
@@ +1519,4 @@
>          'builds_before_reboot': 1,
>          'unittest-env': {'DISPLAY': ':0'},
>          'enable_opt_unittests': True,
> +        'enable_debug_unittests': True,

I'm confused what flipping this gets us?

It looks like on the cedar lines at the bottom we are adding opt_unittests.
hmm, so adding it to BRANCH_UNITTEST_VARS I think will at least not make buildbot barf but it will also add builders outside of Cedar.

So I think something is still not right here
(In reply to Jordan Lund (:jlund) from comment #2)
> Comment on attachment 8438026 [details] [diff] [review]
> Schedule Mnw on cedar on emulator-jb and -kk,
> 
> Review of attachment 8438026 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> so I think for cedar to have emulator-kk in platforms, we need to add that
> key to BRANCH_UNITTEST_VARS ->
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/
> b2g_config.py#154
> 
> But I'm not sure, still parsing what this changes.
> 
> ::: mozilla-tests/b2g_config.py
> @@ +1519,4 @@
> >          'builds_before_reboot': 1,
> >          'unittest-env': {'DISPLAY': ':0'},
> >          'enable_opt_unittests': True,
> > +        'enable_debug_unittests': True,
> 
> I'm confused what flipping this gets us?
> 
> It looks like on the cedar lines at the bottom we are adding opt_unittests.

You're right, since we're not specifically scheduling debug tests, the change on this line isn't needed yet.
Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

on irc, you said you would have a look at this again and the last comment on this bug was about not needing the debug boolean flipped.

r- until next patch just so I can keep my bugzilla request queue organized.
Attachment #8438026 - Flags: review?(jlund) → review-
Removed the unnecesary change to enable_debug_unittests, and added emulator-kk to BRANCH_UNITTEST_VARS.  This may add no-op builders to the other branches, but theoretically shouldn't cause anything to actually be scheduled, due to the opt_unittest_suites: [].  If that doesn't seem to be the case, we could add some logic here:

http://hg.mozilla.org/build/buildbot-configs/file/e3dd7657da2b/mozilla-tests/b2g_config.py#l1476

to skip adding anything for any branch but cedar, but since we haven't done that for emulator-jb, I don't know if it's needed.
Attachment #8439563 - Flags: review?(jlund)
Attachment #8438026 - Attachment is obsolete: true
Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

ah, yes, it should be a no-op for other branches.

But, IIUC now, I think we may have it backwards: outside of cedar, we won't be adding builders, we will be adding schedulers. Those schedulers just won't have any builders attached to them.

something like:
   <buildbot.schedulers.basic.Scheduler> {'builderNames': [],
   'change_filter': <ChangeFilter on branch == fx-team-emulator-kk-opt-unittest>, etc ... }
Attachment #8439563 - Flags: review?(jlund) → review+
Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551
Attachment #8439563 - Flags: checked-in+
buildbot-config patch live in production :)
For some reason, the emulator-kk test is stuck in a pending state.  I must have missed something somewhere!

The emulator-jb job is running but unhappy; the emulator starts up and then dies a horrible death shortly thereafter.  I'm guessing the VM is underpowered, but it could also be a lib problem.  I'll probably end up needing to check out a slave to investigate.
I *think* we need to patch production_masters.py in a similar fashion as emulator-jb. Taking a look at this file, it seems emulator-kk might be out of our master's scope: http://mxr.mozilla.org/build/source/tools/buildfarm/maintenance/production-masters.json#331
Attachment #8441726 - Flags: review?(jlund)
Depends on: 1026802
(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Created attachment 8441726 [details] [diff] [review]
> Add emulator-kk to build masters,

So I think this is right.

However since we don't appear to have emulator-kk tests anywhere yet, this is un-tested, and I have not written any patches for this file before, I am going to poll the experts.

catlee -

1) How do I verify we are ready for emu-kk tests? I see we have builds[1]
2) is this how we should go about adding? We have already enabled it on Cedar in buildbot-configs[2] but it looks like we have not added it to the limit platforms list like we did for emulator-jb on our test masters.
3) what about testing this patch? test-masters.sh should test our masters against this file. Is that enough for checking against spurious results?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=979450
[2] https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551
Flags: needinfo?(catlee)
As long as these tests are only enabled on cedar, that's fine.

As for testing the patch, yeah, look at test-masters output. Perhaps use http://hg.mozilla.org/build/braindump/diff/055f2b128436/buildbot-related/dump_allthethings.sh to see what builders are added/removed across all our master types.
Flags: needinfo?(catlee)
Comment on attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,

Review of attachment 8441726 [details] [diff] [review]:
-----------------------------------------------------------------

Wow, dump_allthethings is pretty neat. ty catlee.

IIUC this looks right, this will only create the builder on cedar and this patch will result in a master picking it up and running it.
Attachment #8441726 - Flags: review?(jlund) → review+
huh, something is still not right here. I had to back it out: http://hg.mozilla.org/build/tools/rev/8da583a956b2

as it blew up when I put it in production. Not sure. Will have to test again. I probably won't be able to look at it until tomorrow unless my buildduty queue clears up a bit.
Attachment #8441726 - Flags: checked-in+ → checked-in-
I had some time today to play with this again.

I don't know what went wrong with the reconfig to be honest. I ran this again off test-masters.sh and everything passed: https://pastebin.mozilla.org/5510612

This might be a situation where production-masters.json can break a reconfig and a rolling restart is required on our linux test machines. Kmoir/callek were forced into this situation last week from a similar change (reconfig failed but a stop/start worked).

I am going to ask coop as he is on buildduty this week. Coop -- jgriffin would like to land this but unless you can think of an alternative, a graceful restart will probably be the best route. I suppose at an off peak time would be best for pending count?
Flags: needinfo?(coop)
Thanks, I was just looking at this again myself and not seeing anything obvious.
jgriffin: I'm fine with you re-landing this. I think there were unresolved issues from an earlier reconfig last week that prevented this from sticking the first time.
Flags: needinfo?(coop)
(In reply to Chris Cooper [:coop] from comment #20)
> jgriffin: I'm fine with you re-landing this. I think there were unresolved
> issues from an earlier reconfig last week that prevented this from sticking
> the first time.

Thanks.  Two two:  https://hg.mozilla.org/build/tools/rev/90fd49290a90
Attachment #8441726 - Flags: checked-in- → checked-in+
Comment on attachment 8450466 [details] [diff] [review]
140703_bug_1023595_emu-kk.patch

https://hg.mozilla.org/build/puppet/rev/83c23b6da439

merged and live in prod
Attachment #8450466 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.