Schedule Mnw on cedar on emulator-jb and emulator-kk

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Assignee)

Description

3 years ago
We'd like to try running emulator tests on emulator-jb and/or emulator-kk so that we can, among other reasons, see if those emulators experience fewer timeouts than emulator-ics:  bug 906716.

We'll start by scheduling one test suite on each of these emulators on cedar, so we can see how well they perform; I've chosen Mnw since they are a little more susceptible to timeouts than the others.  There's some concern that the newer emulators may need more RAM or CPU power, in which case the current AWS VM's that we're using may not produce good results.
(Assignee)

Comment 1

3 years ago
Created attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Hopefully I've added the emulator-kk platform correctly here.
Attachment #8438026 - Flags: review?(jlund)

Comment 2

3 years ago
Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

so I think for cedar to have emulator-kk in platforms, we need to add that key to BRANCH_UNITTEST_VARS -> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/b2g_config.py#154

But I'm not sure, still parsing what this changes.

::: mozilla-tests/b2g_config.py
@@ +1519,4 @@
>          'builds_before_reboot': 1,
>          'unittest-env': {'DISPLAY': ':0'},
>          'enable_opt_unittests': True,
> +        'enable_debug_unittests': True,

I'm confused what flipping this gets us?

It looks like on the cedar lines at the bottom we are adding opt_unittests.

Comment 3

3 years ago
hmm, so adding it to BRANCH_UNITTEST_VARS I think will at least not make buildbot barf but it will also add builders outside of Cedar.

So I think something is still not right here
(Assignee)

Comment 4

3 years ago
(In reply to Jordan Lund (:jlund) from comment #2)
> Comment on attachment 8438026 [details] [diff] [review]
> Schedule Mnw on cedar on emulator-jb and -kk,
> 
> Review of attachment 8438026 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> so I think for cedar to have emulator-kk in platforms, we need to add that
> key to BRANCH_UNITTEST_VARS ->
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/
> b2g_config.py#154
> 
> But I'm not sure, still parsing what this changes.
> 
> ::: mozilla-tests/b2g_config.py
> @@ +1519,4 @@
> >          'builds_before_reboot': 1,
> >          'unittest-env': {'DISPLAY': ':0'},
> >          'enable_opt_unittests': True,
> > +        'enable_debug_unittests': True,
> 
> I'm confused what flipping this gets us?
> 
> It looks like on the cedar lines at the bottom we are adding opt_unittests.

You're right, since we're not specifically scheduling debug tests, the change on this line isn't needed yet.

Comment 5

3 years ago
Comment on attachment 8438026 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Review of attachment 8438026 [details] [diff] [review]:
-----------------------------------------------------------------

on irc, you said you would have a look at this again and the last comment on this bug was about not needing the debug boolean flipped.

r- until next patch just so I can keep my bugzilla request queue organized.
Attachment #8438026 - Flags: review?(jlund) → review-
(Assignee)

Comment 6

3 years ago
Created attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

Removed the unnecesary change to enable_debug_unittests, and added emulator-kk to BRANCH_UNITTEST_VARS.  This may add no-op builders to the other branches, but theoretically shouldn't cause anything to actually be scheduled, due to the opt_unittest_suites: [].  If that doesn't seem to be the case, we could add some logic here:

http://hg.mozilla.org/build/buildbot-configs/file/e3dd7657da2b/mozilla-tests/b2g_config.py#l1476

to skip adding anything for any branch but cedar, but since we haven't done that for emulator-jb, I don't know if it's needed.
Attachment #8439563 - Flags: review?(jlund)
(Assignee)

Updated

3 years ago
Attachment #8438026 - Attachment is obsolete: true

Comment 7

3 years ago
Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

ah, yes, it should be a no-op for other branches.

But, IIUC now, I think we may have it backwards: outside of cedar, we won't be adding builders, we will be adding schedulers. Those schedulers just won't have any builders attached to them.

something like:
   <buildbot.schedulers.basic.Scheduler> {'builderNames': [],
   'change_filter': <ChangeFilter on branch == fx-team-emulator-kk-opt-unittest>, etc ... }
Attachment #8439563 - Flags: review?(jlund) → review+
(Assignee)

Comment 8

3 years ago
Comment on attachment 8439563 [details] [diff] [review]
Schedule Mnw on cedar on emulator-jb and -kk,

https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551
Attachment #8439563 - Flags: checked-in+

Comment 9

3 years ago
buildbot-config patch live in production :)
(Assignee)

Comment 10

3 years ago
For some reason, the emulator-kk test is stuck in a pending state.  I must have missed something somewhere!

The emulator-jb job is running but unhappy; the emulator starts up and then dies a horrible death shortly thereafter.  I'm guessing the VM is underpowered, but it could also be a lib problem.  I'll probably end up needing to check out a slave to investigate.
I *think* we need to patch production_masters.py in a similar fashion as emulator-jb. Taking a look at this file, it seems emulator-kk might be out of our master's scope: http://mxr.mozilla.org/build/source/tools/buildfarm/maintenance/production-masters.json#331
(Assignee)

Comment 12

3 years ago
Created attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,
Attachment #8441726 - Flags: review?(jlund)
(Assignee)

Updated

3 years ago
Depends on: 1026802
Blocks: 1023210
(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Created attachment 8441726 [details] [diff] [review]
> Add emulator-kk to build masters,

So I think this is right.

However since we don't appear to have emulator-kk tests anywhere yet, this is un-tested, and I have not written any patches for this file before, I am going to poll the experts.

catlee -

1) How do I verify we are ready for emu-kk tests? I see we have builds[1]
2) is this how we should go about adding? We have already enabled it on Cedar in buildbot-configs[2] but it looks like we have not added it to the limit platforms list like we did for emulator-jb on our test masters.
3) what about testing this patch? test-masters.sh should test our masters against this file. Is that enough for checking against spurious results?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=979450
[2] https://hg.mozilla.org/build/buildbot-configs/rev/f329d8484551
Flags: needinfo?(catlee)
As long as these tests are only enabled on cedar, that's fine.

As for testing the patch, yeah, look at test-masters output. Perhaps use http://hg.mozilla.org/build/braindump/diff/055f2b128436/buildbot-related/dump_allthethings.sh to see what builders are added/removed across all our master types.
Flags: needinfo?(catlee)
Comment on attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,

Review of attachment 8441726 [details] [diff] [review]:
-----------------------------------------------------------------

Wow, dump_allthethings is pretty neat. ty catlee.

IIUC this looks right, this will only create the builder on cedar and this patch will result in a master picking it up and running it.
Attachment #8441726 - Flags: review?(jlund) → review+
(Assignee)

Comment 16

3 years ago
Comment on attachment 8441726 [details] [diff] [review]
Add emulator-kk to build masters,

https://hg.mozilla.org/build/tools/rev/5920f59b243e
Attachment #8441726 - Flags: checked-in+
huh, something is still not right here. I had to back it out: http://hg.mozilla.org/build/tools/rev/8da583a956b2

as it blew up when I put it in production. Not sure. Will have to test again. I probably won't be able to look at it until tomorrow unless my buildduty queue clears up a bit.

Updated

3 years ago
Attachment #8441726 - Flags: checked-in+ → checked-in-
I had some time today to play with this again.

I don't know what went wrong with the reconfig to be honest. I ran this again off test-masters.sh and everything passed: https://pastebin.mozilla.org/5510612

This might be a situation where production-masters.json can break a reconfig and a rolling restart is required on our linux test machines. Kmoir/callek were forced into this situation last week from a similar change (reconfig failed but a stop/start worked).

I am going to ask coop as he is on buildduty this week. Coop -- jgriffin would like to land this but unless you can think of an alternative, a graceful restart will probably be the best route. I suppose at an off peak time would be best for pending count?
Flags: needinfo?(coop)
(Assignee)

Comment 19

3 years ago
Thanks, I was just looking at this again myself and not seeing anything obvious.
jgriffin: I'm fine with you re-landing this. I think there were unresolved issues from an earlier reconfig last week that prevented this from sticking the first time.
Flags: needinfo?(coop)
(Assignee)

Comment 21

3 years ago
(In reply to Chris Cooper [:coop] from comment #20)
> jgriffin: I'm fine with you re-landing this. I think there were unresolved
> issues from an earlier reconfig last week that prevented this from sticking
> the first time.

Thanks.  Two two:  https://hg.mozilla.org/build/tools/rev/90fd49290a90
(Assignee)

Updated

3 years ago
Attachment #8441726 - Flags: checked-in- → checked-in+
Created attachment 8450466 [details] [diff] [review]
140703_bug_1023595_emu-kk.patch

r=coop
Attachment #8450466 - Flags: review+
Comment on attachment 8450466 [details] [diff] [review]
140703_bug_1023595_emu-kk.patch

https://hg.mozilla.org/build/puppet/rev/83c23b6da439

merged and live in prod
Attachment #8450466 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.