800025 - Need buildbot-config for enabling emulator WebAPI tests on cedar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We're getting close to being able to turn on WebAPI (Marionette) tests on the B2G emulator on cedar.  We need a buildbot-config patch to enable this.

Comment 1

[Armen [:armenzg]]

jgriffin, if I wanted to run these, how would I do it?
Which mozharness scripts to call and with which parameters.

Comment 2

[Jonathan Griffin (:jgriffin)]

From buildbot you would call the script mozharness/scripts/marionette.py with the parameter --cfg /path/to/mozharness/configs/marionette/prod_emulator_config.py.

See http://hg.mozilla.org/build/buildbot-configs/file/5cbde6340581/mozilla-tests/config.py#l1071 for how it's done for desktop Firefox.  It's the same for B2G, but just pointing to a different config file.

Comment 3

[Aki Sasaki (not active)]

Attachment: add webapi cedar tests

This creates a Rev3 Fedora 12 cedar opt test marionette-webapi test builder (modified Armen's patch).

I'm not sure it'll respond to the right sendchange though; I'll do some testing.

Comment 4

[Aki Sasaki (not active)]

Attachment: fix b2g sendchanges

These were going orange because buildbot wasn't in PATH, not because the test scheduler didn't have the appropriate builder/branch.

http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko%20nightly/builds/1
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko%20build/builds/1
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko-debug%20build/builds/0

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 671728 [details] [diff] [review]
fix b2g sendchanges

http://hg.mozilla.org/build/buildbot-configs/rev/3b777865b9ae

Comment 6

[Aki Sasaki (not active)]

We're sendchanging for the mozilla-central-ics_armv7a_gecko-opt-unittest branch, which means I should add a 'ics_armv7a_gecko' test platform with a completely different set of test suites etc.

Shouldn't be too hard to do; the tricky part is keeping it from being too ugly.

Comment 7

[Aki Sasaki (not active)]

Attachment: (broken, but closer) add new mozilla-tests/b2g_config.py

This mostly works.
However, when I actually run it in staging to see if we get the builders, I get this:

ValueError: Schedulers must have unique names, but 'tests-cedar-fedora-opt-unittest' was a duplicate

Bitten by -j production-masters.json again.

Getting new errors for ./test-masters.sh -j /path/to/production-masters.json now.

I think the answer is keep the platform 'linux' with a new slave_platform 'ics_armv7a_gecko'.  Should avoid a lot of the back-bending I had to do to support a new platform of 'ics_armv7a_gecko' in the first place.

Comment 8

[Aki Sasaki (not active)]

Attachment: running

I have a running webapi test run in staging.
It's broken, so attaching the log for debugging purposes.
However, I don't think this will block us from rolling out, since it'll be red on cedar only.

Comment 9

[Jonathan Griffin (:jgriffin)]

I've landed fixes for all the apparent issues:

https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea

We'll see what next run brings.  But no, these kinds of errors should not block rollout to cedar.

Comment 10

[Aki Sasaki (not active)]

Attachment: (working) add mozilla-tests/b2g_config.py

Ben: since you expressed interest.
Let me know if I should switch the r? to someone else; I know you have a release to take care of.
I don't know how this grew to a 60k patch, but that just seems to happen when I'm writing these things =\

* I couldn't use the slave platform 'fedora' because then the scheduler names conflicted with desktop.  I couldn't use the platform 'linux' because the sendchange specifies ics_armv7a_gecko.  Thunderbird can get away with the same slave_platform names because the branches don't overlap.  So I created an ics_armv7a_gecko with an ics_armv7a_gecko slave platform.

* Created a b2g_config.py.

* Avoided the hack I pastebin'ed by adding VALID_PLATFORMS.  I suppose we could also split the production-masters.json limit_platforms into limit_platforms, limit_thunderbird_platforms, and limit_b2g_platforms too.

* We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that.  I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a separate bug.  (Not really sure how ffxbld + ffxbld_dsa have to do with mozilla-tests, anyway.)

Comment 11

[Aki Sasaki (not active)]

Attachment: tools - add ics_armv7a_gecko to linux test masters

Comment 12

[Aki Sasaki (not active)]

Attachment: puppet - add BuildSlaves.py entry

Comment 13

[Aki Sasaki (not active)]

We're going to need a tbpl bug too.

Comment 14

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #13)
> We're going to need a tbpl bug too.

Actually, I think this will show up as Mn due to the existing marionette tests.
Since we don't run non-webapi marionette for ics_armv7a_gecko, this should be fine.

Comment 15

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> I've landed fixes for all the apparent issues:
> 
> https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
> https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea
> 
> We'll see what next run brings.  But no, these kinds of errors should not
> block rollout to cedar.

We're now orange instead of red!  Progress :)

Comment 16

[Jonathan Griffin (:jgriffin)]

Can you pastebin the latest logs, or is there some other way for me to see them?

Comment 17

[Aki Sasaki (not active)]

Attachment: current errors

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #10)
> * We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that.
> I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a
> separate bug.  (Not really sure how ffxbld + ffxbld_dsa have to do with
> mozilla-tests, anyway.)

Yeah, I don't understand why that matters either....test slaves don't have keys. Maybe for the log uploads from the masters? In any case, switching to b2gbld is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=767515.

Comment 19

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #17)
> Created attachment 672426 [details]
> current errors

Thanks.  These errors are due to an emulator crash that occurs during certain tests.  I've disabled them (bug 790463) and merged the changes to cedar.  Will this automatically generate a new test run?

Comment 20

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #19)
> (In reply to Aki Sasaki [:aki] from comment #17)
> > Created attachment 672426 [details]
> > current errors
> 
> Thanks.  These errors are due to an emulator crash that occurs during
> certain tests.  I've disabled them (bug 790463) and merged the changes to
> cedar.  Will this automatically generate a new test run?

Looks like yes.
The most recent run died due to timeout.  I'm re-running to see if this is a one-time issue or not:

11:42:46     INFO - Running command: ['/home/cltbld/talos-slave/test/build/venv/bin/python', '-u', '/home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py', '--emulator', 'arm', '--gecko-path', '/home/cltbld/talos-slave/test/build/gecko/b2g', '--homedir', '/home/cltbld/talos-slave/test/build/emulator/b2g-distro', '--type', 'b2g', '/home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini']
11:42:46     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/python -u /home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py --emulator arm --gecko-path /home/cltbld/talos-slave/test/build/gecko/b2g --homedir /home/cltbld/talos-slave/test/build/emulator/b2g-distro --type b2g /home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini
11:42:47     INFO -  starting httpd
11:42:47     INFO -  running webserver on http://10.12.50.222:58465/
11:42:59     INFO -  MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
11:43:00     INFO -  installing gecko binaries

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1345.603043

Comment 21

[Aki Sasaki (not active)]

Attachment: current errors

From bce9acc85eff .

We really need to get you buildvpn, or we can keep going this way til I rewrite my patches to get r+'ed tomorrow morning (hopefully).

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 672120 [details] [diff] [review]
puppet - add BuildSlaves.py entry

We decided to use this fedora-b2g for this and production-masters.json, I believe.

Comment 23

[Jonathan Griffin (:jgriffin)]

Wow, that's weird.  I'm guessing the emulator crashed right away, looking at the log, but it's hard to tell for sure.  I'm going to push another patch to cedar to kick off another run.

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 672117 [details] [diff] [review]
(working) add mozilla-tests/b2g_config.py

We talked on IRC about this a lot. Most of it is fine. Two things that we agreed need to be fixed:
* Switch to limit_*_platforms instead of VALID_PLATFORMS, as comment #10 talks about.
* Change the slave platform name to something less specific to this exact build (I think we said fedora-b2g).

And if it doesn't prove too difficult, getting rid of the nasty loop that adds the tests after initial definitions of UNITTEST_SUITES in favour of adding them to UNITTEST_SUITES/PLATFORM_UNITTEST_VARS initially. Thanks for making the effort to make this better, Aki.

Comment 25

[Aki Sasaki (not active)]

Attachment: interdiff

In theory, this interdiff is here for Ben to more easily see what changed post-review.
However, this is a 31k interdiff, so, well, it's here.

Comment 26

[Aki Sasaki (not active)]

Attachment: (configs) with post-review fixes

Comment 27

[Aki Sasaki (not active)]

Attachment: (custom) deal with PLATFORM_UNITTEST_VARS stuff

Comment 28

[Aki Sasaki (not active)]

Attachment: (tools) production-masters.json limit_*_platforms

Also cleaned up check-master-json.py to a degree.
It's still complaining about valid scheduler master names and valid test master_dirs; I added TODOs to the code.  I'm still leaving it better off than when I started.

Comment 29

[Aki Sasaki (not active)]

Attachment: (puppet) fedora-b2g

Comment 30

[Aki Sasaki (not active)]

Attachment: warnings from 4a59a1aabbb4

Comment 31

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #30)
> Created attachment 672603 [details]
> warnings from 4a59a1aabbb4

Thanks.  These test failures are mostly being caused by the fact that we're installing a new version of gecko in the emulator; an event that the tests rely on does not get triggered in this scenario.  Will have figure out a way around this.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms

Review of attachment 672594 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/check-master-json.py
@@ -26,1 @@
>  def check_master(master):

Thanks for fixing this script up.

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes

Review of attachment 672589 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/thunderbird_config.py
@@ +86,5 @@
> +            'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> +            'suite_config': {
> +                'marionette-webapi': {
> +                    'extra_args': [
> +                        "--cfg", "marionette/prod_emulator_config.py"

Is this platform specific? If so, it seems like it could go in UNITTEST_SUITES instead.

Also, does "prod" imply only to be used in production? If so, this may need to go in b2g_*_config.py and be imported.

@@ +89,5 @@
> +                    'extra_args': [
> +                        "--cfg", "marionette/prod_emulator_config.py"
> +                    ],
> +                    'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> +                    'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],

I'm a little confused about this block in general. It seems like all of the platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all of the suite specific stuff can go in UNITTEST_SUITES.

It doesn't _seem_ like reboot_command or hg_bin are specific to the platform + suite combination. I don't know anything about the marionette configs, so I'll defer to you there.

Other than the few things noted above, this patch looks good - I'm really happy to see these tests getting added up here rather than way after the fact. Thank you!

I'm r+'ing this, because I don't want to artificially block if my comments aren't applicable, but if the suite_config stuff can go in PLATFORMS and UNITTEST_SUITES, I'd like to see that happen.

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff

Review of attachment 672591 [details] [diff] [review]:
-----------------------------------------------------------------

r+ modulo potential changes needed because of comments on the buildbot-configs patch.

Comment 35

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #33)
> Comment on attachment 672589 [details] [diff] [review]
> (configs) with post-review fixes
> 
> Review of attachment 672589 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/thunderbird_config.py
> @@ +86,5 @@
> > +            'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> > +            'suite_config': {
> > +                'marionette-webapi': {
> > +                    'extra_args': [
> > +                        "--cfg", "marionette/prod_emulator_config.py"
> 
> Is this platform specific? If so, it seems like it could go in
> UNITTEST_SUITES instead.

It is platform specific.  Do you mean "if not"?
We only have one platform here, but if we had, say, windows, it would have a different config file.

> Also, does "prod" imply only to be used in production? If so, this may need
> to go in b2g_*_config.py and be imported.

It does sometimes, depending on contents.
Here it works in staging and prod.

> > +                    'extra_args': [
> > +                        "--cfg", "marionette/prod_emulator_config.py"
> > +                    ],
> > +                    'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> > +                    'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],
> 
> I'm a little confused about this block in general. It seems like all of the
> platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all
> of the suite specific stuff can go in UNITTEST_SUITES.
> 
> It doesn't _seem_ like reboot_command or hg_bin are specific to the platform
> + suite combination. I don't know anything about the marionette configs, so
> I'll defer to you there.

This is true, but we do need platform-specific script-specific extra_args.  Maybe not for this one platform+script, but we will down the line.

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms

http://hg.mozilla.org/build/tools/rev/f82c23bf57af

Comment 37

[Aki Sasaki (not active)]

Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes

Landed, without the duplicate hg_bin and reboot_command info.
http://hg.mozilla.org/build/buildbot-configs/rev/0b4fa391a015

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff

Landed with hg_bin and reboot_command tweaks.
http://hg.mozilla.org/build/buildbotcustom/rev/bb4dab0bde84

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 672596 [details] [diff] [review]
(puppet) fedora-b2g

http://hg.mozilla.org/build/puppet-manifests/rev/45aedcf1d550

Comment 40

[Aki Sasaki (not active)]

Ok, I now see this running on Cedar.

Two things:

1. We keep hanging intermittently after

15:25:41     INFO -  starting httpd
15:25:41     INFO -  running webserver on http://10.12.49.151:57425/
15:26:04     INFO -  MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
15:26:04     INFO -  installing gecko binaries

I rekicked it.

2. We're showing up under Linux opt instead of Armv7a ICS opt.

I think this is a simple name change.

Comment 41

[Jonathan Griffin (:jgriffin)]

Awesome!  I'll be filing a bug about the first problem and debugging it.

Comment 42

[Aki Sasaki (not active)]

Here they are!
Sadly, both runs red at the moment:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16246474&tree=Cedar&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=16246872&tree=Cedar&full=1

Comment 43

[Aki Sasaki (not active)]

I'm going to call this done.
I'll add a new name when I enable mochitests.