Need buildbot-config for enabling emulator WebAPI tests on cedar

RESOLVED FIXED

Status

defect
RESOLVED FIXED
7 years ago
Last year

People

(Reporter: jgriffin, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(7 attachments, 8 obsolete attachments)

3.78 KB, patch
armenzg
: review+
Details | Diff | Splinter Review
31.20 KB, patch
Details | Diff | Splinter Review
53.21 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
4.15 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
17.78 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.18 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
3.20 MB, text/html
Details
Reporter

Description

7 years ago
We're getting close to being able to turn on WebAPI (Marionette) tests on the B2G emulator on cedar.  We need a buildbot-config patch to enable this.
Reporter

Updated

7 years ago
Blocks: 789652

Updated

7 years ago
Assignee: nobody → armenzg

Comment 1

7 years ago
jgriffin, if I wanted to run these, how would I do it?
Which mozharness scripts to call and with which parameters.
Reporter

Comment 2

7 years ago
From buildbot you would call the script mozharness/scripts/marionette.py with the parameter --cfg /path/to/mozharness/configs/marionette/prod_emulator_config.py.

See http://hg.mozilla.org/build/buildbot-configs/file/5cbde6340581/mozilla-tests/config.py#l1071 for how it's done for desktop Firefox.  It's the same for B2G, but just pointing to a different config file.

Updated

7 years ago
Assignee: armenzg → aki
Assignee

Comment 3

7 years ago
Posted patch add webapi cedar tests (obsolete) — Splinter Review
This creates a Rev3 Fedora 12 cedar opt test marionette-webapi test builder (modified Armen's patch).

I'm not sure it'll respond to the right sendchange though; I'll do some testing.

Updated

7 years ago
Attachment #671728 - Flags: review?(armenzg) → review+
Assignee

Comment 6

7 years ago
We're sendchanging for the mozilla-central-ics_armv7a_gecko-opt-unittest branch, which means I should add a 'ics_armv7a_gecko' test platform with a completely different set of test suites etc.

Shouldn't be too hard to do; the tricky part is keeping it from being too ugly.
Assignee

Comment 7

7 years ago
This mostly works.
However, when I actually run it in staging to see if we get the builders, I get this:

ValueError: Schedulers must have unique names, but 'tests-cedar-fedora-opt-unittest' was a duplicate

Bitten by -j production-masters.json again.

Getting new errors for ./test-masters.sh -j /path/to/production-masters.json now.

I think the answer is keep the platform 'linux' with a new slave_platform 'ics_armv7a_gecko'.  Should avoid a lot of the back-bending I had to do to support a new platform of 'ics_armv7a_gecko' in the first place.
Attachment #671656 - Attachment is obsolete: true
Assignee

Comment 8

7 years ago
Posted file running (obsolete) —
I have a running webapi test run in staging.
It's broken, so attaching the log for debugging purposes.
However, I don't think this will block us from rolling out, since it'll be red on cedar only.
Reporter

Comment 9

7 years ago
I've landed fixes for all the apparent issues:

https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea

We'll see what next run brings.  But no, these kinds of errors should not block rollout to cedar.
Assignee

Comment 10

7 years ago
Ben: since you expressed interest.
Let me know if I should switch the r? to someone else; I know you have a release to take care of.
I don't know how this grew to a 60k patch, but that just seems to happen when I'm writing these things =\

* I couldn't use the slave platform 'fedora' because then the scheduler names conflicted with desktop.  I couldn't use the platform 'linux' because the sendchange specifies ics_armv7a_gecko.  Thunderbird can get away with the same slave_platform names because the branches don't overlap.  So I created an ics_armv7a_gecko with an ics_armv7a_gecko slave platform.

* Created a b2g_config.py.

* Avoided the hack I pastebin'ed by adding VALID_PLATFORMS.  I suppose we could also split the production-masters.json limit_platforms into limit_platforms, limit_thunderbird_platforms, and limit_b2g_platforms too.

* We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that.  I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a separate bug.  (Not really sure how ffxbld + ffxbld_dsa have to do with mozilla-tests, anyway.)
Attachment #672030 - Attachment is obsolete: true
Attachment #672117 - Flags: review?(bhearsum)
Assignee

Comment 11

7 years ago
Attachment #672118 - Flags: review?(bhearsum)
Assignee

Comment 12

7 years ago
Attachment #672120 - Flags: review?(bhearsum)
Assignee

Comment 13

7 years ago
We're going to need a tbpl bug too.
Assignee

Comment 14

7 years ago
(In reply to Aki Sasaki [:aki] from comment #13)
> We're going to need a tbpl bug too.

Actually, I think this will show up as Mn due to the existing marionette tests.
Since we don't run non-webapi marionette for ics_armv7a_gecko, this should be fine.
Assignee

Comment 15

7 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> I've landed fixes for all the apparent issues:
> 
> https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
> https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea
> 
> We'll see what next run brings.  But no, these kinds of errors should not
> block rollout to cedar.

We're now orange instead of red!  Progress :)
Reporter

Comment 16

7 years ago
Can you pastebin the latest logs, or is there some other way for me to see them?
Assignee

Comment 17

7 years ago
Posted file current errors (obsolete) —
Attachment #672091 - Attachment is obsolete: true
(In reply to Aki Sasaki [:aki] from comment #10)
> * We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that.
> I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a
> separate bug.  (Not really sure how ffxbld + ffxbld_dsa have to do with
> mozilla-tests, anyway.)

Yeah, I don't understand why that matters either....test slaves don't have keys. Maybe for the log uploads from the masters? In any case, switching to b2gbld is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=767515.
Reporter

Comment 19

7 years ago
(In reply to Aki Sasaki [:aki] from comment #17)
> Created attachment 672426 [details]
> current errors

Thanks.  These errors are due to an emulator crash that occurs during certain tests.  I've disabled them (bug 790463) and merged the changes to cedar.  Will this automatically generate a new test run?
Assignee

Comment 20

7 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #19)
> (In reply to Aki Sasaki [:aki] from comment #17)
> > Created attachment 672426 [details]
> > current errors
> 
> Thanks.  These errors are due to an emulator crash that occurs during
> certain tests.  I've disabled them (bug 790463) and merged the changes to
> cedar.  Will this automatically generate a new test run?

Looks like yes.
The most recent run died due to timeout.  I'm re-running to see if this is a one-time issue or not:

11:42:46     INFO - Running command: ['/home/cltbld/talos-slave/test/build/venv/bin/python', '-u', '/home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py', '--emulator', 'arm', '--gecko-path', '/home/cltbld/talos-slave/test/build/gecko/b2g', '--homedir', '/home/cltbld/talos-slave/test/build/emulator/b2g-distro', '--type', 'b2g', '/home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini']
11:42:46     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/python -u /home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py --emulator arm --gecko-path /home/cltbld/talos-slave/test/build/gecko/b2g --homedir /home/cltbld/talos-slave/test/build/emulator/b2g-distro --type b2g /home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini
11:42:47     INFO -  starting httpd
11:42:47     INFO -  running webserver on http://10.12.50.222:58465/
11:42:59     INFO -  MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
11:43:00     INFO -  installing gecko binaries

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1345.603043
Assignee

Comment 21

7 years ago
Posted file current errors (obsolete) —
From bce9acc85eff .

We really need to get you buildvpn, or we can keep going this way til I rewrite my patches to get r+'ed tomorrow morning (hopefully).
Attachment #672426 - Attachment is obsolete: true
Comment on attachment 672120 [details] [diff] [review]
puppet - add BuildSlaves.py entry

We decided to use this fedora-b2g for this and production-masters.json, I believe.
Attachment #672120 - Flags: review?(bhearsum)
Reporter

Comment 23

7 years ago
Wow, that's weird.  I'm guessing the emulator crashed right away, looking at the log, but it's hard to tell for sure.  I'm going to push another patch to cedar to kick off another run.
Comment on attachment 672117 [details] [diff] [review]
(working) add mozilla-tests/b2g_config.py

We talked on IRC about this a lot. Most of it is fine. Two things that we agreed need to be fixed:
* Switch to limit_*_platforms instead of VALID_PLATFORMS, as comment #10 talks about.
* Change the slave platform name to something less specific to this exact build (I think we said fedora-b2g).

And if it doesn't prove too difficult, getting rid of the nasty loop that adds the tests after initial definitions of UNITTEST_SUITES in favour of adding them to UNITTEST_SUITES/PLATFORM_UNITTEST_VARS initially. Thanks for making the effort to make this better, Aki.
Attachment #672117 - Flags: review?(bhearsum) → review-
Attachment #672118 - Flags: review?(bhearsum)
Assignee

Comment 25

7 years ago
Posted patch interdiffSplinter Review
In theory, this interdiff is here for Ben to more easily see what changed post-review.
However, this is a 31k interdiff, so, well, it's here.
Assignee

Comment 26

7 years ago
Attachment #672117 - Attachment is obsolete: true
Attachment #672589 - Flags: review?(bhearsum)
Assignee

Comment 28

7 years ago
Also cleaned up check-master-json.py to a degree.
It's still complaining about valid scheduler master names and valid test master_dirs; I added TODOs to the code.  I'm still leaving it better off than when I started.
Attachment #672118 - Attachment is obsolete: true
Attachment #672594 - Flags: review?(bhearsum)
Assignee

Comment 29

7 years ago
Attachment #672120 - Attachment is obsolete: true
Attachment #672596 - Flags: review?(bhearsum)
Assignee

Comment 30

7 years ago
Attachment #672485 - Attachment is obsolete: true
Reporter

Comment 31

7 years ago
(In reply to Aki Sasaki [:aki] from comment #30)
> Created attachment 672603 [details]
> warnings from 4a59a1aabbb4

Thanks.  These test failures are mostly being caused by the fact that we're installing a new version of gecko in the emulator; an event that the tests rely on does not get triggered in this scenario.  Will have figure out a way around this.
Attachment #672594 - Attachment is patch: true
Attachment #672596 - Flags: review?(bhearsum) → review+
Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms

Review of attachment 672594 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/check-master-json.py
@@ -26,1 @@
>  def check_master(master):

Thanks for fixing this script up.
Attachment #672594 - Flags: review?(bhearsum) → review+
Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes

Review of attachment 672589 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/thunderbird_config.py
@@ +86,5 @@
> +            'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> +            'suite_config': {
> +                'marionette-webapi': {
> +                    'extra_args': [
> +                        "--cfg", "marionette/prod_emulator_config.py"

Is this platform specific? If so, it seems like it could go in UNITTEST_SUITES instead.

Also, does "prod" imply only to be used in production? If so, this may need to go in b2g_*_config.py and be imported.

@@ +89,5 @@
> +                    'extra_args': [
> +                        "--cfg", "marionette/prod_emulator_config.py"
> +                    ],
> +                    'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> +                    'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],

I'm a little confused about this block in general. It seems like all of the platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all of the suite specific stuff can go in UNITTEST_SUITES.

It doesn't _seem_ like reboot_command or hg_bin are specific to the platform + suite combination. I don't know anything about the marionette configs, so I'll defer to you there.

Other than the few things noted above, this patch looks good - I'm really happy to see these tests getting added up here rather than way after the fact. Thank you!

I'm r+'ing this, because I don't want to artificially block if my comments aren't applicable, but if the suite_config stuff can go in PLATFORMS and UNITTEST_SUITES, I'd like to see that happen.
Attachment #672589 - Flags: review?(bhearsum) → review+
Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff

Review of attachment 672591 [details] [diff] [review]:
-----------------------------------------------------------------

r+ modulo potential changes needed because of comments on the buildbot-configs patch.
Attachment #672591 - Flags: review?(bhearsum) → review+
Assignee

Comment 35

7 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #33)
> Comment on attachment 672589 [details] [diff] [review]
> (configs) with post-review fixes
> 
> Review of attachment 672589 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/thunderbird_config.py
> @@ +86,5 @@
> > +            'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> > +            'suite_config': {
> > +                'marionette-webapi': {
> > +                    'extra_args': [
> > +                        "--cfg", "marionette/prod_emulator_config.py"
> 
> Is this platform specific? If so, it seems like it could go in
> UNITTEST_SUITES instead.

It is platform specific.  Do you mean "if not"?
We only have one platform here, but if we had, say, windows, it would have a different config file.

> Also, does "prod" imply only to be used in production? If so, this may need
> to go in b2g_*_config.py and be imported.

It does sometimes, depending on contents.
Here it works in staging and prod.

> > +                    'extra_args': [
> > +                        "--cfg", "marionette/prod_emulator_config.py"
> > +                    ],
> > +                    'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> > +                    'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],
> 
> I'm a little confused about this block in general. It seems like all of the
> platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all
> of the suite specific stuff can go in UNITTEST_SUITES.
> 
> It doesn't _seem_ like reboot_command or hg_bin are specific to the platform
> + suite combination. I don't know anything about the marionette configs, so
> I'll defer to you there.

This is true, but we do need platform-specific script-specific extra_args.  Maybe not for this one platform+script, but we will down the line.
Assignee

Comment 36

7 years ago
Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms

http://hg.mozilla.org/build/tools/rev/f82c23bf57af
Attachment #672594 - Flags: checked-in+
Assignee

Comment 37

7 years ago
Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes

Landed, without the duplicate hg_bin and reboot_command info.
http://hg.mozilla.org/build/buildbot-configs/rev/0b4fa391a015
Attachment #672589 - Flags: checked-in+
Assignee

Comment 38

7 years ago
Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff

Landed with hg_bin and reboot_command tweaks.
http://hg.mozilla.org/build/buildbotcustom/rev/bb4dab0bde84
Attachment #672591 - Flags: checked-in+
Assignee

Comment 40

7 years ago
Ok, I now see this running on Cedar.

Two things:

1. We keep hanging intermittently after

15:25:41     INFO -  starting httpd
15:25:41     INFO -  running webserver on http://10.12.49.151:57425/
15:26:04     INFO -  MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
15:26:04     INFO -  installing gecko binaries

I rekicked it.

2. We're showing up under Linux opt instead of Armv7a ICS opt.

I think this is a simple name change.
Reporter

Comment 41

7 years ago
Awesome!  I'll be filing a bug about the first problem and debugging it.
Assignee

Comment 43

7 years ago
I'm going to call this done.
I'll add a new name when I enable mochitests.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Reporter

Updated

7 years ago
Depends on: 807050
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.