Closed
Bug 800025
Opened 13 years ago
Closed 13 years ago
Need buildbot-config for enabling emulator WebAPI tests on cedar
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jgriffin, Assigned: mozilla)
References
Details
Attachments
(7 files, 8 obsolete files)
3.78 KB,
patch
|
armenzg
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
31.20 KB,
patch
|
Details | Diff | Splinter Review | |
53.21 KB,
patch
|
bhearsum
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
4.15 KB,
patch
|
bhearsum
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
17.78 KB,
patch
|
bhearsum
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
1.18 KB,
patch
|
bhearsum
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
3.20 MB,
text/html
|
Details |
We're getting close to being able to turn on WebAPI (Marionette) tests on the B2G emulator on cedar. We need a buildbot-config patch to enable this.
Updated•13 years ago
|
Assignee: nobody → armenzg
Comment 1•13 years ago
|
||
jgriffin, if I wanted to run these, how would I do it?
Which mozharness scripts to call and with which parameters.
Reporter | ||
Comment 2•13 years ago
|
||
From buildbot you would call the script mozharness/scripts/marionette.py with the parameter --cfg /path/to/mozharness/configs/marionette/prod_emulator_config.py.
See http://hg.mozilla.org/build/buildbot-configs/file/5cbde6340581/mozilla-tests/config.py#l1071 for how it's done for desktop Firefox. It's the same for B2G, but just pointing to a different config file.
Updated•13 years ago
|
Assignee: armenzg → aki
Assignee | ||
Comment 3•13 years ago
|
||
This creates a Rev3 Fedora 12 cedar opt test marionette-webapi test builder (modified Armen's patch).
I'm not sure it'll respond to the right sendchange though; I'll do some testing.
Assignee | ||
Comment 4•13 years ago
|
||
These were going orange because buildbot wasn't in PATH, not because the test scheduler didn't have the appropriate builder/branch.
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko%20nightly/builds/1
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko%20build/builds/1
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_ics_armv7a_gecko-debug%20build/builds/0
Attachment #671728 -
Flags: review?(armenzg)
Updated•13 years ago
|
Attachment #671728 -
Flags: review?(armenzg) → review+
Assignee | ||
Comment 5•13 years ago
|
||
Comment on attachment 671728 [details] [diff] [review]
fix b2g sendchanges
http://hg.mozilla.org/build/buildbot-configs/rev/3b777865b9ae
Attachment #671728 -
Flags: checked-in+
Assignee | ||
Comment 6•13 years ago
|
||
We're sendchanging for the mozilla-central-ics_armv7a_gecko-opt-unittest branch, which means I should add a 'ics_armv7a_gecko' test platform with a completely different set of test suites etc.
Shouldn't be too hard to do; the tricky part is keeping it from being too ugly.
Assignee | ||
Comment 7•13 years ago
|
||
This mostly works.
However, when I actually run it in staging to see if we get the builders, I get this:
ValueError: Schedulers must have unique names, but 'tests-cedar-fedora-opt-unittest' was a duplicate
Bitten by -j production-masters.json again.
Getting new errors for ./test-masters.sh -j /path/to/production-masters.json now.
I think the answer is keep the platform 'linux' with a new slave_platform 'ics_armv7a_gecko'. Should avoid a lot of the back-bending I had to do to support a new platform of 'ics_armv7a_gecko' in the first place.
Attachment #671656 -
Attachment is obsolete: true
Assignee | ||
Comment 8•13 years ago
|
||
I have a running webapi test run in staging.
It's broken, so attaching the log for debugging purposes.
However, I don't think this will block us from rolling out, since it'll be red on cedar only.
Reporter | ||
Comment 9•13 years ago
|
||
I've landed fixes for all the apparent issues:
https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea
We'll see what next run brings. But no, these kinds of errors should not block rollout to cedar.
Assignee | ||
Comment 10•13 years ago
|
||
Ben: since you expressed interest.
Let me know if I should switch the r? to someone else; I know you have a release to take care of.
I don't know how this grew to a 60k patch, but that just seems to happen when I'm writing these things =\
* I couldn't use the slave platform 'fedora' because then the scheduler names conflicted with desktop. I couldn't use the platform 'linux' because the sendchange specifies ics_armv7a_gecko. Thunderbird can get away with the same slave_platform names because the branches don't overlap. So I created an ics_armv7a_gecko with an ics_armv7a_gecko slave platform.
* Created a b2g_config.py.
* Avoided the hack I pastebin'ed by adding VALID_PLATFORMS. I suppose we could also split the production-masters.json limit_platforms into limit_platforms, limit_thunderbird_platforms, and limit_b2g_platforms too.
* We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that. I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a separate bug. (Not really sure how ffxbld + ffxbld_dsa have to do with mozilla-tests, anyway.)
Attachment #672030 -
Attachment is obsolete: true
Attachment #672117 -
Flags: review?(bhearsum)
Assignee | ||
Comment 11•13 years ago
|
||
Attachment #672118 -
Flags: review?(bhearsum)
Assignee | ||
Comment 12•13 years ago
|
||
Attachment #672120 -
Flags: review?(bhearsum)
Assignee | ||
Comment 13•13 years ago
|
||
We're going to need a tbpl bug too.
Assignee | ||
Comment 14•13 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #13)
> We're going to need a tbpl bug too.
Actually, I think this will show up as Mn due to the existing marionette tests.
Since we don't run non-webapi marionette for ics_armv7a_gecko, this should be fine.
Assignee | ||
Comment 15•13 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> I've landed fixes for all the apparent issues:
>
> https://hg.mozilla.org/projects/cedar/rev/09e5d52cd830
> https://hg.mozilla.org/projects/cedar/rev/484a0c8aacea
>
> We'll see what next run brings. But no, these kinds of errors should not
> block rollout to cedar.
We're now orange instead of red! Progress :)
Reporter | ||
Comment 16•13 years ago
|
||
Can you pastebin the latest logs, or is there some other way for me to see them?
Assignee | ||
Comment 17•13 years ago
|
||
Attachment #672091 -
Attachment is obsolete: true
Comment 18•13 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #10)
> * We seem to be using ffxbld in mozilla/b2g_config.py, so I stuck with that.
> I'm going to guess we want to switch to b2gbld, but I'd prefer that to be a
> separate bug. (Not really sure how ffxbld + ffxbld_dsa have to do with
> mozilla-tests, anyway.)
Yeah, I don't understand why that matters either....test slaves don't have keys. Maybe for the log uploads from the masters? In any case, switching to b2gbld is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=767515.
Reporter | ||
Comment 19•13 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #17)
> Created attachment 672426 [details]
> current errors
Thanks. These errors are due to an emulator crash that occurs during certain tests. I've disabled them (bug 790463) and merged the changes to cedar. Will this automatically generate a new test run?
Assignee | ||
Comment 20•13 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #19)
> (In reply to Aki Sasaki [:aki] from comment #17)
> > Created attachment 672426 [details]
> > current errors
>
> Thanks. These errors are due to an emulator crash that occurs during
> certain tests. I've disabled them (bug 790463) and merged the changes to
> cedar. Will this automatically generate a new test run?
Looks like yes.
The most recent run died due to timeout. I'm re-running to see if this is a one-time issue or not:
11:42:46 INFO - Running command: ['/home/cltbld/talos-slave/test/build/venv/bin/python', '-u', '/home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py', '--emulator', 'arm', '--gecko-path', '/home/cltbld/talos-slave/test/build/gecko/b2g', '--homedir', '/home/cltbld/talos-slave/test/build/emulator/b2g-distro', '--type', 'b2g', '/home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini']
11:42:46 INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/python -u /home/cltbld/talos-slave/test/build/tests/marionette/marionette/runtests.py --emulator arm --gecko-path /home/cltbld/talos-slave/test/build/gecko/b2g --homedir /home/cltbld/talos-slave/test/build/emulator/b2g-distro --type b2g /home/cltbld/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit-tests.ini
11:42:47 INFO - starting httpd
11:42:47 INFO - running webserver on http://10.12.50.222:58465/
11:42:59 INFO - MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
11:43:00 INFO - installing gecko binaries
command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1345.603043
Assignee | ||
Comment 21•13 years ago
|
||
From bce9acc85eff .
We really need to get you buildvpn, or we can keep going this way til I rewrite my patches to get r+'ed tomorrow morning (hopefully).
Attachment #672426 -
Attachment is obsolete: true
Comment 22•13 years ago
|
||
Comment on attachment 672120 [details] [diff] [review]
puppet - add BuildSlaves.py entry
We decided to use this fedora-b2g for this and production-masters.json, I believe.
Attachment #672120 -
Flags: review?(bhearsum)
Reporter | ||
Comment 23•13 years ago
|
||
Wow, that's weird. I'm guessing the emulator crashed right away, looking at the log, but it's hard to tell for sure. I'm going to push another patch to cedar to kick off another run.
Comment 24•13 years ago
|
||
Comment on attachment 672117 [details] [diff] [review]
(working) add mozilla-tests/b2g_config.py
We talked on IRC about this a lot. Most of it is fine. Two things that we agreed need to be fixed:
* Switch to limit_*_platforms instead of VALID_PLATFORMS, as comment #10 talks about.
* Change the slave platform name to something less specific to this exact build (I think we said fedora-b2g).
And if it doesn't prove too difficult, getting rid of the nasty loop that adds the tests after initial definitions of UNITTEST_SUITES in favour of adding them to UNITTEST_SUITES/PLATFORM_UNITTEST_VARS initially. Thanks for making the effort to make this better, Aki.
Attachment #672117 -
Flags: review?(bhearsum) → review-
Updated•13 years ago
|
Attachment #672118 -
Flags: review?(bhearsum)
Assignee | ||
Comment 25•13 years ago
|
||
In theory, this interdiff is here for Ben to more easily see what changed post-review.
However, this is a 31k interdiff, so, well, it's here.
Assignee | ||
Comment 26•13 years ago
|
||
Attachment #672117 -
Attachment is obsolete: true
Attachment #672589 -
Flags: review?(bhearsum)
Assignee | ||
Comment 27•13 years ago
|
||
Attachment #672591 -
Flags: review?(bhearsum)
Assignee | ||
Comment 28•13 years ago
|
||
Also cleaned up check-master-json.py to a degree.
It's still complaining about valid scheduler master names and valid test master_dirs; I added TODOs to the code. I'm still leaving it better off than when I started.
Attachment #672118 -
Attachment is obsolete: true
Attachment #672594 -
Flags: review?(bhearsum)
Assignee | ||
Comment 29•13 years ago
|
||
Attachment #672120 -
Attachment is obsolete: true
Attachment #672596 -
Flags: review?(bhearsum)
Assignee | ||
Comment 30•13 years ago
|
||
Attachment #672485 -
Attachment is obsolete: true
Reporter | ||
Comment 31•13 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #30)
> Created attachment 672603 [details]
> warnings from 4a59a1aabbb4
Thanks. These test failures are mostly being caused by the fact that we're installing a new version of gecko in the emulator; an event that the tests rely on does not get triggered in this scenario. Will have figure out a way around this.
Updated•13 years ago
|
Attachment #672594 -
Attachment is patch: true
Updated•13 years ago
|
Attachment #672596 -
Flags: review?(bhearsum) → review+
Comment 32•13 years ago
|
||
Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms
Review of attachment 672594 [details] [diff] [review]:
-----------------------------------------------------------------
::: buildfarm/maintenance/check-master-json.py
@@ -26,1 @@
> def check_master(master):
Thanks for fixing this script up.
Attachment #672594 -
Flags: review?(bhearsum) → review+
Comment 33•13 years ago
|
||
Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes
Review of attachment 672589 [details] [diff] [review]:
-----------------------------------------------------------------
::: mozilla-tests/thunderbird_config.py
@@ +86,5 @@
> + 'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> + 'suite_config': {
> + 'marionette-webapi': {
> + 'extra_args': [
> + "--cfg", "marionette/prod_emulator_config.py"
Is this platform specific? If so, it seems like it could go in UNITTEST_SUITES instead.
Also, does "prod" imply only to be used in production? If so, this may need to go in b2g_*_config.py and be imported.
@@ +89,5 @@
> + 'extra_args': [
> + "--cfg", "marionette/prod_emulator_config.py"
> + ],
> + 'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> + 'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],
I'm a little confused about this block in general. It seems like all of the platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all of the suite specific stuff can go in UNITTEST_SUITES.
It doesn't _seem_ like reboot_command or hg_bin are specific to the platform + suite combination. I don't know anything about the marionette configs, so I'll defer to you there.
Other than the few things noted above, this patch looks good - I'm really happy to see these tests getting added up here rather than way after the fact. Thank you!
I'm r+'ing this, because I don't want to artificially block if my comments aren't applicable, but if the suite_config stuff can go in PLATFORMS and UNITTEST_SUITES, I'd like to see that happen.
Attachment #672589 -
Flags: review?(bhearsum) → review+
Comment 34•13 years ago
|
||
Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff
Review of attachment 672591 [details] [diff] [review]:
-----------------------------------------------------------------
r+ modulo potential changes needed because of comments on the buildbot-configs patch.
Attachment #672591 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 35•13 years ago
|
||
(In reply to Ben Hearsum [:bhearsum] from comment #33)
> Comment on attachment 672589 [details] [diff] [review]
> (configs) with post-review fixes
>
> Review of attachment 672589 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> ::: mozilla-tests/thunderbird_config.py
> @@ +86,5 @@
> > + 'debug_unittest_suites' : UNITTEST_SUITES['debug_unittest_suites'][:],
> > + 'suite_config': {
> > + 'marionette-webapi': {
> > + 'extra_args': [
> > + "--cfg", "marionette/prod_emulator_config.py"
>
> Is this platform specific? If so, it seems like it could go in
> UNITTEST_SUITES instead.
It is platform specific. Do you mean "if not"?
We only have one platform here, but if we had, say, windows, it would have a different config file.
> Also, does "prod" imply only to be used in production? If so, this may need
> to go in b2g_*_config.py and be imported.
It does sometimes, depending on contents.
Here it works in staging and prod.
> > + 'extra_args': [
> > + "--cfg", "marionette/prod_emulator_config.py"
> > + ],
> > + 'reboot_command': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['reboot_command'],
> > + 'hg_bin': PLATFORMS['ics_armv7a_gecko']['mozharness_config']['hg_bin'],
>
> I'm a little confused about this block in general. It seems like all of the
> platform-specific stuff can go in PLATFORMS[*]['mozharness_config'], and all
> of the suite specific stuff can go in UNITTEST_SUITES.
>
> It doesn't _seem_ like reboot_command or hg_bin are specific to the platform
> + suite combination. I don't know anything about the marionette configs, so
> I'll defer to you there.
This is true, but we do need platform-specific script-specific extra_args. Maybe not for this one platform+script, but we will down the line.
Assignee | ||
Comment 36•13 years ago
|
||
Comment on attachment 672594 [details] [diff] [review]
(tools) production-masters.json limit_*_platforms
http://hg.mozilla.org/build/tools/rev/f82c23bf57af
Attachment #672594 -
Flags: checked-in+
Assignee | ||
Comment 37•13 years ago
|
||
Comment on attachment 672589 [details] [diff] [review]
(configs) with post-review fixes
Landed, without the duplicate hg_bin and reboot_command info.
http://hg.mozilla.org/build/buildbot-configs/rev/0b4fa391a015
Attachment #672589 -
Flags: checked-in+
Assignee | ||
Comment 38•13 years ago
|
||
Comment on attachment 672591 [details] [diff] [review]
(custom) deal with PLATFORM_UNITTEST_VARS stuff
Landed with hg_bin and reboot_command tweaks.
http://hg.mozilla.org/build/buildbotcustom/rev/bb4dab0bde84
Attachment #672591 -
Flags: checked-in+
Assignee | ||
Comment 39•13 years ago
|
||
Comment on attachment 672596 [details] [diff] [review]
(puppet) fedora-b2g
http://hg.mozilla.org/build/puppet-manifests/rev/45aedcf1d550
Attachment #672596 -
Flags: checked-in+
Assignee | ||
Comment 40•13 years ago
|
||
Ok, I now see this running on Cedar.
Two things:
1. We keep hanging intermittently after
15:25:41 INFO - starting httpd
15:25:41 INFO - running webserver on http://10.12.49.151:57425/
15:26:04 INFO - MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
15:26:04 INFO - installing gecko binaries
I rekicked it.
2. We're showing up under Linux opt instead of Armv7a ICS opt.
I think this is a simple name change.
Reporter | ||
Comment 41•13 years ago
|
||
Awesome! I'll be filing a bug about the first problem and debugging it.
Assignee | ||
Comment 42•13 years ago
|
||
Here they are!
Sadly, both runs red at the moment:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16246474&tree=Cedar&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=16246872&tree=Cedar&full=1
Assignee | ||
Comment 43•13 years ago
|
||
I'm going to call this done.
I'll add a new name when I enable mochitests.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Updated•12 years ago
|
Product: mozilla.org → Release Engineering
Updated•7 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•