Closed Bug 1022862 Opened 6 years ago Closed 5 years ago

Split Marionette and WebAPI tasks on B2G so we have Mn and Mnw

Categories

(Testing :: Marionette, defect)

defect
Not set

Tracking

(firefox33 wontfix, firefox34 fixed, firefox35 fixed, b2g-v1.4 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 fixed, b2g-v2.2 fixed)

RESOLVED FIXED
mozilla35
Tracking Status
firefox33 --- wontfix
firefox34 --- fixed
firefox35 --- fixed
b2g-v1.4 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- fixed
b2g-v2.2 --- fixed

People

(Reporter: automatedtester, Assigned: automatedtester)

References

Details

(Whiteboard: [affects=marionette][status-b2g-v1.3:fixed][status-b2g-v1.3t:fixed])

Attachments

(2 files, 6 obsolete files)

Currently we have both the marionette unit tests and the web api tests run on the same job on B2G.

Unfortunately WebAPI tests have a high intermittent count (either from b2g or gecko crashes) and are on the verge of being hidden on TBPL. See https://tbpl.mozilla.org/?tree=Try&rev=8fa5dc94a2d2 as an example of how intermittent they are.

Losing the Mn tests from TBPL has the potential to allow marionette regressions on to b2g which has a bigger impact than potentially just letting through WebAPI regressions.

we should split the suites or have it as a "chunk" in Mn
Thank you for filing this after our meeting Monday - splitting these two into separate jobs will:
* Mean any future hiding of webapi tests will not affect the visibility of marionette sanity tests
* Make it easier for sheriffs and other devs to know where to file bugs against when issues occur (as well as reduce the tainting of Marionette as a harness by flaky webapi tests; which would inevitably be followed by the boy-who-called-wolf effect, similar to what happened with Android infra issues a few years ago).
OS: Mac OS X → All
Hardware: x86 → All
Version: unspecified → Trunk
Whiteboard: [affects=marionette]
Assignee: nobody → dburns
Attachment #8448253 - Flags: feedback?(jgriffin)
Comment on attachment 8448253 [details] [diff] [review]
Part 1 - Update buildconfig to split Mn and Mnw

Review of attachment 8448253 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good so far; you'll also have to add a 'marionette' section to the emulator's 'suite_config', here:

http://hg.mozilla.org/build/buildbot-configs/file/680a941f66fe/mozilla-tests/b2g_config.py#l1095
Attachment #8448253 - Flags: feedback?(jgriffin) → feedback+
Attachment #8488065 - Flags: feedback?(jgriffin)
Attachment #8448253 - Attachment is obsolete: true
Attachment #8488073 - Flags: review?(jgriffin)
now with 100% more manifest file
Attachment #8488078 - Flags: review?(jgriffin)
Attachment #8488081 - Flags: review?(jgriffin)
Attachment #8488073 - Attachment is obsolete: true
Attachment #8488073 - Flags: review?(jgriffin)
Attachment #8488078 - Attachment is obsolete: true
Attachment #8488078 - Flags: review?(jgriffin)
Comment on attachment 8488081 [details] [diff] [review]
Create a manifest just for marionette web-api tests

Review of attachment 8488081 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm
Attachment #8488081 - Flags: review?(jgriffin) → review+
Comment on attachment 8488065 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8488065 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good so far.   A couple of other changes you'll need:

Everywhere you see 'marionette-webapi', like http://hg.mozilla.org/build/buildbot-configs/file/4092544e664c/mozilla-tests/b2g_config.py#l1124, you'll need to add a --test-manifest=webapi-tests.ini argument.

Everywhere you see 'marionette-webapi', you'll need to add a new block for 'marionette' (the one you just created) and pass it the same --cfg arg as 'marionette-webapi' currently uses, but no need to include a --test-manifest argument.
Attachment #8488065 - Flags: feedback?(jgriffin) → feedback+
Keywords: leave-open
Updated as suggested, if you think it is ready for review please defer to the correct person.
Attachment #8489916 - Flags: feedback?(jgriffin)
Attachment #8488065 - Attachment is obsolete: true
Comment on attachment 8489916 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8489916 [details] [diff] [review]:
-----------------------------------------------------------------

Almost ready for review!

::: mozilla-tests/b2g_config.py
@@ +1376,4 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",
> +                        "--manifest", "webapi-tests.ini"

Should be --test-manifest, see http://hg.mozilla.org/build/mozharness/file/d75b03fc4d56/scripts/marionette.py#l99

@@ +1883,3 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",

Need to add the --test-manifest arg here.

@@ +1916,3 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",

Need to add the --test-manifest arg here.
Attachment #8489916 - Flags: feedback?(jgriffin) → feedback+
Attachment #8489916 - Attachment is obsolete: true
Changes made as suggested, if there isnt anything else you can suggest can you recommend a reviewer?
Flags: needinfo?(jgriffin)
Attachment #8491150 - Flags: review?(jlund)
Flags: needinfo?(jgriffin)
Comment on attachment 8491150 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8491150 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

just a sanity check, IIUC the suite we are adding, 'marionette', looks to be the exact same as 'marionette-webapi' used to be prior to this patch. I can see how we add a manifest to marionette-webapi which I assume tells that suite to run webapi component only.

but how does the new suite/job we are adding run only marionette and not the webapi part? Does that need a manifest too or am I grepping things wrong?
Attachment #8491150 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #16)
> Comment on attachment 8491150 [details] [diff] [review]
> Update buildconfig to split Mn and Mnw
> 
> Review of attachment 8491150 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm :)
> 
> just a sanity check, IIUC the suite we are adding, 'marionette', looks to be
> the exact same as 'marionette-webapi' used to be prior to this patch. I can
> see how we add a manifest to marionette-webapi which I assume tells that
> suite to run webapi component only.
> 
> but how does the new suite/job we are adding run only marionette and not the
> webapi part? Does that need a manifest too or am I grepping things wrong?

Currently there is a new manifest in central for only webapi but because this change isnt being rolled out to all twigs/branches I want to get my build config change rolled out and then on cedar remove the webapi tests from the main manifest and just triple check (mainly to keep the sheriffs from locking me up and throwing away the key) that I don't break anything when i try do this on Inbound.

There shouldnt be any weird dependencies but want to check that before the major rollout.
what's the process for getting this landed and rolled out? Do I need to set any flags?
Flags: needinfo?(jlund)
(In reply to David Burns :automatedtester from comment #18)
> what's the process for getting this landed and rolled out? Do I need to set
> any flags?

land on default at hg.m.o/build/buildbot-configs whenever (it's a no-op)

that will run through some tests and show up in #releng whether it passed or failed. releng will merge it to production and roll it out across our integration system ~3 times a week. this bug will be updated whenever that happens.
Flags: needinfo?(jlund)
In prod with reconfig on 2014-09-22 08:20 PT
This broke Mnw on Aurora (and probably the other B2G release branches as well):
https://tbpl.mozilla.org/php/getParsedLog.php?id=48588232&tree=Mozilla-Aurora

Also, is it expected that Mn appears to be running the full webapi suite as well? Based on the test counts, it appears that way.
Mnw before: marionette: 533/0/42
Mn after:   marionette: 535/0/42
Mnw after:  marionette: 203/0/9
I've reverted the buildbot-configs patch.
http://hg.mozilla.org/build/buildbot-configs/rev/08c6e52d150c

I wasn't comfortable backporting the in-tree manifest patch to the various release branches due to not being sure how comparable the list of tests is from branch to branch. I'd rather not try to sort that out while the tree is actively burning.

But to be clear, we will need in-tree manifests backported to Aurora, b2g32, b2g30, and b2g28_v1_3t before this goes live in production again. They can land a=test-only.
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #22)
> Also, is it expected that Mn appears to be running the full webapi suite as
> well? Based on the test counts, it appears that way.
> Mnw before: marionette: 533/0/42
> Mn after:   marionette: 535/0/42
> Mnw after:  marionette: 203/0/9

OK, I guess this part was expected based on the earlier comments in the bug.
Please land the manifest in Aurora, b2g32, b2g30, and b2g28_v1_3t (see comment 23)
Keywords: checkin-needed
Attachment #8488081 - Attachment is obsolete: true
Comment on attachment 8500039 [details] [diff] [review]
remove WebApi tests from Marionette unit tests manifests:

Review of attachment 8500039 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm
Attachment #8500039 - Flags: review?(jgriffin) → review+
https://hg.mozilla.org/mozilla-central/rev/349cd84710f2
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/3a208c39bcb9
https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/3a208c39bcb9

Don't ask.
Whiteboard: [affects=marionette] → [affects=marionette][status-b2g-v1.3:fixed][status-b2g-v1.3t:fixed]
You need to log in before you can comment on or make changes to this bug.