1022862 - Split Marionette and WebAPI tasks on B2G so we have Mn and Mnw

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[David Burns :automatedtester]

Currently we have both the marionette unit tests and the web api tests run on the same job on B2G.

Unfortunately WebAPI tests have a high intermittent count (either from b2g or gecko crashes) and are on the verge of being hidden on TBPL. See https://tbpl.mozilla.org/?tree=Try&rev=8fa5dc94a2d2 as an example of how intermittent they are.

Losing the Mn tests from TBPL has the potential to allow marionette regressions on to b2g which has a bigger impact than potentially just letting through WebAPI regressions.

we should split the suites or have it as a "chunk" in Mn

Comment 1

[Ed Morley [:emorley]]

Thank you for filing this after our meeting Monday - splitting these two into separate jobs will:
* Mean any future hiding of webapi tests will not affect the visibility of marionette sanity tests
* Make it easier for sheriffs and other devs to know where to file bugs against when issues occur (as well as reduce the tainting of Marionette as a harness by flaky webapi tests; which would inevitably be followed by the boy-who-called-wolf effect, similar to what happened with Android infra issues a few years ago).

Comment 2

[David Burns :automatedtester]

Attachment: Part 1 - Update buildconfig to split Mn and Mnw

Comment 3

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8448253 [details] [diff] [review]
Part 1 - Update buildconfig to split Mn and Mnw

Review of attachment 8448253 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good so far; you'll also have to add a 'marionette' section to the emulator's 'suite_config', here:

http://hg.mozilla.org/build/buildbot-configs/file/680a941f66fe/mozilla-tests/b2g_config.py#l1095

Comment 4

[David Burns :automatedtester]

Attachment: Update buildconfig to split Mn and Mnw

Comment 5

[David Burns :automatedtester]

Attachment: Create a manifest just for marionette web-api tests

Comment 6

[David Burns :automatedtester]

Attachment: Create a manifest just for marionette web-api tests

now with 100% more manifest file

Comment 7

[David Burns :automatedtester]

Attachment: Create a manifest just for marionette web-api tests

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8488081 [details] [diff] [review]
Create a manifest just for marionette web-api tests

Review of attachment 8488081 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8488065 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8488065 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good so far.   A couple of other changes you'll need:

Everywhere you see 'marionette-webapi', like http://hg.mozilla.org/build/buildbot-configs/file/4092544e664c/mozilla-tests/b2g_config.py#l1124, you'll need to add a --test-manifest=webapi-tests.ini argument.

Everywhere you see 'marionette-webapi', you'll need to add a new block for 'marionette' (the one you just created) and pass it the same --cfg arg as 'marionette-webapi' currently uses, but no need to include a --test-manifest argument.

Comment 10

[David Burns :automatedtester]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dac91d4a335c

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/dac91d4a335c

Comment 12

[David Burns :automatedtester]

Attachment: Update buildconfig to split Mn and Mnw

Updated as suggested, if you think it is ready for review please defer to the correct person.

Comment 13

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8489916 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8489916 [details] [diff] [review]:
-----------------------------------------------------------------

Almost ready for review!

::: mozilla-tests/b2g_config.py
@@ +1376,4 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",
> +                        "--manifest", "webapi-tests.ini"

Should be --test-manifest, see http://hg.mozilla.org/build/mozharness/file/d75b03fc4d56/scripts/marionette.py#l99

@@ +1883,3 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",

Need to add the --test-manifest arg here.

@@ +1916,3 @@
>                  'marionette-webapi': {
>                      'extra_args': [
>                          "--cfg", "marionette/automation_emulator_config.py",

Need to add the --test-manifest arg here.

Comment 14

[David Burns :automatedtester]

Attachment: Update buildconfig to split Mn and Mnw

Comment 15

[David Burns :automatedtester]

Changes made as suggested, if there isnt anything else you can suggest can you recommend a reviewer?

Comment 16

[Jordan Lund (:jlund)]

Comment on attachment 8491150 [details] [diff] [review]
Update buildconfig to split Mn and Mnw

Review of attachment 8491150 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

just a sanity check, IIUC the suite we are adding, 'marionette', looks to be the exact same as 'marionette-webapi' used to be prior to this patch. I can see how we add a manifest to marionette-webapi which I assume tells that suite to run webapi component only.

but how does the new suite/job we are adding run only marionette and not the webapi part? Does that need a manifest too or am I grepping things wrong?

Comment 17

[David Burns :automatedtester]

(In reply to Jordan Lund (:jlund) from comment #16)
> Comment on attachment 8491150 [details] [diff] [review]
> Update buildconfig to split Mn and Mnw
> 
> Review of attachment 8491150 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm :)
> 
> just a sanity check, IIUC the suite we are adding, 'marionette', looks to be
> the exact same as 'marionette-webapi' used to be prior to this patch. I can
> see how we add a manifest to marionette-webapi which I assume tells that
> suite to run webapi component only.
> 
> but how does the new suite/job we are adding run only marionette and not the
> webapi part? Does that need a manifest too or am I grepping things wrong?

Currently there is a new manifest in central for only webapi but because this change isnt being rolled out to all twigs/branches I want to get my build config change rolled out and then on cedar remove the webapi tests from the main manifest and just triple check (mainly to keep the sheriffs from locking me up and throwing away the key) that I don't break anything when i try do this on Inbound.

There shouldnt be any weird dependencies but want to check that before the major rollout.

Comment 18

[David Burns :automatedtester]

what's the process for getting this landed and rolled out? Do I need to set any flags?

Comment 19

[Jordan Lund (:jlund)]

(In reply to David Burns :automatedtester from comment #18)
> what's the process for getting this landed and rolled out? Do I need to set
> any flags?

land on default at hg.m.o/build/buildbot-configs whenever (it's a no-op)

that will run through some tests and show up in #releng whether it passed or failed. releng will merge it to production and roll it out across our integration system ~3 times a week. this bug will be updated whenever that happens.

Comment 20

[David Burns :automatedtester]

https://hg.mozilla.org/build/buildbot-configs/rev/ed3cf36cdc64

Comment 21

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-22 08:20 PT

Comment 22

[Ryan VanderMeulen [:RyanVM]]

This broke Mnw on Aurora (and probably the other B2G release branches as well):
https://tbpl.mozilla.org/php/getParsedLog.php?id=48588232&tree=Mozilla-Aurora

Also, is it expected that Mn appears to be running the full webapi suite as well? Based on the test counts, it appears that way.
Mnw before: marionette: 533/0/42
Mn after:   marionette: 535/0/42
Mnw after:  marionette: 203/0/9

Comment 23

[Ryan VanderMeulen [:RyanVM]]

I've reverted the buildbot-configs patch.
http://hg.mozilla.org/build/buildbot-configs/rev/08c6e52d150c

I wasn't comfortable backporting the in-tree manifest patch to the various release branches due to not being sure how comparable the list of tests is from branch to branch. I'd rather not try to sort that out while the tree is actively burning.

But to be clear, we will need in-tree manifests backported to Aurora, b2g32, b2g30, and b2g28_v1_3t before this goes live in production again. They can land a=test-only.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #22)
> Also, is it expected that Mn appears to be running the full webapi suite as
> well? Based on the test counts, it appears that way.
> Mnw before: marionette: 533/0/42
> Mn after:   marionette: 535/0/42
> Mnw after:  marionette: 203/0/9

OK, I guess this part was expected based on the earlier comments in the bug.

Comment 25

[David Burns :automatedtester]

Please land the manifest in Aurora, b2g32, b2g30, and b2g28_v1_3t (see comment 23)

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/205e1704dcb6
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/f18695cae418
https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/0b494a495ca3

v1.3t is near death, so I'm not going to bother landing this there.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/f18695cae418

Comment 28

[David Burns :automatedtester]

https://hg.mozilla.org/build/buildbot-configs/rev/8d46a609ae7e

Comment 29

[Ryan VanderMeulen [:RyanVM]]

So apparently the manifest cleanups I snuck in with these patches made Mnw perma-fail (see bug 1067363). Fantastic.

Reverted that part:
https://hg.mozilla.org/mozilla-central/rev/ad2e5aff3a3a
https://hg.mozilla.org/releases/mozilla-aurora/rev/973e99d258a8
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/75521744e0e8
https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/1dad764fbd52

Comment 30

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=0491f7375052

Comment 31

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=4d51396a2d76

Comment 32

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=8023ae10407a

Comment 33

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=02864fc30b79

Comment 34

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=e6194ab3ac7c

Comment 35

[David Burns :automatedtester]

https://tbpl.mozilla.org/?tree=Try&rev=e863fdfbd477

Comment 36

[David Burns :automatedtester]

Attachment: remove WebApi tests from Marionette unit tests manifests:

Comment 37

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8500039 [details] [diff] [review]
remove WebApi tests from Marionette unit tests manifests:

Review of attachment 8500039 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 38

[David Burns :automatedtester]

https://hg.mozilla.org/integration/mozilla-inbound/rev/349cd84710f2

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b9d04c58580a
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/4c0eb73dfac7
https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/aae7c12029c2

Comment 40

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/349cd84710f2

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/3a208c39bcb9
https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/3a208c39bcb9

Don't ask.