1031083 - buildbot changes to run selected b2g tests on c3.xlarge

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We have a few B2G test suites that fail on the current m1.medium nodes either consistently of frequently because they become CPU bound.

I've experimented with different node types in bug 1026800 and found that the tests seem to run acceptably (albeit slowly) on c3.large instances.

Can we create a new platform that would allow us to assign jobs selectively to this AWS node type?

Comment 1

[Aki Sasaki (not active)]

Dup of bug 1034055 ?

Comment 2

[Jonathan Griffin (:jgriffin)]

Works for me!

Comment 3

[Kim Moir [:kmoir] ET]

This isn't a duplicate of bug 1034055.  Bug 1034055 will implement the new slave class.  But there are still changes required in the buildbot configs so that the b2g tests use the new slave class which this bug should address.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We may need to scale the pool further for this, but bug 1034055 is fixed. Could we get a list of suites that should move to the more powerful instances ?

Comment 5

[Jonathan Griffin (:jgriffin)]

There are only two jobs that need this node type, both only running on cedar:

Gip (aka gaia-ui-test) on B2G ICS Emulator Opt
mochitest-media on B2G ICS Emulator Opt

Once they're running on cedar on the new node type, we'll still likely have to do some work to green them up before we can roll them out everywhere.

So, in the short term, the impact on the slave pool should be minimal.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

kmoir, is this something you could do relatively quickly ?

Comment 7

[Kim Moir [:kmoir] ET]

It would take a day or so to test, write patches and get reviews barring any unforseen problems.  We need to create a new slave class for the instance type so this means puppet, cloud tools changes etc in addition to buildbot-configs.  According to coop, my first priority is bug 1019724 right now but I can ask him when he returns tomorrow if this is a higher priority.

Comment 8

[Kim Moir [:kmoir] ET]

I talked to coop.  He was hesitant to add more jobs given our current load (89K test jobs yesterday!!).  Since bug 1042835 will remove significant jobs from the slave class, I'll fix this first and then take a look at this bug. But bug 1019724 is my current top priority.

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug1031083.patch

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1031083puppet.patch

new puppet slave class

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: bug1031083c-t.patch

cloud tools watch pending

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug1031083builder.diff

builder diff

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8466333 [details] [diff] [review]
bug1031083.patch

Review of attachment 8466333 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. We can cleanup later based on Jonathan responses.

::: mozilla-tests/b2g_config.py
@@ +123,5 @@
>      'hg_bin': 'hg',
>      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
>  }
>  
> +PLATFORMS['emulator']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator', 'ubuntu64_vm-b2g-lg-emulator', 'ubuntu64_hw-b2g-emulator']

Looks like ubuntu64_hw-b2g-emulator can be deprecated, assuming we have no further plans to use it. jgriffin ?

@@ +1640,5 @@
>  BRANCHES['cedar']['branch_name'] = "Cedar"
>  BRANCHES['cedar']['repo_path'] = "projects/cedar"
>  BRANCHES['cedar']['mozharness_tag'] = "default"
>  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['opt_unittest_suites'] = \
> +    MOCHITEST + CRASHTEST + XPCSHELL + MARIONETTE + JSREFTEST + GAIA_UI + CPPUNIT

jgriffin, are we wanting to run gaia ui-test side-by-side on slow and fast VMs ? If not, it looks like GAIA_UI could be removed here.

::: mozilla-tests/production_config.py
@@ +93,5 @@
>  SLAVES['ubuntu32_vm-b2gdt'] = SLAVES['ubuntu32_vm']
>  SLAVES['ubuntu64_vm-b2g'] = SLAVES['ubuntu64_vm']
>  SLAVES['ubuntu64_vm-b2gdt'] = SLAVES['ubuntu64_vm']
>  SLAVES['ubuntu64_vm-b2g-emulator'] = SLAVES['ubuntu64_vm']
> +SLAVES['ubuntu64_vm-b2g-lg-emulator'] = SLAVES['ubuntu64_vm_large'] 

Nit, trailing whitespace.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8466338 [details] [diff] [review]
bug1031083c-t.patch

Not sure what the trailing .* is for, given builders names of
  b2g_emulator_vm_large cedar opt test gaia-ui-test
  b2g_emulator_vm_large cedar opt test mochitest-media
but otherwise looks good.

Comment 15

[Jonathan Griffin (:jgriffin)]

(In reply to Nick Thomas [:nthomas] from comment #13)
> Comment on attachment 8466333 [details] [diff] [review]
> bug1031083.patch
> 
> Review of attachment 8466333 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm. We can cleanup later based on Jonathan responses.
> 
> ::: mozilla-tests/b2g_config.py
> @@ +123,5 @@
> >      'hg_bin': 'hg',
> >      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
> >  }
> >  
> > +PLATFORMS['emulator']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator', 'ubuntu64_vm-b2g-lg-emulator', 'ubuntu64_hw-b2g-emulator']
> 
> Looks like ubuntu64_hw-b2g-emulator can be deprecated, assuming we have no
> further plans to use it. jgriffin ?

That's correct, it can be nuked.

> 
> @@ +1640,5 @@
> >  BRANCHES['cedar']['branch_name'] = "Cedar"
> >  BRANCHES['cedar']['repo_path'] = "projects/cedar"
> >  BRANCHES['cedar']['mozharness_tag'] = "default"
> >  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['opt_unittest_suites'] = \
> > +    MOCHITEST + CRASHTEST + XPCSHELL + MARIONETTE + JSREFTEST + GAIA_UI + CPPUNIT
> 
> jgriffin, are we wanting to run gaia ui-test side-by-side on slow and fast
> VMs ? If not, it looks like GAIA_UI could be removed here.

No, we only need the fast VM, we know it doesn't work at all on the slow one.

> 
> ::: mozilla-tests/production_config.py
> @@ +93,5 @@
> >  SLAVES['ubuntu32_vm-b2gdt'] = SLAVES['ubuntu32_vm']
> >  SLAVES['ubuntu64_vm-b2g'] = SLAVES['ubuntu64_vm']
> >  SLAVES['ubuntu64_vm-b2gdt'] = SLAVES['ubuntu64_vm']
> >  SLAVES['ubuntu64_vm-b2g-emulator'] = SLAVES['ubuntu64_vm']
> > +SLAVES['ubuntu64_vm-b2g-lg-emulator'] = SLAVES['ubuntu64_vm_large'] 
> 
> Nit, trailing whitespace.

Comment 16

[Kim Moir [:kmoir] ET]

Attachment: bug1031083-2.patch

patch to address whitespace, remove duplicate gaia-ui tests and deprecate ubuntu64_hw-b2g-emulator as per review comments

Comment 17

[Kim Moir [:kmoir] ET]

Attachment: bug1031083builder-2.diff

Comment 18

[Kim Moir [:kmoir] ET]

Comment on attachment 8466336 [details] [diff] [review]
bug1031083puppet.patch

and merged to production

Comment 19

[Kim Moir [:kmoir] ET]

Attachment: bug1031083c-t-2.patch

fixed cloud tools patch

Comment 20

[Kim Moir [:kmoir] ET]

Attachment: bug1031083puppet-2.patch

remove ubuntu64_hw-b2g-emulator from puppet after reconfig

Comment 21

[Kim Moir [:kmoir] ET]

Comment on attachment 8467806 [details] [diff] [review]
bug1031083c-t-2.patch

except removed extra , at eol

Comment 22

[Kim Moir [:kmoir] ET]

In production

Comment 23

[Kim Moir [:kmoir] ET]

Verified on tbpl that they are running on the correct instance type.  However, the GIP tests failed with a harness failure (red) and the gaia ui tests are orange. :jgriffin could you investigate?

Comment 24

[Jonathan Griffin (:jgriffin)]

Yes, I will look.  Thanks for making the switch!  Greening up the tests will be a separate project, so I think we can close this as resolved.