buildbot changes to run selected b2g tests on c3.xlarge

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jgriffin, Assigned: kmoir)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(7 attachments, 1 obsolete attachment)

(Reporter)

Description

3 years ago
We have a few B2G test suites that fail on the current m1.medium nodes either consistently of frequently because they become CPU bound.

I've experimented with different node types in bug 1026800 and found that the tests seem to run acceptably (albeit slowly) on c3.large instances.

Can we create a new platform that would allow us to assign jobs selectively to this AWS node type?

Comment 1

3 years ago
Dup of bug 1034055 ?
(Reporter)

Comment 2

3 years ago
Works for me!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1034055
(Assignee)

Updated

3 years ago
Assignee: nobody → kmoir
(Assignee)

Updated

3 years ago
Assignee: kmoir → nobody
(Assignee)

Comment 3

3 years ago
This isn't a duplicate of bug 1034055.  Bug 1034055 will implement the new slave class.  But there are still changes required in the buildbot configs so that the b2g tests use the new slave class which this bug should address.
Status: RESOLVED → REOPENED
Depends on: 1034055
Resolution: DUPLICATE → ---
(Assignee)

Updated

3 years ago
Summary: Add new buildbot platform for running tests on c3.large AWS instances → buildbot changes to run selected b2g tests on c3.xlarge
We may need to scale the pool further for this, but bug 1034055 is fixed. Could we get a list of suites that should move to the more powerful instances ?
(Reporter)

Comment 5

3 years ago
There are only two jobs that need this node type, both only running on cedar:

Gip (aka gaia-ui-test) on B2G ICS Emulator Opt
mochitest-media on B2G ICS Emulator Opt

Once they're running on cedar on the new node type, we'll still likely have to do some work to green them up before we can roll them out everywhere.

So, in the short term, the impact on the slave pool should be minimal.
kmoir, is this something you could do relatively quickly ?
Flags: needinfo?(kmoir)
(Assignee)

Comment 7

3 years ago
It would take a day or so to test, write patches and get reviews barring any unforseen problems.  We need to create a new slave class for the instance type so this means puppet, cloud tools changes etc in addition to buildbot-configs.  According to coop, my first priority is bug 1019724 right now but I can ask him when he returns tomorrow if this is a higher priority.
Flags: needinfo?(kmoir)
(Assignee)

Comment 8

3 years ago
I talked to coop.  He was hesitant to add more jobs given our current load (89K test jobs yesterday!!).  Since bug 1042835 will remove significant jobs from the slave class, I'll fix this first and then take a look at this bug. But bug 1019724 is my current top priority.
Depends on: 1042835
(Assignee)

Updated

3 years ago
Assignee: nobody → kmoir
(Assignee)

Comment 9

3 years ago
Created attachment 8466333 [details] [diff] [review]
bug1031083.patch
Attachment #8466333 - Flags: review?(nthomas)
(Assignee)

Comment 10

3 years ago
Created attachment 8466336 [details] [diff] [review]
bug1031083puppet.patch

new puppet slave class
(Assignee)

Updated

3 years ago
Attachment #8466336 - Flags: review?(nthomas)
(Assignee)

Comment 11

3 years ago
Created attachment 8466338 [details] [diff] [review]
bug1031083c-t.patch

cloud tools watch pending
Attachment #8466338 - Flags: review?(nthomas)
(Assignee)

Comment 12

3 years ago
Created attachment 8466339 [details]
bug1031083builder.diff

builder diff
Comment on attachment 8466333 [details] [diff] [review]
bug1031083.patch

Review of attachment 8466333 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. We can cleanup later based on Jonathan responses.

::: mozilla-tests/b2g_config.py
@@ +123,5 @@
>      'hg_bin': 'hg',
>      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
>  }
>  
> +PLATFORMS['emulator']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator', 'ubuntu64_vm-b2g-lg-emulator', 'ubuntu64_hw-b2g-emulator']

Looks like ubuntu64_hw-b2g-emulator can be deprecated, assuming we have no further plans to use it. jgriffin ?

@@ +1640,5 @@
>  BRANCHES['cedar']['branch_name'] = "Cedar"
>  BRANCHES['cedar']['repo_path'] = "projects/cedar"
>  BRANCHES['cedar']['mozharness_tag'] = "default"
>  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['opt_unittest_suites'] = \
> +    MOCHITEST + CRASHTEST + XPCSHELL + MARIONETTE + JSREFTEST + GAIA_UI + CPPUNIT

jgriffin, are we wanting to run gaia ui-test side-by-side on slow and fast VMs ? If not, it looks like GAIA_UI could be removed here.

::: mozilla-tests/production_config.py
@@ +93,5 @@
>  SLAVES['ubuntu32_vm-b2gdt'] = SLAVES['ubuntu32_vm']
>  SLAVES['ubuntu64_vm-b2g'] = SLAVES['ubuntu64_vm']
>  SLAVES['ubuntu64_vm-b2gdt'] = SLAVES['ubuntu64_vm']
>  SLAVES['ubuntu64_vm-b2g-emulator'] = SLAVES['ubuntu64_vm']
> +SLAVES['ubuntu64_vm-b2g-lg-emulator'] = SLAVES['ubuntu64_vm_large'] 

Nit, trailing whitespace.
Attachment #8466333 - Flags: review?(nthomas) → review+

Updated

3 years ago
Attachment #8466336 - Flags: review?(nthomas) → review+
Comment on attachment 8466338 [details] [diff] [review]
bug1031083c-t.patch

Not sure what the trailing .* is for, given builders names of
  b2g_emulator_vm_large cedar opt test gaia-ui-test
  b2g_emulator_vm_large cedar opt test mochitest-media
but otherwise looks good.
Attachment #8466338 - Flags: review?(nthomas) → review+
(Reporter)

Comment 15

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #13)
> Comment on attachment 8466333 [details] [diff] [review]
> bug1031083.patch
> 
> Review of attachment 8466333 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm. We can cleanup later based on Jonathan responses.
> 
> ::: mozilla-tests/b2g_config.py
> @@ +123,5 @@
> >      'hg_bin': 'hg',
> >      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
> >  }
> >  
> > +PLATFORMS['emulator']['slave_platforms'] = ['ubuntu64_vm-b2g-emulator', 'ubuntu64_vm-b2g-lg-emulator', 'ubuntu64_hw-b2g-emulator']
> 
> Looks like ubuntu64_hw-b2g-emulator can be deprecated, assuming we have no
> further plans to use it. jgriffin ?

That's correct, it can be nuked.

> 
> @@ +1640,5 @@
> >  BRANCHES['cedar']['branch_name'] = "Cedar"
> >  BRANCHES['cedar']['repo_path'] = "projects/cedar"
> >  BRANCHES['cedar']['mozharness_tag'] = "default"
> >  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['opt_unittest_suites'] = \
> > +    MOCHITEST + CRASHTEST + XPCSHELL + MARIONETTE + JSREFTEST + GAIA_UI + CPPUNIT
> 
> jgriffin, are we wanting to run gaia ui-test side-by-side on slow and fast
> VMs ? If not, it looks like GAIA_UI could be removed here.

No, we only need the fast VM, we know it doesn't work at all on the slow one.

> 
> ::: mozilla-tests/production_config.py
> @@ +93,5 @@
> >  SLAVES['ubuntu32_vm-b2gdt'] = SLAVES['ubuntu32_vm']
> >  SLAVES['ubuntu64_vm-b2g'] = SLAVES['ubuntu64_vm']
> >  SLAVES['ubuntu64_vm-b2gdt'] = SLAVES['ubuntu64_vm']
> >  SLAVES['ubuntu64_vm-b2g-emulator'] = SLAVES['ubuntu64_vm']
> > +SLAVES['ubuntu64_vm-b2g-lg-emulator'] = SLAVES['ubuntu64_vm_large'] 
> 
> Nit, trailing whitespace.
(Assignee)

Comment 16

3 years ago
Created attachment 8467795 [details] [diff] [review]
bug1031083-2.patch

patch to address whitespace, remove duplicate gaia-ui tests and deprecate ubuntu64_hw-b2g-emulator as per review comments
(Assignee)

Comment 17

3 years ago
Created attachment 8467796 [details] [diff] [review]
bug1031083builder-2.diff
Attachment #8466339 - Attachment is obsolete: true
(Assignee)

Comment 18

3 years ago
Comment on attachment 8466336 [details] [diff] [review]
bug1031083puppet.patch

and merged to production
Attachment #8466336 - Flags: checked-in+
(Assignee)

Comment 19

3 years ago
Created attachment 8467806 [details] [diff] [review]
bug1031083c-t-2.patch

fixed cloud tools patch
(Assignee)

Comment 20

3 years ago
Created attachment 8467815 [details] [diff] [review]
bug1031083puppet-2.patch

remove ubuntu64_hw-b2g-emulator from puppet after reconfig
Attachment #8467815 - Flags: review?(nthomas)
(Assignee)

Comment 21

3 years ago
Comment on attachment 8467806 [details] [diff] [review]
bug1031083c-t-2.patch

except removed extra , at eol
Attachment #8467806 - Flags: checked-in+
(Assignee)

Comment 22

3 years ago
In production

Updated

3 years ago
Attachment #8467815 - Flags: review?(nthomas) → review+
(Assignee)

Updated

3 years ago
Attachment #8467815 - Flags: checked-in+
(Assignee)

Comment 23

3 years ago
Verified on tbpl that they are running on the correct instance type.  However, the GIP tests failed with a harness failure (red) and the gaia ui tests are orange. :jgriffin could you investigate?
(Reporter)

Comment 24

3 years ago
Yes, I will look.  Thanks for making the switch!  Greening up the tests will be a separate project, so I think we can close this as resolved.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.