Closed Bug 1330310 Opened 7 years ago Closed 7 years ago

Switch macosx debug tests to buildbot-bridge

Categories

(Firefox Build System :: Task Configuration, task, P1)

All
macOS
task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wcosta, Assigned: wcosta)

References

Details

User Story

Enable talos and unit tests in builbbot-bridge, and disable buildbot scheduling.

Attachments

(1 file, 4 obsolete files)

      No description provided.
Comment on attachment 8825904 [details]
Bug 1330310: Switch macosx tests to buildbot-bridge.

https://reviewboard.mozilla.org/r/103978/#review104660

the mozharness config changes look fine.
Attachment #8825904 - Flags: review?(catlee) → review+
Comment on attachment 8825904 [details]
Bug 1330310: Switch macosx tests to buildbot-bridge.

See try build in mozharness.
Attachment #8825904 - Flags: feedback?(jmaher)
Comment on attachment 8825904 [details]
Bug 1330310: Switch macosx tests to buildbot-bridge.

https://reviewboard.mozilla.org/r/103978/#review104892

I have a lot of confusion reading this patch, most of it applies generically to the use of chunked: {true|false}.  Possibly I don't understand the differences in getting tests running in BBB- if there are existing docs to outline what is going on that would be ok to point me at.

::: taskcluster/ci/test/tests.yml:74
(Diff revision 3)
>                      - android/androidarm_4_3-tc.py
>                  extra-options:
>                      - --test-suite=crashtest
>              default:
>                  script: desktop_unittest.py
> -                chunked: true
> +                chunked: false

will we need by-test-platform here?

::: taskcluster/ci/test/tests.yml:503
(Diff revision 3)
>                      # note that Android runs fewer suites than other platforms
>                      - --test-suite=mochitest-plain-clipboard
>              default:
>                  script: desktop_unittest.py
>                  no-read-buildbot-config: true
> -                chunked: true
> +                chunked: false

I don't understand chunked at the top level and then again inside mozharness:default, it is hard to figure out which platforms are chunked here.

::: taskcluster/ci/test/tests.yml:546
(Diff revision 3)
>              windows.*: false
>              default: both
>      mozharness:
>          script: desktop_unittest.py
>          no-read-buildbot-config: true
> +        chunked: true

is this needed as we have definitions of the chunks earlier in the mochitest-devtools section?

::: taskcluster/ci/test/tests.yml:736
(Diff revision 3)
>  
>  mochitest-webgl:
>      description: "Mochitest webgl run"
>      suite: mochitest/mochitest-gl
>      treeherder-symbol: tc-M(gl)
> +    unittest-try-name: "mochitest-gl"

this field 'unittest-try-name' is either confusing or wrong.  Does this mean this is the name we use on try server, but nowhere else?

I really wish we would just call this mochitest-gl instead of mochitest-webgl to save a headache or two.  That would help try serve syntax.

::: taskcluster/ci/test/tests.yml:813
(Diff revision 3)
>              default: default
>      chunks:
>          by-test-platform:
>              android-4.3-arm7-api-15/debug: 48
>              android.*: 16
> -            macosx.*: 1
> +            macosx.*: 2

I believe this is 3 for osx.

::: taskcluster/taskgraph/transforms/tests.py:78
(Diff revision 3)
> +TASKCLUSTER_WORKER_TESTS = [
> +    'cppunit',
> +    'crashtest',
> +    'jsreftest',
> +    'mochitest-webgl',
> +]

could you add a comment here indicating why we have this block of different tests?
Attachment #8825904 - Flags: review-
besides the gtest failures (bug 1331049), there are 5 tests which are not scheduling properly, they are these buildernames:
Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-media
  - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-media-e10s

Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-clipboard
  - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-clipboard-e10s

Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-1
Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-2
Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-3
  - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-gl-e10s-{x}

if no notice, these specific jobs have mochitest-{subsuite}-e10s[-{x}].  Also s/webgl/gl/

Doing those 5 things should get us to 100%.  I would like to measure this with |--rebuild 20| against a buildbot build with --rebuild 20 and validate the runtimes are similar- weekends are good for making try pushes like that.
From what I understand, when this lands, OS X debug builds made in TC will be tier-2, and we will be immediately disabling tests of OS X debug builds made in Buildbot (and possibly disabling the Buildbot builds themselves).
I think this would have to be tier-1 if we are disabling testing of BB builds (and ideally disabling the BB builds themselves).

I am in agreement that this is OSX Debug only and would be replacing the default testing done on BB OSX Debug builds.
as a note, I am doing a --rebuild 20 with/without the patch to get an idea for runtimes and intermittents- since gtest is problematic, this will uncover anything else.
a few missing jobs:
jittest
mochitest-gpu (we run e10s, but not the non-e10s type)
run as VP(b-m, b-y):
 - media-youtube-tests (tier-2 tests)
 - media-tests (tier-2 tests)

https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded&filter-searchStr=crashtest&fromchange=4632c4a3e0840090c8f5f32e8c7412ac3f1d29e5&tochange=f0b4f6d23c1e9889f6753be2389d8867ea4a08b5

I will get to the analysis of failure rates/runtime when I get some time- I have a script, just need to sanity check it and filter out what might be actionable.
I analyzed the failure rates and we are looking good.  In terms of running faster or slower, I see:
testtype,avg_greentime (TC),avg_greentime (BB),TC % slower
crashtest-e10s,725,592,0.224662162162
gtest,1294,719,0.799721835883
cppunit,124,116,0.0689655172414
marionette,936,890,0.0516853932584

we have already filed bug 1331049 for gtest.  I am surprised the crashtest-e10s job is 22% slower, that is something to look at probably.  cppunit and marionette are not much longer, but it is odd that the same revision shows a longer trend on osx debug.
looking at crashtests, there is one job that hit an edge case and timed out in 51 minutes- the rest of the jobs are at parity.

looking at cppunit, there is a difference, but 6 seconds doesn't seem like much, looking at 4 different sets of logs, I don't see any patterns in test time differences (max difference of 4ms)

looking at marionette, there are 1 green job that had 8 extra minutes of runtime, I believe it is a bad machine:  t-yosemite-r7-0239.  This took 90 seconds longer to setup mozharness bits prior to running the test, and the testrun itself too 416 seconds longer to run- since this is an isolated case vs that 19 other green runs that all took the same time, I am going to ignore it.

In summary, when we get the missing jobs from comment 20 added, and resolve bug 1331049, we should be ready to go with osx debug tests.
as a note, jittest is a single chunk and note jittest-1, but just jittest.

I see gpu fixed and the two VP jobs :)
Comment on attachment 8825904 [details]
Bug 1330310: Switch macosx tests to buildbot-bridge.

https://reviewboard.mozilla.org/r/103978/#review105954

::: taskcluster/taskgraph/transforms/tests.py:966
(Diff revisions 4 - 16)
> -    # in buildbot, mochitest-webgl is called mochitest-gl
> -    test_name = test_name.replace('mochitest-webgl', 'mochitest-gl')
> -    mozharness = test['mozharness']
>  
> -    if test['e10s'] and not test_name.endswith('-e10s'):
> +    # mochitest e10s follows the pattern mochitest-e10s-<suffix>
> +    # in buildbot, except for these special cases

It's going to be so nice to see the backside of these unique, special things.

::: taskcluster/taskgraph/transforms/tests.py:989
(Diff revisions 4 - 16)
> -    if mozharness.get('chunked', False):
> +    if test_name.startswith('mochitest') \
> +            and test_name.endswith('e10s') \
> +            and not any(map(
> +                lambda name: test_name.startswith(name),
> +                buildbot_specials
> +            )):

A generator expression would be a little better here:

    any(name.startswith(name) for name in buildbot_specials)
    
No need to change it -- just pointing out a nicer Python syntax for next time :)

::: testing/mozharness/configs/builds/releng_base_mac_64_builds.py:35
(Diff revision 4)
>      'objdir': 'obj-firefox',
>      'tooltool_script': ["/builds/tooltool.py"],
>      'tooltool_bootstrap': "setup.sh",
>      'enable_count_ctors': False,
> -    'enable_talos_sendchange': True,
> -    'enable_unittest_sendchange': True,
> +    'enable_talos_sendchange': False,
> +    'enable_unittest_sendchange': False,

/me claps hands with glee
Attachment #8825904 - Flags: review?(dustin) → review+
it is odd that media-youtube-test VP(b-y) are showing 1-2 minutes longer on TC vs BB.  In looking at this I think there is some difference in seconds and in some cases a master lag issue in buildbot which can delay things like blobber upload/cleanup (at the end of the script) by up to a minute.

I don't see anything odd other than that- the jittest and gpu tests are a-ok :)
Pushed by wcosta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9f402afa3864
Switch macosx tests to buildbot-bridge. r=catlee,dustin
Keywords: leave-open
* media-tests-youtube is named media-youtube-tests in builder.
* media-tests-twitch runs on youtube builder.
Attachment #8836088 - Flags: review?(jmaher)
Attachment #8836093 - Flags: review?(jmaher)
Comment on attachment 8836093 [details] [diff] [review]
jsreftests are two chunks. r=jmaher

Review of attachment 8836093 [details] [diff] [review]:
-----------------------------------------------------------------

this is fine, but we still need to run jsreftest for osx via bbb
Attachment #8836093 - Flags: review?(jmaher) → review+
Comment on attachment 8836088 [details] [diff] [review]
use correct builder names for media tests. r=jmaher

Review of attachment 8836088 [details] [diff] [review]:
-----------------------------------------------------------------

the twitch tests do not run on osx, possibly we don't need to do anything here?
Attachment #8836088 - Flags: review?(jmaher) → review-
Attachment #8836088 - Attachment is obsolete: true
Attachment #8836124 - Flags: review?(jmaher)
Comment on attachment 8836124 [details] [diff] [review]
jsreftests are two chunks. r=jmaher

Review of attachment 8836124 [details] [diff] [review]:
-----------------------------------------------------------------

I already reviewed this one
Attachment #8836124 - Flags: review?(jmaher)
Attachment #8836124 - Attachment is obsolete: true
* media-tests-youtube is named media-youtube-tests in builder.
* media-tests-twitch don't run on mac.
Attachment #8836135 - Flags: review?(jmaher)
Backout by archaeopteryx@coole-files.de:
https://hg.mozilla.org/integration/autoland/rev/705e1bc5b69b
Backed out changeset 9f402afa3864 on wcosta's request. r=backout on a CLOSED TREE
Comment on attachment 8836135 [details] [diff] [review]
use correct builder names for media tests. r=jmaher

Review of attachment 8836135 [details] [diff] [review]:
-----------------------------------------------------------------

everything else is great, just confused on the twitch/youtube thing.

::: taskcluster/taskgraph/transforms/tests.py
@@ +1076,5 @@
>      taskdesc['worker-type'] = 'buildbot-bridge/buildbot-bridge'
>  
> +    if test_name == 'media-twitch-tests':
> +        # twitch tests run on youtube builder
> +        test_name = 'media-youtube-tests'

this is the only piece I don't understand.
Attachment #8836135 - Flags: review?(jmaher) → review-
Attachment #8836093 - Attachment is obsolete: true
Attachment #8836135 - Attachment is obsolete: true
Summary: Switch macosx tests to buildbot-bridge → Switch macosx debug tests to buildbot-bridge
Blocks: 1340606
Depends on: 1340619
Depends on: 1339133
Pushed by wcosta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8121f087abc4
Switch macosx tests to buildbot-bridge. r=catlee,dustin
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.