Closed
Bug 1330310
Opened 7 years ago
Closed 7 years ago
Switch macosx debug tests to buildbot-bridge
Categories
(Firefox Build System :: Task Configuration, task, P1)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: wcosta, Assigned: wcosta)
References
Details
User Story
Enable talos and unit tests in builbbot-bridge, and disable buildbot scheduling.
Attachments
(1 file, 4 obsolete files)
No description provided.
Comment hidden (mozreview-request) |
Comment 2•7 years ago
|
||
mozreview-review |
Comment on attachment 8825904 [details] Bug 1330310: Switch macosx tests to buildbot-bridge. https://reviewboard.mozilla.org/r/103978/#review104660 the mozharness config changes look fine.
Attachment #8825904 -
Flags: review?(catlee) → review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 5•7 years ago
|
||
Comment on attachment 8825904 [details] Bug 1330310: Switch macosx tests to buildbot-bridge. See try build in mozharness.
Attachment #8825904 -
Flags: feedback?(jmaher)
Comment 6•7 years ago
|
||
mozreview-review |
Comment on attachment 8825904 [details] Bug 1330310: Switch macosx tests to buildbot-bridge. https://reviewboard.mozilla.org/r/103978/#review104892 I have a lot of confusion reading this patch, most of it applies generically to the use of chunked: {true|false}. Possibly I don't understand the differences in getting tests running in BBB- if there are existing docs to outline what is going on that would be ok to point me at. ::: taskcluster/ci/test/tests.yml:74 (Diff revision 3) > - android/androidarm_4_3-tc.py > extra-options: > - --test-suite=crashtest > default: > script: desktop_unittest.py > - chunked: true > + chunked: false will we need by-test-platform here? ::: taskcluster/ci/test/tests.yml:503 (Diff revision 3) > # note that Android runs fewer suites than other platforms > - --test-suite=mochitest-plain-clipboard > default: > script: desktop_unittest.py > no-read-buildbot-config: true > - chunked: true > + chunked: false I don't understand chunked at the top level and then again inside mozharness:default, it is hard to figure out which platforms are chunked here. ::: taskcluster/ci/test/tests.yml:546 (Diff revision 3) > windows.*: false > default: both > mozharness: > script: desktop_unittest.py > no-read-buildbot-config: true > + chunked: true is this needed as we have definitions of the chunks earlier in the mochitest-devtools section? ::: taskcluster/ci/test/tests.yml:736 (Diff revision 3) > > mochitest-webgl: > description: "Mochitest webgl run" > suite: mochitest/mochitest-gl > treeherder-symbol: tc-M(gl) > + unittest-try-name: "mochitest-gl" this field 'unittest-try-name' is either confusing or wrong. Does this mean this is the name we use on try server, but nowhere else? I really wish we would just call this mochitest-gl instead of mochitest-webgl to save a headache or two. That would help try serve syntax. ::: taskcluster/ci/test/tests.yml:813 (Diff revision 3) > default: default > chunks: > by-test-platform: > android-4.3-arm7-api-15/debug: 48 > android.*: 16 > - macosx.*: 1 > + macosx.*: 2 I believe this is 3 for osx. ::: taskcluster/taskgraph/transforms/tests.py:78 (Diff revision 3) > +TASKCLUSTER_WORKER_TESTS = [ > + 'cppunit', > + 'crashtest', > + 'jsreftest', > + 'mochitest-webgl', > +] could you add a comment here indicating why we have this block of different tests?
Attachment #8825904 -
Flags: review-
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 12•7 years ago
|
||
besides the gtest failures (bug 1331049), there are 5 tests which are not scheduling properly, they are these buildernames: Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-media - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-media-e10s Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-clipboard - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-clipboard-e10s Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-1 Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-2 Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-webgl-3 - should be: Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-gl-e10s-{x} if no notice, these specific jobs have mochitest-{subsuite}-e10s[-{x}]. Also s/webgl/gl/ Doing those 5 things should get us to 100%. I would like to measure this with |--rebuild 20| against a buildbot build with --rebuild 20 and validate the runtimes are similar- weekends are good for making try pushes like that.
Comment 13•7 years ago
|
||
From what I understand, when this lands, OS X debug builds made in TC will be tier-2, and we will be immediately disabling tests of OS X debug builds made in Buildbot (and possibly disabling the Buildbot builds themselves).
Comment 14•7 years ago
|
||
I think this would have to be tier-1 if we are disabling testing of BB builds (and ideally disabling the BB builds themselves). I am in agreement that this is OSX Debug only and would be replacing the default testing done on BB OSX Debug builds.
Comment hidden (mozreview-request) |
Comment 16•7 years ago
|
||
I have a patch here (ugly hacks) that does the job: https://hg.mozilla.org/try/rev/67a3f98c62b930391bb1a622850b6f684a1e1ed4 and you can see it on try server: https://treeherder.mozilla.org/#/jobs?repo=try&revision=639fb3f1cc1770b9fb5a607c575db211bbf51721
Comment hidden (mozreview-request) |
Comment 18•7 years ago
|
||
as a note, I am doing a --rebuild 20 with/without the patch to get an idea for runtimes and intermittents- since gtest is problematic, this will uncover anything else.
Comment hidden (mozreview-request) |
Comment 20•7 years ago
|
||
a few missing jobs: jittest mochitest-gpu (we run e10s, but not the non-e10s type) run as VP(b-m, b-y): - media-youtube-tests (tier-2 tests) - media-tests (tier-2 tests) https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded&filter-searchStr=crashtest&fromchange=4632c4a3e0840090c8f5f32e8c7412ac3f1d29e5&tochange=f0b4f6d23c1e9889f6753be2389d8867ea4a08b5 I will get to the analysis of failure rates/runtime when I get some time- I have a script, just need to sanity check it and filter out what might be actionable.
Comment 21•7 years ago
|
||
I analyzed the failure rates and we are looking good. In terms of running faster or slower, I see: testtype,avg_greentime (TC),avg_greentime (BB),TC % slower crashtest-e10s,725,592,0.224662162162 gtest,1294,719,0.799721835883 cppunit,124,116,0.0689655172414 marionette,936,890,0.0516853932584 we have already filed bug 1331049 for gtest. I am surprised the crashtest-e10s job is 22% slower, that is something to look at probably. cppunit and marionette are not much longer, but it is odd that the same revision shows a longer trend on osx debug.
Comment 22•7 years ago
|
||
looking at crashtests, there is one job that hit an edge case and timed out in 51 minutes- the rest of the jobs are at parity. looking at cppunit, there is a difference, but 6 seconds doesn't seem like much, looking at 4 different sets of logs, I don't see any patterns in test time differences (max difference of 4ms) looking at marionette, there are 1 green job that had 8 extra minutes of runtime, I believe it is a bad machine: t-yosemite-r7-0239. This took 90 seconds longer to setup mozharness bits prior to running the test, and the testrun itself too 416 seconds longer to run- since this is an isolated case vs that 19 other green runs that all took the same time, I am going to ignore it. In summary, when we get the missing jobs from comment 20 added, and resolve bug 1331049, we should be ready to go with osx debug tests.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 27•7 years ago
|
||
as a note, jittest is a single chunk and note jittest-1, but just jittest. I see gpu fixed and the two VP jobs :)
Comment hidden (mozreview-request) |
Comment 29•7 years ago
|
||
mozreview-review |
Comment on attachment 8825904 [details] Bug 1330310: Switch macosx tests to buildbot-bridge. https://reviewboard.mozilla.org/r/103978/#review105954 ::: taskcluster/taskgraph/transforms/tests.py:966 (Diff revisions 4 - 16) > - # in buildbot, mochitest-webgl is called mochitest-gl > - test_name = test_name.replace('mochitest-webgl', 'mochitest-gl') > - mozharness = test['mozharness'] > > - if test['e10s'] and not test_name.endswith('-e10s'): > + # mochitest e10s follows the pattern mochitest-e10s-<suffix> > + # in buildbot, except for these special cases It's going to be so nice to see the backside of these unique, special things. ::: taskcluster/taskgraph/transforms/tests.py:989 (Diff revisions 4 - 16) > - if mozharness.get('chunked', False): > + if test_name.startswith('mochitest') \ > + and test_name.endswith('e10s') \ > + and not any(map( > + lambda name: test_name.startswith(name), > + buildbot_specials > + )): A generator expression would be a little better here: any(name.startswith(name) for name in buildbot_specials) No need to change it -- just pointing out a nicer Python syntax for next time :) ::: testing/mozharness/configs/builds/releng_base_mac_64_builds.py:35 (Diff revision 4) > 'objdir': 'obj-firefox', > 'tooltool_script': ["/builds/tooltool.py"], > 'tooltool_bootstrap': "setup.sh", > 'enable_count_ctors': False, > - 'enable_talos_sendchange': True, > - 'enable_unittest_sendchange': True, > + 'enable_talos_sendchange': False, > + 'enable_unittest_sendchange': False, /me claps hands with glee
Attachment #8825904 -
Flags: review?(dustin) → review+
Comment 30•7 years ago
|
||
it is odd that media-youtube-test VP(b-y) are showing 1-2 minutes longer on TC vs BB. In looking at this I think there is some difference in seconds and in some cases a master lag issue in buildbot which can delay things like blobber upload/cleanup (at the end of the script) by up to a minute. I don't see anything odd other than that- the jittest and gpu tests are a-ok :)
Comment hidden (mozreview-request) |
Comment 33•7 years ago
|
||
Pushed by wcosta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9f402afa3864 Switch macosx tests to buildbot-bridge. r=catlee,dustin
Assignee | ||
Updated•7 years ago
|
Keywords: leave-open
Assignee | ||
Comment 34•7 years ago
|
||
* media-tests-youtube is named media-youtube-tests in builder. * media-tests-twitch runs on youtube builder.
Assignee | ||
Comment 35•7 years ago
|
||
Assignee | ||
Updated•7 years ago
|
Attachment #8836088 -
Flags: review?(jmaher)
Assignee | ||
Updated•7 years ago
|
Attachment #8836093 -
Flags: review?(jmaher)
Comment 36•7 years ago
|
||
Comment on attachment 8836093 [details] [diff] [review] jsreftests are two chunks. r=jmaher Review of attachment 8836093 [details] [diff] [review]: ----------------------------------------------------------------- this is fine, but we still need to run jsreftest for osx via bbb
Attachment #8836093 -
Flags: review?(jmaher) → review+
Comment 37•7 years ago
|
||
Comment on attachment 8836088 [details] [diff] [review] use correct builder names for media tests. r=jmaher Review of attachment 8836088 [details] [diff] [review]: ----------------------------------------------------------------- the twitch tests do not run on osx, possibly we don't need to do anything here?
Attachment #8836088 -
Flags: review?(jmaher) → review-
Assignee | ||
Updated•7 years ago
|
Attachment #8836088 -
Attachment is obsolete: true
Assignee | ||
Comment 38•7 years ago
|
||
Assignee | ||
Updated•7 years ago
|
Attachment #8836124 -
Flags: review?(jmaher)
Comment 39•7 years ago
|
||
Comment on attachment 8836124 [details] [diff] [review] jsreftests are two chunks. r=jmaher Review of attachment 8836124 [details] [diff] [review]: ----------------------------------------------------------------- I already reviewed this one
Attachment #8836124 -
Flags: review?(jmaher)
Assignee | ||
Updated•7 years ago
|
Attachment #8836124 -
Attachment is obsolete: true
Assignee | ||
Comment 40•7 years ago
|
||
* media-tests-youtube is named media-youtube-tests in builder. * media-tests-twitch don't run on mac.
Assignee | ||
Updated•7 years ago
|
Attachment #8836135 -
Flags: review?(jmaher)
Comment 41•7 years ago
|
||
Backout by archaeopteryx@coole-files.de: https://hg.mozilla.org/integration/autoland/rev/705e1bc5b69b Backed out changeset 9f402afa3864 on wcosta's request. r=backout on a CLOSED TREE
Comment 42•7 years ago
|
||
Comment on attachment 8836135 [details] [diff] [review] use correct builder names for media tests. r=jmaher Review of attachment 8836135 [details] [diff] [review]: ----------------------------------------------------------------- everything else is great, just confused on the twitch/youtube thing. ::: taskcluster/taskgraph/transforms/tests.py @@ +1076,5 @@ > taskdesc['worker-type'] = 'buildbot-bridge/buildbot-bridge' > > + if test_name == 'media-twitch-tests': > + # twitch tests run on youtube builder > + test_name = 'media-youtube-tests' this is the only piece I don't understand.
Attachment #8836135 -
Flags: review?(jmaher) → review-
Assignee | ||
Updated•7 years ago
|
Attachment #8836093 -
Attachment is obsolete: true
Assignee | ||
Updated•7 years ago
|
Attachment #8836135 -
Attachment is obsolete: true
Comment 43•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/9f402afa3864
Updated•7 years ago
|
Summary: Switch macosx tests to buildbot-bridge → Switch macosx debug tests to buildbot-bridge
Comment hidden (mozreview-request) |
Comment 45•7 years ago
|
||
Pushed by wcosta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8121f087abc4 Switch macosx tests to buildbot-bridge. r=catlee,dustin
Comment 46•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/8121f087abc4
Updated•7 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Comment 47•6 years ago
|
||
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Updated•6 years ago
|
Product: TaskCluster → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•