Closed Bug 1364157 Opened 7 years ago Closed 7 years ago

disable non-e10s talos tests on all platforms for Firefox 55+

Categories

(Release Engineering :: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

Attachments

(4 files)

given our load issues and the increase in new tests needing to run, we need to find machine time somewhere.  Since in <3 months we will not be running non-e10s talos tests, lets disable them now so we can reduce feedback time for developers on try, not randomize developers with non-e10s only small performance regressions, and allow for a focus on what we want to ship.
odd, I did some initial work here and on osx it ran all the tests as well as duplicate e10s tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e017d2c4d557dce3c4801a2c0ec4c9b86c081425

linux did the right thing.
what is more odd is that using the same patch I run the decision task locally and get the right set of jobs:
./mach taskgraph target -p ~/Downloads/parameters.yml
 0:02.06 Loading kinds
 0:02.38 Generating full task set
 0:02.70 Generated 5 tasks for kind android-stuff
 0:03.00 Generated 1 tasks for kind artifact-build
 0:03.39 Generated 48 tasks for kind build
 0:03.48 Generated 7 tasks for kind build-signing
 0:03.87 Generated 6 tasks for kind docker-image
 0:03.87 Generated 2 tasks for kind hazard
 0:03.95 Generated 3 tasks for kind l10n
 0:04.15 Generated 26 tasks for kind nightly-l10n
 0:04.27 Generated 26 tasks for kind nightly-l10n-signing
 0:04.32 Generated 1 tasks for kind push-apk-breakpoint
 0:04.42 Generated 1 tasks for kind repackage
 0:04.77 Generated 98 tasks for kind repackage-l10n
 0:04.92 Generated 10 tasks for kind source-test
 0:04.96 Generated 13 tasks for kind spidermonkey
 0:04.98 Generated 8 tasks for kind static-analysis
 0:05.06 Starting new HTTPS connection (1): s3.amazonaws.com
 0:24.76 Generated 2958 tasks for kind test
 0:25.46 Generated 14 tasks for kind toolchain
 0:25.49 Generated 9 tasks for kind upload-symbols
 0:25.49 Generated 1 tasks for kind valgrind
 0:25.52 Generated 1 tasks for kind beetmover-repackage
 0:26.48 Generated 385 tasks for kind beetmover-l10n
 0:26.50 Generated 6 tasks for kind beetmover
 0:27.25 Generated 392 tasks for kind balrog
 0:27.97 Generated 392 tasks for kind checksums-signing
 0:29.08 Generated 392 tasks for kind beetmover-checksums
 0:29.14 Generated 1 tasks for kind push-apk
 0:29.17 Generating full task graph
 0:29.27 Full task graph contains 4806 tasks and 7841 dependencies
 0:29.27 Generating target task set
 0:29.28 Filter filter_servo pruned 0 tasks (4806 remain)
 0:29.29 platform linux64 triggers ridealong builds linux64-l10n, sm-plain, sm-nonunified, sm-arm-sim, sm-arm64-sim, sm-compacting, sm-rootanalysis, sm-package, sm-tsan, sm-asan, sm-mozjs-sys, sm-msan, sm-fuzzing
 0:29.33 Filter filter_target_tasks pruned 4753 tasks (53 remain)
android-api-15-gradle-dependencies
android-checkstyle
android-findbugs
android-lint
android-test
build-linux64/opt
build-macosx64/opt
l10n-linux64-l10n/opt
source-test-mozharness
source-test-mozlint-eslint
source-test-mozlint-flake8
source-test-sphinx
source-test-webidl-test
source-test-wptlint-gecko
spidermonkey-sm-asan/opt
spidermonkey-sm-fuzzing/opt
spidermonkey-sm-msan/opt
spidermonkey-sm-package/opt
spidermonkey-sm-plain/opt
spidermonkey-sm-tsan/opt
test-linux64/opt-talos-chrome-e10s
test-linux64/opt-talos-dromaeojs-e10s
test-linux64/opt-talos-g1-e10s
test-linux64/opt-talos-g2-e10s
test-linux64/opt-talos-g3-e10s
test-linux64/opt-talos-g4-e10s
test-linux64/opt-talos-other-e10s
test-linux64/opt-talos-perf-reftest-e10s
test-linux64/opt-talos-svgr-e10s
test-linux64/opt-talos-tp5o-e10s
test-macosx64/opt-talos-chrome-e10s
test-macosx64/opt-talos-dromaeojs-e10s
test-macosx64/opt-talos-g1-e10s
test-macosx64/opt-talos-g2-e10s
test-macosx64/opt-talos-g4-e10s
test-macosx64/opt-talos-other-e10s
test-macosx64/opt-talos-perf-reftest-e10s
test-macosx64/opt-talos-svgr-e10s
test-macosx64/opt-talos-tp5o-e10s
toolchain-linux64-binutils/opt
toolchain-linux64-cctools-port/opt
toolchain-linux64-clang-tidy/opt
toolchain-linux64-clang/opt
toolchain-linux64-gcc/opt
toolchain-linux64-hfsplus/opt
toolchain-linux64-libdmg/opt
toolchain-macosx64-cctools-port/opt
toolchain-macosx64-clang-tidy/opt
toolchain-macosx64-clang/opt
toolchain-win32-clang-cl/opt
toolchain-win32-clang-tidy/opt
toolchain-win64-clang-cl/opt
toolchain-win64-clang-tidy/opt

and same for the decision task that ran:
https://public-artifacts.taskcluster.net/aq9EDP9tS8-8TakGMqxkyA/0/public/target-tasks.json

something seems wrong with taskcluster here.

:wcosta, do you know why in taskcluster we would schedule talos non-e10s jobs and a second talos-e10s job when the decision task doesn't request it...specifically for osx- maybe there is something special on try for osx for bbb + native worker?
Flags: needinfo?(wcosta)
(In reply to Joel Maher ( :jmaher) from comment #2)
> what is more odd is that using the same patch I run the decision task
> locally and get the right set of jobs:
> ./mach taskgraph target -p ~/Downloads/parameters.yml
>  0:02.06 Loading kinds
>  0:02.38 Generating full task set
>  0:02.70 Generated 5 tasks for kind android-stuff
>  0:03.00 Generated 1 tasks for kind artifact-build
>  0:03.39 Generated 48 tasks for kind build
>  0:03.48 Generated 7 tasks for kind build-signing
>  0:03.87 Generated 6 tasks for kind docker-image
>  0:03.87 Generated 2 tasks for kind hazard
>  0:03.95 Generated 3 tasks for kind l10n
>  0:04.15 Generated 26 tasks for kind nightly-l10n
>  0:04.27 Generated 26 tasks for kind nightly-l10n-signing
>  0:04.32 Generated 1 tasks for kind push-apk-breakpoint
>  0:04.42 Generated 1 tasks for kind repackage
>  0:04.77 Generated 98 tasks for kind repackage-l10n
>  0:04.92 Generated 10 tasks for kind source-test
>  0:04.96 Generated 13 tasks for kind spidermonkey
>  0:04.98 Generated 8 tasks for kind static-analysis
>  0:05.06 Starting new HTTPS connection (1): s3.amazonaws.com
>  0:24.76 Generated 2958 tasks for kind test
>  0:25.46 Generated 14 tasks for kind toolchain
>  0:25.49 Generated 9 tasks for kind upload-symbols
>  0:25.49 Generated 1 tasks for kind valgrind
>  0:25.52 Generated 1 tasks for kind beetmover-repackage
>  0:26.48 Generated 385 tasks for kind beetmover-l10n
>  0:26.50 Generated 6 tasks for kind beetmover
>  0:27.25 Generated 392 tasks for kind balrog
>  0:27.97 Generated 392 tasks for kind checksums-signing
>  0:29.08 Generated 392 tasks for kind beetmover-checksums
>  0:29.14 Generated 1 tasks for kind push-apk
>  0:29.17 Generating full task graph
>  0:29.27 Full task graph contains 4806 tasks and 7841 dependencies
>  0:29.27 Generating target task set
>  0:29.28 Filter filter_servo pruned 0 tasks (4806 remain)
>  0:29.29 platform linux64 triggers ridealong builds linux64-l10n, sm-plain,
> sm-nonunified, sm-arm-sim, sm-arm64-sim, sm-compacting, sm-rootanalysis,
> sm-package, sm-tsan, sm-asan, sm-mozjs-sys, sm-msan, sm-fuzzing
>  0:29.33 Filter filter_target_tasks pruned 4753 tasks (53 remain)
> android-api-15-gradle-dependencies
> android-checkstyle
> android-findbugs
> android-lint
> android-test
> build-linux64/opt
> build-macosx64/opt
> l10n-linux64-l10n/opt
> source-test-mozharness
> source-test-mozlint-eslint
> source-test-mozlint-flake8
> source-test-sphinx
> source-test-webidl-test
> source-test-wptlint-gecko
> spidermonkey-sm-asan/opt
> spidermonkey-sm-fuzzing/opt
> spidermonkey-sm-msan/opt
> spidermonkey-sm-package/opt
> spidermonkey-sm-plain/opt
> spidermonkey-sm-tsan/opt
> test-linux64/opt-talos-chrome-e10s
> test-linux64/opt-talos-dromaeojs-e10s
> test-linux64/opt-talos-g1-e10s
> test-linux64/opt-talos-g2-e10s
> test-linux64/opt-talos-g3-e10s
> test-linux64/opt-talos-g4-e10s
> test-linux64/opt-talos-other-e10s
> test-linux64/opt-talos-perf-reftest-e10s
> test-linux64/opt-talos-svgr-e10s
> test-linux64/opt-talos-tp5o-e10s
> test-macosx64/opt-talos-chrome-e10s
> test-macosx64/opt-talos-dromaeojs-e10s
> test-macosx64/opt-talos-g1-e10s
> test-macosx64/opt-talos-g2-e10s
> test-macosx64/opt-talos-g4-e10s
> test-macosx64/opt-talos-other-e10s
> test-macosx64/opt-talos-perf-reftest-e10s
> test-macosx64/opt-talos-svgr-e10s
> test-macosx64/opt-talos-tp5o-e10s
> toolchain-linux64-binutils/opt
> toolchain-linux64-cctools-port/opt
> toolchain-linux64-clang-tidy/opt
> toolchain-linux64-clang/opt
> toolchain-linux64-gcc/opt
> toolchain-linux64-hfsplus/opt
> toolchain-linux64-libdmg/opt
> toolchain-macosx64-cctools-port/opt
> toolchain-macosx64-clang-tidy/opt
> toolchain-macosx64-clang/opt
> toolchain-win32-clang-cl/opt
> toolchain-win32-clang-tidy/opt
> toolchain-win64-clang-cl/opt
> toolchain-win64-clang-tidy/opt
> 
> and same for the decision task that ran:
> https://public-artifacts.taskcluster.net/aq9EDP9tS8-8TakGMqxkyA/0/public/
> target-tasks.json
> 
> something seems wrong with taskcluster here.
> 
> :wcosta, do you know why in taskcluster we would schedule talos non-e10s
> jobs and a second talos-e10s job when the decision task doesn't request
> it...specifically for osx- maybe there is something special on try for osx
> for bbb + native worker?

From task generation POV it is seems alright, 303 :kmoir
Flags: needinfo?(wcosta) → needinfo?(kmoir)
I looked at it, from task graph generation it seems fine. Not sure why several non e10s mac jobs were triggered and there were duplicates.  The bbb should just trigger the jobs that were requested by tc.
Flags: needinfo?(kmoir)
it appears using --artifact for the osx builds triggers extra talos jobs:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=ad15a0450f0e4e4229a853ae9098b9ec66600613&tochange=5645ac93f0f9a59105ff7d86c154a6711b285948

Chris, do you have thoughts on why osx would schedule a full duplicate set of talos tests only when using --artifact?  Maybe we somehow post two messages that we are done?  Although I think that is odd since in my patch I disabled non-e10s talos and that still got scheduled once.
Flags: needinfo?(cmanchester)
after this we will need a buildbot patch for the windows jobs.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8867118 - Flags: review?(rwood)
Attachment #8867118 - Flags: review?(rwood) → review+
this turns off all jobs- we would need the taskcluster bits to land and merge before landing this.
Attachment #8867292 - Flags: review?(kmoir)
for reference to the patch to disable non-e10s.
Attachment #8867292 - Flags: review?(kmoir) → review+
The artifact build configs diverged at some point, it looks like we need to flip `enable_talos_sendchange` to false in testing/mozharness/configs/builds/releng_sub_mac_configs/64_artifact.py and/or remove these builders from buildbot so there's nothing to trigger there.
Flags: needinfo?(cmanchester)
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c92e76bad29b
disable non-e10s talos tests on all platforms for Firefox 55+; taskcluster patch. r=rwood
looking at the 64_artifacct.py files, I see that we have sendchange enabled for unittests and talos, but looking at the linux artifact config it is also enabled:
https://dxr.mozilla.org/mozilla-central/search?q=path%3Atesting%2Fmozharness+enable_talos_sendchange%27%3A+T&redirect=false

I wonder if we should set this to false for linux as well?  I see linux32 needs to be false and same with Android.

Chris- should I make linux32, linux64, mac, and android all false?
Flags: needinfo?(cmanchester)
Keywords: leave-open
Blocks: 1364941
https://hg.mozilla.org/build/buildbot-configs/rev/f51dfdef83c3
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
(In reply to Joel Maher ( :jmaher) from comment #11)
> looking at the 64_artifacct.py files, I see that we have sendchange enabled
> for unittests and talos, but looking at the linux artifact config it is also
> enabled:
> https://dxr.mozilla.org/mozilla-central/
> search?q=path%3Atesting%2Fmozharness+enable_talos_sendchange%27%3A+T&redirect
> =false
> 
> I wonder if we should set this to false for linux as well?  I see linux32
> needs to be false and same with Android.
> 
> Chris- should I make linux32, linux64, mac, and android all false?

I'm not entirely sure, but it seems like those would all be scheduled through TC now, so that sounds right.
Flags: needinfo?(cmanchester)
Blocks: 1365604
Comment on attachment 8868173 [details] [diff] [review]
remove talos_sendchange from configurations where it is not needed

Review of attachment 8868173 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok to me. I think removing the entries from the configs would have the same effect.
Attachment #8868173 - Flags: review?(cmanchester) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3f87d6c1d378
do not run talos via sendchange unnecessarily. r=chmanchester
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: