Closed Bug 1423055 Opened 8 years ago Closed 8 years ago

disable some Buildbot testing so we can enable the full suite of tests on project-branches

Categories

(Release Engineering :: General, enhancement)

Product:
Release Engineering
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlund, Assigned: aselagea, Mentored)

Details

Attachments

(5 files)

graphics_tests.txt
55.22 KB, text/plain
Details
bb_jobs.txt
15.92 KB, text/plain
nthomas
: feedback+
jmaher
: feedback+
Details
bug1423055.patch
2.11 KB, patch
Callek
: review+
aselagea
: checked-in+
Details | Diff | Splinter Review
diff.txt
660.69 KB, text/plain
Details
remaining_tests.txt
51.10 KB, text/plain
Details
from callek: context: Would be awesome if we could disable a lot of BB testing so we can enable the full suite of tests on project-branches (e.g. graphics) so we can actually cause the decision task to fail if a BBB task is attempted when no BB job is configured. Motive: We hit “too many builders” when I tried to enable the full set of stuff on graphics/stuff, so we had to disable that error.
Attached file graphics_tests.txt —
We can help with this, but would need some additional info. These would be the current graphics tests present on the the bb-masters. As for the builds, we have only two of them: WINNT 5.2 graphics leak test build SigningScriptFactory WINNT 6.1 x86-64 graphics leak test build SigningScriptFactory I'd have two questions: 1. which of the above builds/tests would you like to remove? 2. does 1 apply for the other project branches as well?
Flags: needinfo?(bugspam.Callek)
In particular this issue is that there are too many test builders configured in buildbot in general. For example the suites defined here: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#559 we disable some here, and do similar stuff around there: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#3107 you can get the list of jobs that roughly should be supported on a branch with a gecko checkout on that branch using `./mach taskgraph full --json > /tmp/jobs.json` and then using gron or some other inspection tool to find the buildbot-bridge jobs. From my current local copy of m-c, this is what I see that still needs to be defined, but many of the tests in the configs can go. We likely need MOST of those tests for at least mozilla-esr52, but I bet 57 and newer can drop a large portion. If you are confused or need more info, lets meet via vidyo tomorrow morning. callek@Centaurus:~/mozilla/hg/mozilla-central$ for job in `grep workerType /tmp/jobs_test1.json | grep buildbot-bridge | sed 's/^json\[\"\(.*\)\"\].*/\1/' | sort -u`; do grep $job /tmp/jobs_test1.json ; done | grep task.payload.buildername | sed 's/^.*\= \"\(.*\)\".*/\1/' | sort -u Linux x86-64 mozilla-central periodic file update release-mozilla-central-fennec_bncr_sub release-mozilla-central-fennec_bouncer_aliases release-mozilla-central-fennec_mark_as_shipped release-mozilla-central-fennec_uptake_monitoring release-mozilla-central-fennec_version_bump Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g2-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g2-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g3-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g3-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g4-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g4-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g5-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos g5-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos h1-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos h2-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos other-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos other-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-singletons-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-singletons-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos speedometer-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos speedometer-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-stylo-threads-e10s Ubuntu HW 12.04 x64 mozilla-central talos chromez-e10s Ubuntu HW 12.04 x64 mozilla-central talos chromez-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs-e10s Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos g1-e10s Ubuntu HW 12.04 x64 mozilla-central talos g1-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos g2-e10s Ubuntu HW 12.04 x64 mozilla-central talos g2-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos g3-e10s Ubuntu HW 12.04 x64 mozilla-central talos g3-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos g4-e10s Ubuntu HW 12.04 x64 mozilla-central talos g4-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos g5-e10s Ubuntu HW 12.04 x64 mozilla-central talos g5-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos h1-e10s Ubuntu HW 12.04 x64 mozilla-central talos h2-e10s Ubuntu HW 12.04 x64 mozilla-central talos other-e10s Ubuntu HW 12.04 x64 mozilla-central talos other-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-e10s Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-singletons-e10s Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-singletons-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos speedometer-e10s Ubuntu HW 12.04 x64 mozilla-central talos speedometer-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos svgr-e10s Ubuntu HW 12.04 x64 mozilla-central talos svgr-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos tp5o-e10s Ubuntu HW 12.04 x64 mozilla-central talos tp5o-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos tp6-e10s Ubuntu HW 12.04 x64 mozilla-central talos tp6-stylo-disabled-e10s Ubuntu HW 12.04 x64 mozilla-central talos tp6-stylo-threads-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos chromez-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos dromaeojs-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos g1-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos g3-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos g4-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos g5-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos h1-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos h2-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos other-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos perf-reftest-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos perf-reftest-singletons-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos speedometer-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos svgr-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos tp5o-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos tp6-e10s Ubuntu HW 12.04 x64 qr mozilla-central talos tp6-stylo-threads-e10s Windows 10 64-bit Code Coverage mozilla-central debug test reftest-e10s-1 Windows 10 64-bit Code Coverage mozilla-central debug test reftest-e10s-2 Windows 10 64-bit Code Coverage mozilla-central debug test reftest-no-accel-e10s-1 Windows 10 64-bit Code Coverage mozilla-central debug test reftest-no-accel-e10s-2 Windows 10 64-bit DevEdition mozilla-central opt test reftest-e10s Windows 10 64-bit DevEdition mozilla-central opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-central debug test reftest-1 Windows 10 64-bit mozilla-central debug test reftest-2 Windows 10 64-bit mozilla-central debug test reftest-e10s-1 Windows 10 64-bit mozilla-central debug test reftest-e10s-2 Windows 10 64-bit mozilla-central debug test reftest-no-accel-1 Windows 10 64-bit mozilla-central debug test reftest-no-accel-2 Windows 10 64-bit mozilla-central debug test reftest-no-accel-e10s-1 Windows 10 64-bit mozilla-central debug test reftest-no-accel-e10s-2 Windows 10 64-bit mozilla-central opt test reftest-e10s Windows 10 64-bit mozilla-central opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-central pgo talos chromez-e10s Windows 10 64-bit mozilla-central pgo talos dromaeojs-e10s Windows 10 64-bit mozilla-central pgo talos g1-e10s Windows 10 64-bit mozilla-central pgo talos g2-e10s Windows 10 64-bit mozilla-central pgo talos g4-e10s Windows 10 64-bit mozilla-central pgo talos g5-e10s Windows 10 64-bit mozilla-central pgo talos h1-e10s Windows 10 64-bit mozilla-central pgo talos h2-e10s Windows 10 64-bit mozilla-central pgo talos other-e10s Windows 10 64-bit mozilla-central pgo talos perf-reftest-e10s Windows 10 64-bit mozilla-central pgo talos perf-reftest-singletons-e10s Windows 10 64-bit mozilla-central pgo talos speedometer-e10s Windows 10 64-bit mozilla-central pgo talos svgr-e10s Windows 10 64-bit mozilla-central pgo talos tp5o-e10s Windows 10 64-bit mozilla-central pgo talos tp6-e10s Windows 10 64-bit mozilla-central pgo talos tp6-stylo-threads-e10s Windows 10 64-bit mozilla-central pgo talos xperf-e10s Windows 10 64-bit mozilla-central pgo test reftest-e10s Windows 10 64-bit mozilla-central pgo test reftest-no-accel-e10s Windows 10 64-bit mozilla-central talos chromez-e10s Windows 10 64-bit mozilla-central talos dromaeojs-e10s Windows 10 64-bit mozilla-central talos g1-e10s Windows 10 64-bit mozilla-central talos g2-e10s Windows 10 64-bit mozilla-central talos g4-e10s Windows 10 64-bit mozilla-central talos g5-e10s Windows 10 64-bit mozilla-central talos h1-e10s Windows 10 64-bit mozilla-central talos h2-e10s Windows 10 64-bit mozilla-central talos other-e10s Windows 10 64-bit mozilla-central talos perf-reftest-e10s Windows 10 64-bit mozilla-central talos perf-reftest-singletons-e10s Windows 10 64-bit mozilla-central talos speedometer-e10s Windows 10 64-bit mozilla-central talos svgr-e10s Windows 10 64-bit mozilla-central talos tp5o-e10s Windows 10 64-bit mozilla-central talos tp6-e10s Windows 10 64-bit mozilla-central talos tp6-stylo-threads-e10s Windows 10 64-bit mozilla-central talos xperf-e10s Windows 7 32-bit mozilla-central pgo talos chromez-e10s Windows 7 32-bit mozilla-central pgo talos dromaeojs-e10s Windows 7 32-bit mozilla-central pgo talos g1-e10s Windows 7 32-bit mozilla-central pgo talos g2-e10s Windows 7 32-bit mozilla-central pgo talos g4-e10s Windows 7 32-bit mozilla-central pgo talos g5-e10s Windows 7 32-bit mozilla-central pgo talos h1-e10s Windows 7 32-bit mozilla-central pgo talos h2-e10s Windows 7 32-bit mozilla-central pgo talos other-e10s Windows 7 32-bit mozilla-central pgo talos perf-reftest-e10s Windows 7 32-bit mozilla-central pgo talos perf-reftest-singletons-e10s Windows 7 32-bit mozilla-central pgo talos speedometer-e10s Windows 7 32-bit mozilla-central pgo talos svgr-e10s Windows 7 32-bit mozilla-central pgo talos tp5o-e10s Windows 7 32-bit mozilla-central pgo talos tp6-e10s Windows 7 32-bit mozilla-central pgo talos tp6-stylo-threads-e10s Windows 7 32-bit mozilla-central pgo talos xperf-e10s Windows 7 32-bit mozilla-central talos chromez-e10s Windows 7 32-bit mozilla-central talos dromaeojs-e10s Windows 7 32-bit mozilla-central talos g1-e10s Windows 7 32-bit mozilla-central talos g2-e10s Windows 7 32-bit mozilla-central talos g4-e10s Windows 7 32-bit mozilla-central talos g5-e10s Windows 7 32-bit mozilla-central talos h1-e10s Windows 7 32-bit mozilla-central talos h2-e10s Windows 7 32-bit mozilla-central talos other-e10s Windows 7 32-bit mozilla-central talos perf-reftest-e10s Windows 7 32-bit mozilla-central talos perf-reftest-singletons-e10s Windows 7 32-bit mozilla-central talos speedometer-e10s Windows 7 32-bit mozilla-central talos svgr-e10s Windows 7 32-bit mozilla-central talos tp5o-e10s Windows 7 32-bit mozilla-central talos tp6-e10s Windows 7 32-bit mozilla-central talos tp6-stylo-threads-e10s Windows 7 32-bit mozilla-central talos xperf-e10s
Flags: needinfo?(bugspam.Callek)
Thanks Callek for the hint! So I did a similar thing for a local copy of m-i: e.g. "./mach taskgraph full --json > m-i.json" Then I used a little Python snippet to search for the jobs that run via BBB: " import json buildernames = [] data = json.load(open('m-i.json')) for key in data.keys(): if data[key]['task']['workerType']== 'buildbot-bridge': buildernames.append(data[key]['task']['payload']['buildername']) for build in sorted(buildernames): print build " I then copied the resulted entries to a local file. As far as I know, all the talos jobs run via BBB so we shouldn't touch them. Besides talos, we still have the following jobs running via BBB that we'll need to keep: $ cat m-i.json | grep -v "talos" Linux x86-64 mozilla-inbound periodic file update release-mozilla-inbound-fennec_bncr_sub release-mozilla-inbound-fennec_bouncer_aliases release-mozilla-inbound-fennec_mark_as_shipped release-mozilla-inbound-fennec_uptake_monitoring release-mozilla-inbound-fennec_version_bump Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-e10s-1 Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-e10s-2 Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-no-accel-e10s-1 Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-no-accel-e10s-2 Windows 10 64-bit DevEdition mozilla-inbound opt test reftest-e10s Windows 10 64-bit DevEdition mozilla-inbound opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-inbound debug test reftest-1 Windows 10 64-bit mozilla-inbound debug test reftest-2 Windows 10 64-bit mozilla-inbound debug test reftest-e10s-1 Windows 10 64-bit mozilla-inbound debug test reftest-e10s-2 Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-1 Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-2 Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-e10s-1 Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-e10s-2 Windows 10 64-bit mozilla-inbound opt test reftest-e10s Windows 10 64-bit mozilla-inbound opt test reftest-e10s Windows 10 64-bit mozilla-inbound opt test reftest-e10s Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s Windows 10 64-bit mozilla-inbound pgo test reftest-e10s Windows 10 64-bit mozilla-inbound pgo test reftest-no-accel-e10s
Attached file bb_jobs.txt —
I then checked Buildbot DB for the jobs that ran during the past month: e.g 'select buildername from buildrequests where submitted_at>1510012800 and complete=1;' Dumped the output into a file, then removed the following: - duplicate entries - talos jobs - already mentioned above why we still need them - ESR52 jobs - most of them run in Buildbot, so I think we shouldn't touch them for now - Try jobs - :kmoir worked in bug 1384706 to stop invoking buildbot jobs by default, yet we should still have a way of testing ESR/Release patches (available via "--buildot" in the TryChooser synthax), so I'd say we shouldn't touch Try as well - Thunderbird jobs e.g. 'cat bb_jobs.txt | sort -u | grep -Ev "talos"\|"esr52"\|"try"\|"comm"' The remaining jobs can be seen in this attachment. Based on that, I *think* we can do the following: 1. keep the talos jobs as they are now 2. don't touch ESR52 and Try 3. for Fx >= 57, drop *all* the tests except: - reftest - reftest-e10s - reftest-no-accel - reftest-no-accel-e10s - mochitest-clipboard - mochitest-clipboard-e10s - mochitest-media-e10s 4. leave the builders as they are - noticed that ReleaseDuty still uses some project branches (e.g. jamun) for repacks, update verify builds, etc. Looping in :nthomas and :jmaher here so that we can have a clear idea of what we still want to keep before proceeding with removing these BB jobs.
Attachment #8935439 - Flags: feedback?(nthomas)
Attachment #8935439 - Flags: feedback?(jmaher)
Attachment #8935439 - Flags: feedback?(bugspam.Callek)
thanks for the heads up, this looks reasonable for now, hopefully in due time we can get that list reduced.
(In reply to Alin Selagea [:aselagea][buildduty] from comment #4) > Based on that, I *think* we can do the following: > 1. keep the talos jobs as they are now > 2. don't touch ESR52 and Try > 3. for Fx >= 57, drop *all* the tests except: > - reftest > - reftest-e10s > - reftest-no-accel > - reftest-no-accel-e10s > - mochitest-clipboard > - mochitest-clipboard-e10s > - mochitest-media-e10s > 4. leave the builders as they are > - noticed that ReleaseDuty still uses some project branches (e.g. jamun) > for repacks, update verify builds, etc. That plan sounds good to me as well.
Your inspection methods seem fine to me. I defer to jmaher and Callek for details on the tests.
Attachment #8935439 - Flags: feedback?(nthomas) → feedback+
Attachment #8935439 - Flags: feedback?(jmaher) → feedback+
Attached patch bug1423055.patch — — Splinter Review
Attachment #8935772 - Flags: review?(bugspam.Callek)
Attached file diff.txt —
Test diff. (test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat old_sorted | wc -l 13650 (test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat new_sorted | wc -l 5072 That means 8578 tests were dropped.
Attached file remaining_tests.txt —
And here are the remaining tests (excluding talos, esr52 and try). (test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat new_sorted | grep -Ev talos\|esr52\|try | wc -l 721 I also ran 'test-masters.sh' script to confirm the patch is working fine on other masters, all tests passed.
Assignee: nobody → aselagea
Comment on attachment 8935772 [details] [diff] [review] bug1423055.patch Review of attachment 8935772 [details] [diff] [review]: ----------------------------------------------------------------- Lets do it, I think this is good based on joel's feedback earlier.
Attachment #8935772 - Flags: review?(bugspam.Callek) → review+
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Attachment #8935439 - Flags: feedback?(bugspam.Callek)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: