1531358 - [meta] ensure windows 10 ux machines running performance are only running on mozilla-central/try as tier-2

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P1)

Comment 1

[Greg Mierzwinski [:sparky]]

I think this is a duplicate of bug 1539856. It also refers to a platform specification which doesn't exist anymore because of bug 1548614.

:davehunt, :jmaher, the only thing that isn't done is that some Raptor performance tests are still running at tier 1 - should we reduce them to tier 2? https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=2%2C3%2C1&searchStr=windows10-64-ref&revision=28cf163158a673037d20ccc1aa7b825e406e927b

Comment 2

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

we should mark those as tier-2; do we need all the tests running on this platform? what are the goals of running tests on this hardware platform?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There's no context for the original bug. What's the reason for these being marked as tier 2? Should all tests (not just performance tests) running on this platform be tier 2? These results provide insight into performance on our reference hardware, but we're not sheriffing alerts on this platform. I would suggest running all benchmarks and at least a subset of cold page load tests, however we could run these on a daily cron instead of per push to mozilla-central.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Note that bug 1609295 will temporarily disable performance tests running on this platform.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

tier-2 as we have limited capacity. I want to review what we run on these in general so we can determine if we can get along with fewer machines. I agree core benchmarks and a representative sample of pageloads makes sense.

Comment 6

[Bob Clary [:bc] (inactive)]

searchfox:

• talos-webgl
• talos-g4-profiling
• talos-g4
• raptor job defaults
• taskcluster/taskgraph/transforms/tests.py set_worker_type
• WINDOWS_WORKER_TYPES 'windows10-64-ref-hw-2017'
• tools/tryselect/selectors/fuzzy.py TARGET_TASK_FILTERS

There are a scattering of Tier 1, 2, 3 windows10-64-ref-hw-2017 tasks.

The tier 2, 3 tasks running on mozilla-central are:

label	tier	run_on_projects
test-windows10-64-ref-hw-2017/opt-raptor-jetstream2-firefox-e10s	3	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-youtube-playback-firefox-e10s	2	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-talos-g4-e10s	2	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-talos-webgl-e10s	2	mozilla-central, try

The tier 1 windows10-64-ref-hw-2017 tasks are

label	run_on_projects
test-windows10-64-ref-hw-2017/opt-raptor-ares6-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-motionmark-animometer-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-motionmark-htmlsuite-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-speedometer-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-stylebench-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-sunspider-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-1-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-1-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-10-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-10-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-11-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-12-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-13-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-14-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-15-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-16-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-17-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-18-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-19-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-2-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-2-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-20-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-21-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-22-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-23-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-24-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-25-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-26-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-27-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-28-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-29-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-3-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-3-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-30-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-4-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-4-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-5-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-5-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-6-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-6-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-7-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-7-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-8-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-8-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-9-firefox-cold-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-9-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-tp6-binast-1-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-wasm-godot-firefox-e10s	mozilla-central, try
test-windows10-64-ref-hw-2017/opt-raptor-webaudio-firefox-e10s	mozilla-central, try

Do we need all of these? Could they be run with the Nightly schedule?

I'll wait to hear more about what is desired before proceeding further.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

we should really be running either tp6 or tp6-cold, that will reduce a lot of the jobs.

In addition there are many benchmarks that we run, I suspect we don't need them all.

We will want to use these machines to test Fission for relative perf impact- that is months out, so no time soon.

One option would be to run a reduced set of the above tests on the nightly scheduler (2x/day)- that would probably give us enough data to see trends and allow for us to have a small pool of machines (<8) generating results.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

I also think some tests will need to be added such as:

• talos sessionrestore
• talos ts_paint

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Okay, let's do the following:

1. Regrade all tests as tier 2
2. Stop running warm page load tests
3. Schedule all tests to run nightly
4. Review Talos tests that are not currently running

We can also consider a subset of page load tests if the load is still considered excessive after these changes. I'd prefer this to be a dynamically selected subset based on the value of the tests, otherwise we'll need to manually update this subset periodically.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

in bug 1609295 40 raptor tests were reduced, but we still run 10 raptor tests on this platform.

Yesterday bitbar reimaged the 16 machines, so we should have full capacity. Who will be doing the work in comment 9?

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #11)

in bug 1609295 40 raptor tests were reduced, but we still run 10 raptor tests on this platform.

Who will be doing the work in comment 9?

Setting as a P2 so this will be picked up by someone on the team this quarter.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #9)

Okay, let's do the following:

1. Regrade all tests as tier 2
2. Stop running warm page load tests
3. Schedule all tests to run nightly
4. Review Talos tests that are not currently running

We can also consider a subset of page load tests if the load is still considered excessive after these changes. I'd prefer this to be a dynamically selected subset based on the value of the tests, otherwise we'll need to manually update this subset periodically.

It looks like this didn't get done in Q1 as planned. Bebe can someone pick this up in Q3? Feel free to split any of the tasks out into separate bugs as this one was originally just for changing the tier, but I suspect we can take care of a few of them here. Now that we have bug 1591466 perhaps we can even extend this to only run these high-value tests on this platform.

Comment 14

[Florin Strugariu [:Bebe]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #9)

Okay, let's do the following:

1. Regrade all tests as tier 2
2. Stop running warm page load tests
3. Schedule all tests to run nightly
4. Review Talos tests that are not currently running

We can also consider a subset of page load tests if the load is still considered excessive after these changes. I'd prefer this to be a dynamically selected subset based on the value of the tests, otherwise we'll need to manually update this subset periodically.

This list is for test-windows10-64-ref-hw-2017 right? Are there any other hw instances we have to change?

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #14)

This list is for test-windows10-64-ref-hw-2017 right? Are there any other hw instances we have to change?

This bug only concerns the 2017 reference hardware.

Comment 16

[Florin Strugariu [:Bebe]]

Attachment: Bug 1531358 - [meta] ensure windows 10 ux machines running performance are only running on mozilla-central/try as tier-2

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c5a1307dc658
[meta] ensure windows 10 ux machines running performance are only running on mozilla-central/try as tier-2 r=perftest-reviewers,sparky

Comment 18

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/c5a1307dc658

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #14)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #9)

Okay, let's do the following:

1. Regrade all tests as tier 2
2. Stop running warm page load tests
3. Schedule all tests to run nightly
4. Review Talos tests that are not currently running

We can also consider a subset of page load tests if the load is still considered excessive after these changes. I'd prefer this to be a dynamically selected subset based on the value of the tests, otherwise we'll need to manually update this subset periodically.

This list is for test-windows10-64-ref-hw-2017 right? Are there any other hw instances we have to change?

It looks like #1 is now done. :bebe could you file new bugs for #2-4?

Comment 20

[Florin Strugariu [:Bebe]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #19)

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #14)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #9)

Okay, let's do the following:

1. Regrade all tests as tier 2
2. Stop running warm page load tests
3. Schedule all tests to run nightly
4. Review Talos tests that are not currently running

We can also consider a subset of page load tests if the load is still considered excessive after these changes. I'd prefer this to be a dynamically selected subset based on the value of the tests, otherwise we'll need to manually update this subset periodically.

This list is for test-windows10-64-ref-hw-2017 right? Are there any other hw instances we have to change?

It looks like #1 is now done. :bebe could you file new bugs for #2-4?

where done in https://hg.mozilla.org/integration/autoland/rev/c5a1307dc658

1. Regrade all tests as tier 2
2. Stop running warm page load tests
   I have open Bug 1652438 - Schedule all tests to run nightly on test-windows10-64-ref-hw-2017