1342825 - Don't run mochitest browser screenshots on PGO builds

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Matthew N. [:MattN]]

M(ss) jobs are running both on opt and PGO but the screenshots should be the same so this isn't necessary. Is there an easy way to disable running the job on PGO for both buildbot and TC? If there isn't an easy solution in TC and/or BB then we can do this in the browser.ini but there will still be wasted resources from starting the job and then doing nothing.

Taskcluster config: https://dxr.mozilla.org/mozilla-central/rev/a08ec245fa24d573fc99e81210ecc09de734cdd3/taskcluster/ci/test/tests.yml#501
Buildbot config: https://hg.mozilla.org/build/buildbot-configs/annotate/d816f8ed797e49d73755ddcf16f616b153f30ffe/mozilla-tests/config.py#l407

Comment 1

[Kim Moir [:kmoir] ET]

So looking at treeherder, these jobs seem to be limited to running on linux64 on m-c and try via taskcluster.  They only seem to run on e10s on linux64 opt and linux64 nightly opt

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-job_type_symbol=ss

Looking at taskcluster/ci/test/tests.yml 

they are defined like this

mochitest-browser-screenshots:
 531     description: "Mochitest Browser Screenshots"
 532     suite: mochitest/browser-chrome-screenshots
 533     treeherder-symbol: tc-M(ss)
 534     loopback-video: true
 535     run-on-projects:
 536         by-test-platform:
 537             linux64/opt: ['mozilla-central', 'try']
 538             default: []
 539     e10s: true
 540     max-run-time: 3600
 541     mozharness:
 542         mochitest-flavor: browser
 543         script: desktop_unittest.py
 544         no-read-buildbot-config: true
 545         config:
 546             by-test-platform:
 547                 windows.*:
 548                     - unittests/win_taskcluster_unittest.py
 549                 macosx.*:
 550                     - remove_executables.py
 551                     - unittests/mac_unittest.py
 552                 linux.*:
 553                     - unittests/linux_unittest.py
 554                     - remove_executables.py
 555         extra-options:
 556             - --mochitest-suite=browser-chrome-screenshots
 557     instance-size: legacy
 558     allow-software-gl-layers: false


I don't see them running on buildbot, perhaps the buildbot code that references them can be deleted as a cleanup.

I don't see the jobs running on both pgo and opt builds now, perhaps this changed recently?

Comment 2

[Kim Moir [:kmoir] ET]

Sebastian

From irc

kmoir: it seems they are being scheduled...      http://buildbot-master109.bb.releng.scl3.mozilla.com:8201/builders/Windows%207%2032-bit%20mozilla-central%20pgo%20test%20mochitest-browser-screenshots/builds/85

I don't know how to turn off pgo tests without turning off the associated opt tests.  I looked it it briefly but you could investigate more.  If these windows tests aren't showing up in treeherder, are they needed? Perhaps ask the original requestor on this bug.

Comment 3

[Sebastian Pacurar [:spacurar]]

I noticed these tests are indeed running in buildbot, in Tier 3. I will continue to investigate this issue.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=f9362554866b327700c7f9b18050d7b7eb3d2b23&filter-searchStr=screenshots&filter-tier=1&filter-tier=2&filter-tier=3

Comment 4

[Kim Moir [:kmoir] ET]

If they are tier3 and hidden, is someone actually looking at the results?  Will they need to be migrated to taskcluster eventually or can they be turned off?

Comment 5

[Matthew N. [:MattN]]

> run on e10s on linux64 opt and linux64 nightly opt

If that means that there are two sets of linux64 opt screenshots for the same revision can we disable the "linux64 nightly opt"?

(In reply to Kim Moir [:kmoir] from comment #4)
> If they are tier3 and hidden, is someone actually looking at the results? 
> Will they need to be migrated to taskcluster eventually or can they be
> turned off?

The job isn't eligible for higher tiers because it's not a pass/fail job, it simply takes screenshots and uploads them. The results are processed hourly and changes are sent to https://www.mozilla.org/en-US/about/forums/#dev-ui-alerts. They need to continue to run on all opt platforms regardless of BB or TC but we don't need screenshots from both of those for the same platform.

See also https://developer.mozilla.org/en-US/docs/Mozilla/QA/Browser_screenshots

Comment 6

[Sebastian Pacurar [:spacurar]]

This would be an example of parameter.yml:
https://public-artifacts.taskcluster.net/KsXhsnIXQkKleP6gVCJRgA/0/public/parameters.yml


And these are the commands I tried, all refering to the same parameters.yml file:

./mach taskgraph tasks --json -p parameters.yml > old-tasks.json
./mach taskgraph tasks --json -p url/to/parameters.yml > old-tasks.json
./mach taskgraph tasks --json -p task-id=<task-id> > old-tasks.json

Comment 7

[Kim Moir [:kmoir] ET]

Dustin landed a patch yesterday which requires some new values to the parameters.yml

such as

include_nightly: false

You can look at the the decision graph associated with a recent job on inbound and at the paramaters.yml

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=84020595

https://queue.taskcluster.net/v1/task/etgzxEVDSlWz_kwzSis_jQ/runs/0/artifacts/public/parameters.yml

Comment 8

[Sebastian Pacurar [:spacurar]]

Looks like it's working now. I don't receive that exception when I generate the graphs, after I did a pull and update from hg.

Comment 9

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1342825_linux64-nightly-opt.patch

Comment 10

[Sebastian Pacurar [:spacurar]]

Attachment: diff_linux64-nightly-opt

Diff resulting from applying the above patch. As it can be noticed, test-linux64-nightly/opt-mochitest-browser-screenshots-e10s is being removed.

Comment 11

[Matthew N. [:MattN]]

I guess that patch is just for TC, I hope that BB will be handled too.

I'm fine switching jobs from BB to TC but I believe screenshots don't work on some platforms over TC yet so those would have to remain in BB for now.

Comment 12

[Kim Moir [:kmoir] ET]

Comment on attachment 8847671 [details] [diff] [review]
bug_1342825_linux64-nightly-opt.patch

Like the other comment, I think there is a patch required for buildbot too

Comment 13

[Alin Selagea [:aselagea]]

Did some investigation here along Sebastian and found the following:

BB mochitest-browser-screenshots tests are enabled on both m-c and try here:
https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#2667

However, the PGO variant only runs on m-c and it's disabled on try:
https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#2402
https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#2444

It seems the 'pgo_strategy' parameter is then referenced in misc.py:
https://dxr.mozilla.org/build-central/source/buildbotcustom/misc.py#809 
meaning that we most likely cannot tell buildbot to do/not to do PGO for a particular test suite.

Taking a look at a buildbot-master, there are only two such PGO tests currently running in BB:
Windows 8 64-bit mozilla-central pgo test mochitest-browser-screenshots ScriptFactory
Windows 7 32-bit mozilla-central pgo test mochitest-browser-screenshots ScriptFactory

Not sure what we could do about those though given the above stats.

Comment 14

[Alin Selagea [:aselagea]]

@Matthew: how would a convenient solution look in this case?

Sebastian already provided a patch to remove "linux64-nightly opt" variant in TaskCluster (not landed yet).
As for Buildbot, I don't see an easy way to remove the pgo variant of mochitest-browser-screenshots.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5402e5e97e21
don't run mochitest browser screenshots on linux64-nightly opt, r=kmoir

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/5402e5e97e21