Closed Bug 1332727 Opened 7 years ago Closed 7 years ago

Run mochitest-browser-screenshots on taskcluster

Categories

(Firefox Build System :: Task Configuration, task)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla54

People

(Reporter: MattN, Assigned: MattN)

References

(
URL
)

Details

Attachments

(1 file)

Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).
59 bytes, text/x-review-board-request
jmaher
: review+
Details 
For whatever reason mochitest-browser-screenshots isn't running on taskcluster yet. It's just a variant of mochitest-browser-chrome so should be easy to get running I would think. It needs blobber support to upload screenshots but I'm hoping that just works.

It should run on the same configurations as on buildbot[1]. Currently it's try and m-c only and not on builds that don't change visuals such as ASAN[2]

[1] https://dxr.mozilla.org/build-central/search?q=screenshots+path%3Abuildbot-configs%2Fmozilla-tests%2Fconfig.py&redirect=false
[2] https://hg.mozilla.org/build/buildbot-configs/annotate/6ca3cd6cefdb371790bad227995dc8c84fc379b4/mozilla-tests/config.py#l2625
Since bug 1330680 we're not getting any screenshots on linux.
Blocks: 1330680
Severity: enhancement → major
Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff250e0d7c33ee7a15ddf382d8017a6b6ecf5bf4
Assignee: nobody → MattN+bmo
Status: NEW → ASSIGNED
Comment on attachment 8829199 [details]
Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).

https://reviewboard.mozilla.org/r/106352/#review107296

::: taskcluster/ci/test/test-platforms.yml:110
(Diff revision 2)
>      test-sets:
>          - windows-vm-tests

I didn't realize that we had Windows 10 running on TC now. I should capture images there too so I'll give that a try.

::: taskcluster/ci/test/tests.yml:441
(Diff revision 2)
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']
> +            default: []

How can I make it so that the TC screenshot jobs run by default for m-c linux builds but available to schedule from TH (add new job) for other platforms and repos? Is that possible?

::: taskcluster/ci/test/tests.yml:443
(Diff revision 2)
> +    suite: mochitest/browser-chrome-screenshots
> +    treeherder-symbol: tc-M(ss)
> +    loopback-video: true
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']

I'm not sure if we need to get screenshots for linux32 and linux64 since they appear to use the same distro and theme on TC but there is a difference in border colours which is unusual:
https://public-artifacts.taskcluster.net/a-wRmkm9Qraax6doz1XAoA/0/public/test_info//20170122061158-controlCenter_01_noLWT_about.png
https://public-artifacts.taskcluster.net/a-mdhBe-SvmaouvRuxI9JQ/0/public/test_info//20170122061158-controlCenter_01_noLWT_about.png

::: taskcluster/ci/test/tests.yml:445
(Diff revision 2)
> +    loopback-video: true
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']
> +            default: []
> +    e10s: true

I need to make sure this works properly still. We only need screenshots of e10s on or off so I went with on since it's the future.

::: taskcluster/ci/test/tests.yml:462
(Diff revision 2)
> +    instance-size: legacy
> +    allow-software-gl-layers: false

I just copied these from m-bc
Comment on attachment 8829199 [details]
Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).

https://reviewboard.mozilla.org/r/106352/#review107302

a few pending issues, even ones that you raised.

::: taskcluster/ci/test/test-platforms.yml:27
(Diff revision 2)
>  linux32/opt:
>      build-platform: linux/opt
>      test-sets:
>          -  linux32-tests
>          -  linux32-opt-tests
> +        -  desktop-screenshot-capture

does linux32 make a difference here?  We have been trending to do less linux32 work vs more.

::: taskcluster/ci/test/tests.yml:441
(Diff revision 2)
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']
> +            default: []

right now with this patch these are run by default for every m-c and try push.  I would verify pgo vs opt do what you want.

Regarding other platforms, you would need to add the test set as an option for other platforms, then I believe (not 100% sure) you could add other platforms via 'add new job' button.

::: taskcluster/ci/test/tests.yml:443
(Diff revision 2)
> +    suite: mochitest/browser-chrome-screenshots
> +    treeherder-symbol: tc-M(ss)
> +    loopback-video: true
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']

this statement will run screenshots on all try pushes and all mozilla-central pushes.  I think you just want 'try' here?  I believe this week we should have a working example of the new cron.yml feature for scheduled runs of tests which then we could do this in 'mozilla-central' properly.

::: taskcluster/ci/test/tests.yml:445
(Diff revision 2)
> +    loopback-video: true
> +    run-on-projects:
> +        by-test-platform:
> +            linux.*: ['mozilla-central', 'try']
> +            default: []
> +    e10s: true

right now this ends up scheduling both on e10s and non e10s, I agree that e10s only is the future.

::: taskcluster/ci/test/tests.yml:462
(Diff revision 2)
> +    instance-size: legacy
> +    allow-software-gl-layers: false

that is fine, some day we will resolve the instance-size issue, but everything else is in place to make browser-chrome tests run smoothly
Attachment #8829199 - Flags: review?(jmaher) → review-
Comment on attachment 8829199 [details]
Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).

https://reviewboard.mozilla.org/r/106352/#review107302

> right now with this patch these are run by default for every m-c and try push.  I would verify pgo vs opt do what you want.
> 
> Regarding other platforms, you would need to add the test set as an option for other platforms, then I believe (not 100% sure) you could add other platforms via 'add new job' button.

> right now with this patch these are run by default for every m-c and try push.

That's what I'd like now for m-c (see bug 1332760). Things have been working well enough on Nightlies so in order to become tier 2 and for the other reasons in that bug I'm moving to run on every central push.

For try, the syntax is still taken into account, right? We already handle this in m-c by not capturing any screenshots if the developer didn't specify the env. var in the try syntax so the job appears on TH but we try to do as little work as possible.

> I would verify pgo vs opt do what you want.

On try with this patch it didn't add a PGO task so I think it's doing the correct thing. If PGO is periodic then based on what I said above I think opt is best.

> Regarding other platforms, you would need to add the test set as an option for other platforms, then I believe (not 100% sure) you could add other platforms via 'add new job' button.

> this statement will run screenshots on all try pushes and all mozilla-central pushes.  I think you just want 'try' here?  I believe this week we should have a working example of the new cron.yml feature for scheduled runs of tests which then we could do this in 'mozilla-central' properly.

See above

> right now this ends up scheduling both on e10s and non e10s, I agree that e10s only is the future.

Are you sure? I thought 'both' was for both. When I look at the task graph I only see e10s with 'true'.
Comment on attachment 8829199 [details]
Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).

https://reviewboard.mozilla.org/r/106352/#review107310

::: taskcluster/ci/test/test-platforms.yml:107
(Diff revision 3)
>  # win64 vm
>  windows10-64-vm/debug:
>      build-platform: win64/debug
>      test-sets:
>          - windows-vm-tests
>  windows10-64-vm/opt:

Thanks for the quick review.

I did a try push with Windows 10 as well and it captured all the images but all of them were solid black. Maybe it's not setup for loopback video or something? I'll move that to a follow-up
Blocks: 1332945
Comment on attachment 8829199 [details]
Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt).

https://reviewboard.mozilla.org/r/106352/#review107516

thanks for the explanation.  A couple of things:
1) I believe these will always be scheduled on try, possibly we can live with that until there is a better solution for tasks which require custom try syntax?
2) windows 10- that is very experimental right now, I will make a note to validate screenshots on failure are real before calling the platform ready for testing
Attachment #8829199 - Flags: review?(jmaher) → review+
(In reply to Matthew N. [:MattN] (PM me if requests are blocking you) from comment #7)
> Are you sure? I thought 'both' was for both. When I look at the task graph I
> only see e10s with 'true'.

You are correct -- `e10s: true` will schedule only one test run, with e10s enabled.
(In reply to Joel Maher ( :jmaher) from comment #10)
> Comment on attachment 8829199 [details]
> Bug 1332727 - Run mochitest-browser-screenshots on taskcluster (by default
> on linux64/opt).
> 
> https://reviewboard.mozilla.org/r/106352/#review107516
> 
> thanks for the explanation.  A couple of things:
> 1) I believe these will always be scheduled on try, possibly we can live
> with that until there is a better solution for tasks which require custom
> try syntax?

Do you mean always regardless of try syntax or you mean always when -u "all", "mochitests", or "mochitest-browser-chrome" are requested (like the current BB job)?

> 2) windows 10- that is very experimental right now, I will make a note to
> validate screenshots on failure are real before calling the platform ready
> for testing

Thanks. I filed bug 1332945 for screenshots on Win10.
Pushed by mozilla@noorenberghe.ca:
https://hg.mozilla.org/integration/autoland/rev/fffe9c114f88
Run mochitest-browser-screenshots on taskcluster (by default on linux64/opt). r=jmaher
I believe |-u all| will pickup the screenshots, that is probably ok, people shouldn't be using that too much :)
https://hg.mozilla.org/mozilla-central/rev/fffe9c114f88
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: