1242103 - Add a job to run mochitest-browser-chrome "screenshots" subsuite on Nightlies

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Matthew N. [:MattN]]

In bug 1231408 I added a mochitest-browser-screenshots subsuite to take screenshots of the browser chrome to detect regressions for yosemite only for testing. 

Since things seem to be working I'd like to get this running on m-c (Nightlies) too and on all desktop opt platforms.

Since my understanding is that test jobs can't be run just for Nightlies on m-c, bug 1242101 skips capturing screenshots within the test suite itself if the update channel isn't "nightly".

Comment 1

[Matthew N. [:MattN]]

Attachment: MozReview Request: Bug 1242103 - Add browser-chrome-screenshots test configs for Windows and Linux. r=armenzg

Review commit: https://reviewboard.mozilla.org/r/32057/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32057/

Comment 2

[Matthew N. [:MattN]]

Attachment: MozReview Request: Bug 1242103 - Add browser-chrome-screenshots to m-c and other try. r=armenzg

Review commit: https://reviewboard.mozilla.org/r/32061/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32061/

Comment 3

[Matthew N. [:MattN]]

Builders added:
+ Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-screenshots
+ Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-browser-screenshots
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central pgo test mochitest-browser-screenshots
+ Ubuntu VM 12.04 try opt test mochitest-browser-screenshots
+ Windows 10 64-bit try opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 7 32-bit try opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 8 64-bit try opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows XP 32-bit try opt test mochitest-browser-screenshots

The main unknown for me now is PGO. Are nightlies (channel=nightly) PGO? Do we do more than one channel=nightly build on a normal day per platform?

Comment 4

[Armen [:armenzg]]

Comment on attachment 8711275 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots test configs for Windows and Linux. r=armenzg

https://reviewboard.mozilla.org/r/32057/#review28907

Comment 5

[Armen [:armenzg]]

Nightly builds are PGO for Linux and Windows (it does not apply for Mac)
PGO builds on integration branches happen on a schedule.
All builds on release branches (including central) are PGO (where it applies).

We sometimes do more than one nightly a day.

Comment 6

[Armen [:armenzg]]

(In reply to Matthew N. [:MattN] from comment #0)
> In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> screenshots of the browser chrome to detect regressions for yosemite only
> for testing. 
> 
> Since things seem to be working I'd like to get this running on m-c
> (Nightlies) too and on all desktop opt platforms.
> 
> Since my understanding is that test jobs can't be run just for Nightlies on
> m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> the update channel isn't "nightly".

Let me make sure I understand before I address the review.

Do you want this subsuite to run on every push of mozilla-central? Or just after a nightly build?

I don't know if we can *only* enable tests for nightly builds but if we could, would you prefer that?

Comment 7

[Armen [:armenzg]]

I think every build on m-c has *nightly* as its update channel. I could be wrong though.

Comment 8

[Matthew N. [:MattN]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #7)
> I think every build on m-c has *nightly* as its update channel. I could be
> wrong though.

No, a regular m-c build on a push has app.update.channel = "default", I just double-checked.

Comment 9

[Matthew N. [:MattN]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #5)
> All builds on release branches (including central) are PGO (where it
> applies).

I see regular "opt" builds at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.

> We sometimes do more than one nightly a day.

Sure

Comment 10

[Matthew N. [:MattN]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6)
> (In reply to Matthew N. [:MattN] from comment #0)
> > In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> > screenshots of the browser chrome to detect regressions for yosemite only
> > for testing. 
> > 
> > Since things seem to be working I'd like to get this running on m-c
> > (Nightlies) too and on all desktop opt platforms.
> > 
> > Since my understanding is that test jobs can't be run just for Nightlies on
> > m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> > the update channel isn't "nightly".
> 
> Let me make sure I understand before I address the review.
> 
> Do you want this subsuite to run on every push of mozilla-central? Or just
> after a nightly build?

Ideally I want this subsuite to only run on a Nightly build but since that's not currently possible, I made the test quit right away if it's not a nightly (by looking at the update channel).
 
> I don't know if we can *only* enable tests for nightly builds but if we
> could, would you prefer that?

Yes I would prefer that but if it's not trivial then I think the current solution is fine.

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/7eb0141079d3

Comment 12

[Armen [:armenzg]]

Comment on attachment 8711280 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots to m-c and other try. r=armenzg

https://reviewboard.mozilla.org/r/32061/#review29185

Comment 13

[Armen [:armenzg]]

(In reply to Matthew N. [:MattN] from comment #9)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #5)
> > All builds on release branches (including central) are PGO (where it
> > applies).
> 
> I see regular "opt" builds at
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.
> 
That's news to me! :)


Could you please add a note about preferring to run on nightly if possible and refer back to this bug?
It would be ideal if one day we could *only* run it on nightly.

Comment 14

[Phil Ringnalda (:philor)]

When it occurs to someone that "omg, nightly-only, that means it's going to be hidden and then broken and then be a waste of resources!!1!," MattN and I talked about it, and he's fine with it winding up tier-3, and his script making use of the output will let him know when someone breaks it and he has to sort through a day or two of pushes to both m-i and f-t to figure out who it was that broke him.

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7eb0141079d3

Comment 16

[Matthew N. [:MattN]]

https://hg.mozilla.org/build/buildbot-configs/rev/63614b6564ee

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Marked these as tier 2 for now until bug 1244199 gets deployed to Treeherder to let us mark buildbot jobs as tier 3.