Closed Bug 1242103 Opened 7 years ago Closed 7 years ago

Add a job to run mochitest-browser-chrome "screenshots" subsuite on Nightlies

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: MattN, Assigned: MattN)

References

(Depends on 1 open bug)

Details

(Whiteboard: [browser-chrome-screenshots])

Attachments

(2 files)

In bug 1231408 I added a mochitest-browser-screenshots subsuite to take screenshots of the browser chrome to detect regressions for yosemite only for testing. 

Since things seem to be working I'd like to get this running on m-c (Nightlies) too and on all desktop opt platforms.

Since my understanding is that test jobs can't be run just for Nightlies on m-c, bug 1242101 skips capturing screenshots within the test suite itself if the update channel isn't "nightly".
Builders added:
+ Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-screenshots
+ Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-browser-screenshots
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central pgo test mochitest-browser-screenshots
+ Ubuntu VM 12.04 try opt test mochitest-browser-screenshots
+ Windows 10 64-bit try opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 7 32-bit try opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 8 64-bit try opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows XP 32-bit try opt test mochitest-browser-screenshots

The main unknown for me now is PGO. Are nightlies (channel=nightly) PGO? Do we do more than one channel=nightly build on a normal day per platform?
Comment on attachment 8711275 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots test configs for Windows and Linux. r=armenzg

https://reviewboard.mozilla.org/r/32057/#review28907
Attachment #8711275 - Flags: review?(armenzg) → review+
Nightly builds are PGO for Linux and Windows (it does not apply for Mac)
PGO builds on integration branches happen on a schedule.
All builds on release branches (including central) are PGO (where it applies).

We sometimes do more than one nightly a day.
(In reply to Matthew N. [:MattN] from comment #0)
> In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> screenshots of the browser chrome to detect regressions for yosemite only
> for testing. 
> 
> Since things seem to be working I'd like to get this running on m-c
> (Nightlies) too and on all desktop opt platforms.
> 
> Since my understanding is that test jobs can't be run just for Nightlies on
> m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> the update channel isn't "nightly".

Let me make sure I understand before I address the review.

Do you want this subsuite to run on every push of mozilla-central? Or just after a nightly build?

I don't know if we can *only* enable tests for nightly builds but if we could, would you prefer that?
I think every build on m-c has *nightly* as its update channel. I could be wrong though.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #7)
> I think every build on m-c has *nightly* as its update channel. I could be
> wrong though.

No, a regular m-c build on a push has app.update.channel = "default", I just double-checked.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #5)
> All builds on release branches (including central) are PGO (where it
> applies).

I see regular "opt" builds at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.

> We sometimes do more than one nightly a day.

Sure
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6)
> (In reply to Matthew N. [:MattN] from comment #0)
> > In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> > screenshots of the browser chrome to detect regressions for yosemite only
> > for testing. 
> > 
> > Since things seem to be working I'd like to get this running on m-c
> > (Nightlies) too and on all desktop opt platforms.
> > 
> > Since my understanding is that test jobs can't be run just for Nightlies on
> > m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> > the update channel isn't "nightly".
> 
> Let me make sure I understand before I address the review.
> 
> Do you want this subsuite to run on every push of mozilla-central? Or just
> after a nightly build?

Ideally I want this subsuite to only run on a Nightly build but since that's not currently possible, I made the test quit right away if it's not a nightly (by looking at the update channel).
 
> I don't know if we can *only* enable tests for nightly builds but if we
> could, would you prefer that?

Yes I would prefer that but if it's not trivial then I think the current solution is fine.
Keywords: leave-open
Attachment #8711280 - Flags: review?(armenzg) → review+
Comment on attachment 8711280 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots to m-c and other try. r=armenzg

https://reviewboard.mozilla.org/r/32061/#review29185
(In reply to Matthew N. [:MattN] from comment #9)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #5)
> > All builds on release branches (including central) are PGO (where it
> > applies).
> 
> I see regular "opt" builds at
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.
> 
That's news to me! :)


Could you please add a note about preferring to run on nightly if possible and refer back to this bug?
It would be ideal if one day we could *only* run it on nightly.
When it occurs to someone that "omg, nightly-only, that means it's going to be hidden and then broken and then be a waste of resources!!1!," MattN and I talked about it, and he's fine with it winding up tier-3, and his script making use of the output will let him know when someone breaks it and he has to sort through a day or two of pushes to both m-i and f-t to figure out who it was that broke him.
https://hg.mozilla.org/build/buildbot-configs/rev/63614b6564ee
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Keywords: leave-open
Resolution: --- → FIXED
Marked these as tier 2 for now until bug 1244199 gets deployed to Treeherder to let us mark buildbot jobs as tier 3.
Depends on: 1275549
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.