Closed Bug 1242103 Opened 9 years ago Closed 9 years ago

Add a job to run mochitest-browser-chrome "screenshots" subsuite on Nightlies

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: MattN, Assigned: MattN)

References

(Depends on 1 open bug)

Details

(Whiteboard: [browser-chrome-screenshots])

Attachments

(2 files)

In bug 1231408 I added a mochitest-browser-screenshots subsuite to take screenshots of the browser chrome to detect regressions for yosemite only for testing. Since things seem to be working I'd like to get this running on m-c (Nightlies) too and on all desktop opt platforms. Since my understanding is that test jobs can't be run just for Nightlies on m-c, bug 1242101 skips capturing screenshots within the test suite itself if the update channel isn't "nightly".
Builders added: + Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-screenshots + Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-browser-screenshots + Rev7 MacOSX Yosemite 10.10.5 mozilla-central opt test mochitest-browser-screenshots + Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-screenshots + Ubuntu VM 12.04 mozilla-central pgo test mochitest-browser-screenshots + Ubuntu VM 12.04 try opt test mochitest-browser-screenshots + Windows 10 64-bit try opt test mochitest-browser-screenshots + Windows 7 32-bit mozilla-central opt test mochitest-browser-screenshots + Windows 7 32-bit mozilla-central pgo test mochitest-browser-screenshots + Windows 7 32-bit try opt test mochitest-browser-screenshots + Windows 8 64-bit mozilla-central opt test mochitest-browser-screenshots + Windows 8 64-bit mozilla-central pgo test mochitest-browser-screenshots + Windows 8 64-bit try opt test mochitest-browser-screenshots + Windows XP 32-bit mozilla-central opt test mochitest-browser-screenshots + Windows XP 32-bit mozilla-central pgo test mochitest-browser-screenshots + Windows XP 32-bit try opt test mochitest-browser-screenshots The main unknown for me now is PGO. Are nightlies (channel=nightly) PGO? Do we do more than one channel=nightly build on a normal day per platform?
Comment on attachment 8711275 [details] MozReview Request: Bug 1242103 - Add browser-chrome-screenshots test configs for Windows and Linux. r=armenzg https://reviewboard.mozilla.org/r/32057/#review28907
Attachment #8711275 - Flags: review?(armenzg) → review+
Nightly builds are PGO for Linux and Windows (it does not apply for Mac) PGO builds on integration branches happen on a schedule. All builds on release branches (including central) are PGO (where it applies). We sometimes do more than one nightly a day.
(In reply to Matthew N. [:MattN] from comment #0) > In bug 1231408 I added a mochitest-browser-screenshots subsuite to take > screenshots of the browser chrome to detect regressions for yosemite only > for testing. > > Since things seem to be working I'd like to get this running on m-c > (Nightlies) too and on all desktop opt platforms. > > Since my understanding is that test jobs can't be run just for Nightlies on > m-c, bug 1242101 skips capturing screenshots within the test suite itself if > the update channel isn't "nightly". Let me make sure I understand before I address the review. Do you want this subsuite to run on every push of mozilla-central? Or just after a nightly build? I don't know if we can *only* enable tests for nightly builds but if we could, would you prefer that?
I think every build on m-c has *nightly* as its update channel. I could be wrong though.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #7) > I think every build on m-c has *nightly* as its update channel. I could be > wrong though. No, a regular m-c build on a push has app.update.channel = "default", I just double-checked.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #5) > All builds on release branches (including central) are PGO (where it > applies). I see regular "opt" builds at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though. > We sometimes do more than one nightly a day. Sure
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6) > (In reply to Matthew N. [:MattN] from comment #0) > > In bug 1231408 I added a mochitest-browser-screenshots subsuite to take > > screenshots of the browser chrome to detect regressions for yosemite only > > for testing. > > > > Since things seem to be working I'd like to get this running on m-c > > (Nightlies) too and on all desktop opt platforms. > > > > Since my understanding is that test jobs can't be run just for Nightlies on > > m-c, bug 1242101 skips capturing screenshots within the test suite itself if > > the update channel isn't "nightly". > > Let me make sure I understand before I address the review. > > Do you want this subsuite to run on every push of mozilla-central? Or just > after a nightly build? Ideally I want this subsuite to only run on a Nightly build but since that's not currently possible, I made the test quit right away if it's not a nightly (by looking at the update channel). > I don't know if we can *only* enable tests for nightly builds but if we > could, would you prefer that? Yes I would prefer that but if it's not trivial then I think the current solution is fine.
Keywords: leave-open
Attachment #8711280 - Flags: review?(armenzg) → review+
Comment on attachment 8711280 [details] MozReview Request: Bug 1242103 - Add browser-chrome-screenshots to m-c and other try. r=armenzg https://reviewboard.mozilla.org/r/32061/#review29185
(In reply to Matthew N. [:MattN] from comment #9) > (In reply to Armen Zambrano [:armenzg] - Engineering productivity from > comment #5) > > All builds on release branches (including central) are PGO (where it > > applies). > > I see regular "opt" builds at > https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though. > That's news to me! :) Could you please add a note about preferring to run on nightly if possible and refer back to this bug? It would be ideal if one day we could *only* run it on nightly.
When it occurs to someone that "omg, nightly-only, that means it's going to be hidden and then broken and then be a waste of resources!!1!," MattN and I talked about it, and he's fine with it winding up tier-3, and his script making use of the output will let him know when someone breaks it and he has to sort through a day or two of pushes to both m-i and f-t to figure out who it was that broke him.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Keywords: leave-open
Resolution: --- → FIXED
Marked these as tier 2 for now until bug 1244199 gets deployed to Treeherder to let us mark buildbot jobs as tier 3.
Depends on: 1275549
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: