Closed
Bug 1242103
Opened 9 years ago
Closed 9 years ago
Add a job to run mochitest-browser-chrome "screenshots" subsuite on Nightlies
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: MattN, Assigned: MattN)
References
(Depends on 1 open bug)
Details
(Whiteboard: [browser-chrome-screenshots])
Attachments
(2 files)
In bug 1231408 I added a mochitest-browser-screenshots subsuite to take screenshots of the browser chrome to detect regressions for yosemite only for testing.
Since things seem to be working I'd like to get this running on m-c (Nightlies) too and on all desktop opt platforms.
Since my understanding is that test jobs can't be run just for Nightlies on m-c, bug 1242101 skips capturing screenshots within the test suite itself if the update channel isn't "nightly".
Assignee | ||
Comment 1•9 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/32057/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32057/
Attachment #8711275 -
Flags: review?(armenzg)
Assignee | ||
Comment 2•9 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/32061/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32061/
Attachment #8711280 -
Flags: review?(armenzg)
Assignee | ||
Comment 3•9 years ago
|
||
Builders added:
+ Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-screenshots
+ Rev4 MacOSX Snow Leopard 10.6 try opt test mochitest-browser-screenshots
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-screenshots
+ Ubuntu VM 12.04 mozilla-central pgo test mochitest-browser-screenshots
+ Ubuntu VM 12.04 try opt test mochitest-browser-screenshots
+ Windows 10 64-bit try opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 7 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 7 32-bit try opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows 8 64-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows 8 64-bit try opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central opt test mochitest-browser-screenshots
+ Windows XP 32-bit mozilla-central pgo test mochitest-browser-screenshots
+ Windows XP 32-bit try opt test mochitest-browser-screenshots
The main unknown for me now is PGO. Are nightlies (channel=nightly) PGO? Do we do more than one channel=nightly build on a normal day per platform?
Comment 4•9 years ago
|
||
Comment on attachment 8711275 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots test configs for Windows and Linux. r=armenzg
https://reviewboard.mozilla.org/r/32057/#review28907
Attachment #8711275 -
Flags: review?(armenzg) → review+
Comment 5•9 years ago
|
||
Nightly builds are PGO for Linux and Windows (it does not apply for Mac)
PGO builds on integration branches happen on a schedule.
All builds on release branches (including central) are PGO (where it applies).
We sometimes do more than one nightly a day.
Comment 6•9 years ago
|
||
(In reply to Matthew N. [:MattN] from comment #0)
> In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> screenshots of the browser chrome to detect regressions for yosemite only
> for testing.
>
> Since things seem to be working I'd like to get this running on m-c
> (Nightlies) too and on all desktop opt platforms.
>
> Since my understanding is that test jobs can't be run just for Nightlies on
> m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> the update channel isn't "nightly".
Let me make sure I understand before I address the review.
Do you want this subsuite to run on every push of mozilla-central? Or just after a nightly build?
I don't know if we can *only* enable tests for nightly builds but if we could, would you prefer that?
Comment 7•9 years ago
|
||
I think every build on m-c has *nightly* as its update channel. I could be wrong though.
Assignee | ||
Comment 8•9 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #7)
> I think every build on m-c has *nightly* as its update channel. I could be
> wrong though.
No, a regular m-c build on a push has app.update.channel = "default", I just double-checked.
Assignee | ||
Comment 9•9 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #5)
> All builds on release branches (including central) are PGO (where it
> applies).
I see regular "opt" builds at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.
> We sometimes do more than one nightly a day.
Sure
Assignee | ||
Comment 10•9 years ago
|
||
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6)
> (In reply to Matthew N. [:MattN] from comment #0)
> > In bug 1231408 I added a mochitest-browser-screenshots subsuite to take
> > screenshots of the browser chrome to detect regressions for yosemite only
> > for testing.
> >
> > Since things seem to be working I'd like to get this running on m-c
> > (Nightlies) too and on all desktop opt platforms.
> >
> > Since my understanding is that test jobs can't be run just for Nightlies on
> > m-c, bug 1242101 skips capturing screenshots within the test suite itself if
> > the update channel isn't "nightly".
>
> Let me make sure I understand before I address the review.
>
> Do you want this subsuite to run on every push of mozilla-central? Or just
> after a nightly build?
Ideally I want this subsuite to only run on a Nightly build but since that's not currently possible, I made the test quit right away if it's not a nightly (by looking at the update channel).
> I don't know if we can *only* enable tests for nightly builds but if we
> could, would you prefer that?
Yes I would prefer that but if it's not trivial then I think the current solution is fine.
Assignee | ||
Updated•9 years ago
|
Keywords: leave-open
Updated•9 years ago
|
Attachment #8711280 -
Flags: review?(armenzg) → review+
Comment 12•9 years ago
|
||
Comment on attachment 8711280 [details]
MozReview Request: Bug 1242103 - Add browser-chrome-screenshots to m-c and other try. r=armenzg
https://reviewboard.mozilla.org/r/32061/#review29185
Comment 13•9 years ago
|
||
(In reply to Matthew N. [:MattN] from comment #9)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #5)
> > All builds on release branches (including central) are PGO (where it
> > applies).
>
> I see regular "opt" builds at
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central too though.
>
That's news to me! :)
Could you please add a note about preferring to run on nightly if possible and refer back to this bug?
It would be ideal if one day we could *only* run it on nightly.
Comment 14•9 years ago
|
||
When it occurs to someone that "omg, nightly-only, that means it's going to be hidden and then broken and then be a waste of resources!!1!," MattN and I talked about it, and he's fine with it winding up tier-3, and his script making use of the output will let him know when someone breaks it and he has to sort through a day or two of pushes to both m-i and f-t to figure out who it was that broke him.
Comment 15•9 years ago
|
||
bugherder |
Assignee | ||
Comment 16•9 years ago
|
||
Marked these as tier 2 for now until bug 1244199 gets deployed to Treeherder to let us mark buildbot jobs as tier 3.
Updated•7 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•