Closed Bug 1407398 Opened 7 years ago Closed 7 years ago

Run talos tp6 and ts_paint in duplication but with a heavy user-profile

Categories

(Testing :: Talos, enhancement)

Product:
Testing
Component:
Talos
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox58 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla58
Tracking Flags:
Tracking Status
firefox58 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

Details

(Whiteboard: [PI:October])

Attachments

(3 files, 1 obsolete file)

Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;
59 bytes, text/x-review-board-request
jmaher
: review+
Details
bug1407398_buildbot_updated.patch
2.73 KB, patch
rwood
: review+
Details | Diff | Splinter Review
[treeherder] rwood-moz:bug1407398 > mozilla:master
47 bytes, text/x-github-pull-request
camd
: review+
Details | Review 
In Bug 1397180, Tarek landed the ability to run talos locally against a heavy-user profile downloaded from taskcluster.

We'd like to apply the functionality to production and have the talos tp6 and ts_paint tests run in duplication but against a heavy user profile downloaded from tc.
I see this being much like the 'g5' job where we run ['tp5', 'ts_paint'] with a web_extension; in this case it will be a heavy profile :)

Marking October to spend a day or two working on this- if it is harder we can finish in November.
Whiteboard: [PI:October]
(In reply to Robert Wood [:rwood] from comment #2)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7c497e924a170f85bb2db015933d76fd594cc25

Awesome so ts_paint and tp6 both run green on try using a heavy user profile (at least on linux 64).

One issue:

With the current architecture for tp6 it's not possible to have a single job for ts_paint and tp6 with heavy-user profile turned on. This is because tp6 uses mitmproxy, and mitmproxy is set at the suite level in talos [1]. 

This is done at the talos suite level and not the test level because we start mitmproxy up first before initializing the browser, because we need to import the CA cert into Firefox, setup and turn on the proxy in Firefox, as well as playback all the mitmproxy recordings. We don't want to do that on a per test basis i.e. do those steps once for tp6_google, then repeat all and do it again for tp6_amazon, etc.

So because of that we cannot have a test inside the suite test list which doesn't use mitmproxy - because the playback and proxy are turned on for all the tp6 tests / for that entire suite and the test will fail as mitmproxy tries to serve up the page etc.

Either we do some hacking so that mitmproxy setup etc. is done at the test level and not the suite level, which is a fairly big job and also will require additional code to disable the Firefox proxy etc. because right now we don't do that since no other tests are run in tp6.

Or the other option is we have two separate heavy-user profile jobs, 'h1-e10s' that runs ts_paint_heavy, and 'h2-e10s' that runs tp6_google_heavy, and the other 3 tp6 tests. What do you think? Personally I think this is the best option, and it also leaves room for expanding the jobs in the future.

[1] http://searchfox.org/mozilla-central/rev/31606bbabc50b08895d843b9f5f3da938ccdfbbf/testing/talos/talos.json#125
Flags: needinfo?(jmaher)
thanks for pointing this out- I think h1/h2 are good job names.  Keeping things simple we should go with your proposal.

I would like to get some bugs on file to make this more flexible- ideally all our tests should be able to be swapped around, run locally, etc.

That seems like a bit more work that we should be doing right now.
Flags: needinfo?(jmaher)
Ok, here's what's required:

1. talos changes to add 'h1-e10s' and 'h2-e10s' jobs (ts_paint_heavy and tp6_*_heavy)
2. buildbot configs to add new h1 and h2 jobs
3. taskcluster configs to add new h1 and h2 jobs
4. treeherder etl updates for new h1 and h2 jobs
FYI, I filed: Bug 1407989 - Make talos suites more flexible
Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

https://reviewboard.mozilla.org/r/188774/#review194038

this looks good, one of a few patches to get this going
Attachment #8917825 - Flags: review?(jmaher) → review+
Thanks :jmaher. Do we want either of these on stylo-disabled and stylo-threads? Maybe h1 on stylo disabled, and h2 on stylo-disabled and stylo-threads to mirror current ts_paint and tp6_*. Or do we want it consistent, maybe just both on stylo_disabled...
I am fine not running those on stylo-disabled; we are <5 weeks away from shipping 57 and if all goes well we won't need anymore stylo-disabled.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #10)
> I am fine not running those on stylo-disabled; we are <5 weeks away from
> shipping 57 and if all goes well we won't need anymore stylo-disabled.

Ok awesome thx
Attached patch bug1407398_buildbot.patch (obsolete) — — Splinter Review 
Buildbot configs for new talos heavy-user profile jobs
Attachment #8917857 - Flags: review?(jmaher)
Comment on attachment 8917857 [details] [diff] [review]
bug1407398_buildbot.patch

Review of attachment 8917857 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +2904,5 @@
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 58):
> +    if branch.get('enable_talos') is False:
> +        continue
> +    branch['h1-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> +    branch['h2-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)

could you put this in the speedometer block so it is easier to see all the tests added for 58?
Attachment #8917857 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #13)
> Comment on attachment 8917857 [details] [diff] [review]
> bug1407398_buildbot.patch
> 
> Review of attachment 8917857 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/config.py
> @@ +2904,5 @@
> > +for name, branch in items_at_least(BRANCHES, 'gecko_version', 58):
> > +    if branch.get('enable_talos') is False:
> > +        continue
> > +    branch['h1-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> > +    branch['h2-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> 
> could you put this in the speedometer block so it is easier to see all the
> tests added for 58?

Ah yes good call, thanks
(In reply to Robert Wood [:rwood] from comment #15)
> Comment on attachment 8917825 [details]
> Bug 1407398 - Taskcluster configs for new talos heavy user-profile jobs;
> 
> Review request updated; see interdiff:
> https://reviewboard.mozilla.org/r/188774/diff/1-2/

I didn't mean to combine these into the same review. I'm going to update this to have one single patch for the talos changes with the taskcluster configs together. I'll r? when I'm ready for another review of that combined patch, thanks
Attachment #8917825 - Flags: review+ → review?(jmaher)
Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

https://reviewboard.mozilla.org/r/188774/#review194116

still r+, the taskcluster configs look standard.  Ideally we will be able to test on try, look for noise, etc.
Attachment #8917825 - Flags: review?(jmaher) → review+
Post-review update on buildbot configs, carrying fw the r+
Attachment #8917857 - Attachment is obsolete: true
Attachment #8917912 - Flags: review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #18)
> Comment on attachment 8917825 [details]
> Bug 1407398 - Add new talos heavy-user profile test jobs;
> 
> https://reviewboard.mozilla.org/r/188774/#review194116
> 
> still r+, the taskcluster configs look standard.  Ideally we will be able to
> test on try, look for noise, etc.

Thanks. I'll land the treeherder etl patch first; and once that's in, with the buildbot configs already landed, I'll be able to run the talos changes with taskcluster configs on try.
Comment on attachment 8917922 [details] [review]
[treeherder] rwood-moz:bug1407398 > mozilla:master

Sending this your way :camd since :emorley is PTO, if you don't mind, thanks
Attachment #8917922 - Flags: review?(cdawson)
Running a bunch of retriggers, with chromez used for 'h1' and dromaeojs used for 'h2':

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a5f66eab6c4ff30858e94d7dcb701a996f57e32
Attachment #8917922 - Flags: review?(cdawson) → review+
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/8ae17157aa5be86fe908c7998df985876ddd3a91
Bug 1407398 - Treeherder buildbot etl updates for new talos heavy-user profile jobs
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/85e56b82f496
Run talos tp6 and ts_paint in duplication but with a heavy user-profile; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/85e56b82f496
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox58: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: