1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Robert Wood [:rwood]]

In Bug 1397180, Tarek landed the ability to run talos locally against a heavy-user profile downloaded from taskcluster.

We'd like to apply the functionality to production and have the talos tp6 and ts_paint tests run in duplication but against a heavy user profile downloaded from tc.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I see this being much like the 'g5' job where we run ['tp5', 'ts_paint'] with a web_extension; in this case it will be a heavy profile :)

Marking October to spend a day or two working on this- if it is harder we can finish in November.

Comment 2

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7c497e924a170f85bb2db015933d76fd594cc25

Comment 3

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #2)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7c497e924a170f85bb2db015933d76fd594cc25

Awesome so ts_paint and tp6 both run green on try using a heavy user profile (at least on linux 64).

One issue:

With the current architecture for tp6 it's not possible to have a single job for ts_paint and tp6 with heavy-user profile turned on. This is because tp6 uses mitmproxy, and mitmproxy is set at the suite level in talos [1]. 

This is done at the talos suite level and not the test level because we start mitmproxy up first before initializing the browser, because we need to import the CA cert into Firefox, setup and turn on the proxy in Firefox, as well as playback all the mitmproxy recordings. We don't want to do that on a per test basis i.e. do those steps once for tp6_google, then repeat all and do it again for tp6_amazon, etc.

So because of that we cannot have a test inside the suite test list which doesn't use mitmproxy - because the playback and proxy are turned on for all the tp6 tests / for that entire suite and the test will fail as mitmproxy tries to serve up the page etc.

Either we do some hacking so that mitmproxy setup etc. is done at the test level and not the suite level, which is a fairly big job and also will require additional code to disable the Firefox proxy etc. because right now we don't do that since no other tests are run in tp6.

Or the other option is we have two separate heavy-user profile jobs, 'h1-e10s' that runs ts_paint_heavy, and 'h2-e10s' that runs tp6_google_heavy, and the other 3 tp6 tests. What do you think? Personally I think this is the best option, and it also leaves room for expanding the jobs in the future.

[1] http://searchfox.org/mozilla-central/rev/31606bbabc50b08895d843b9f5f3da938ccdfbbf/testing/talos/talos.json#125

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for pointing this out- I think h1/h2 are good job names.  Keeping things simple we should go with your proposal.

I would like to get some bugs on file to make this more flexible- ideally all our tests should be able to be swapped around, run locally, etc.

That seems like a bit more work that we should be doing right now.

Comment 5

[Robert Wood [:rwood]]

Ok, here's what's required:

1. talos changes to add 'h1-e10s' and 'h2-e10s' jobs (ts_paint_heavy and tp6_*_heavy)
2. buildbot configs to add new h1 and h2 jobs
3. taskcluster configs to add new h1 and h2 jobs
4. treeherder etl updates for new h1 and h2 jobs

Comment 6

[Robert Wood [:rwood]]

FYI, I filed: Bug 1407989 - Make talos suites more flexible

Comment 7

[Robert Wood [:rwood]]

Attachment: Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

Review commit: https://reviewboard.mozilla.org/r/188774/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/188774/

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

https://reviewboard.mozilla.org/r/188774/#review194038

this looks good, one of a few patches to get this going

Comment 9

[Robert Wood [:rwood]]

Thanks :jmaher. Do we want either of these on stylo-disabled and stylo-threads? Maybe h1 on stylo disabled, and h2 on stylo-disabled and stylo-threads to mirror current ts_paint and tp6_*. Or do we want it consistent, maybe just both on stylo_disabled...

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I am fine not running those on stylo-disabled; we are <5 weeks away from shipping 57 and if all goes well we won't need anymore stylo-disabled.

Comment 11

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #10)
> I am fine not running those on stylo-disabled; we are <5 weeks away from
> shipping 57 and if all goes well we won't need anymore stylo-disabled.

Ok awesome thx

Comment 12

[Robert Wood [:rwood]]

Attachment: bug1407398_buildbot.patch

Buildbot configs for new talos heavy-user profile jobs

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8917857 [details] [diff] [review]
bug1407398_buildbot.patch

Review of attachment 8917857 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +2904,5 @@
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 58):
> +    if branch.get('enable_talos') is False:
> +        continue
> +    branch['h1-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> +    branch['h2-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)

could you put this in the speedometer block so it is easier to see all the tests added for 58?

Comment 14

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #13)
> Comment on attachment 8917857 [details] [diff] [review]
> bug1407398_buildbot.patch
> 
> Review of attachment 8917857 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/config.py
> @@ +2904,5 @@
> > +for name, branch in items_at_least(BRANCHES, 'gecko_version', 58):
> > +    if branch.get('enable_talos') is False:
> > +        continue
> > +    branch['h1-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> > +    branch['h2-e10s_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS)
> 
> could you put this in the speedometer block so it is easier to see all the
> tests added for 58?

Ah yes good call, thanks

Comment 15

[Robert Wood [:rwood]]

Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/188774/diff/1-2/

Comment 16

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #15)
> Comment on attachment 8917825 [details]
> Bug 1407398 - Taskcluster configs for new talos heavy user-profile jobs;
> 
> Review request updated; see interdiff:
> https://reviewboard.mozilla.org/r/188774/diff/1-2/

I didn't mean to combine these into the same review. I'm going to update this to have one single patch for the talos changes with the taskcluster configs together. I'll r? when I'm ready for another review of that combined patch, thanks

Comment 17

[Robert Wood [:rwood]]

Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/188774/diff/2-3/

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

https://reviewboard.mozilla.org/r/188774/#review194116

still r+, the taskcluster configs look standard.  Ideally we will be able to test on try, look for noise, etc.

Comment 19

[Robert Wood [:rwood]]

Attachment: bug1407398_buildbot_updated.patch

Post-review update on buildbot configs, carrying fw the r+

Comment 20

[Robert Wood [:rwood]]

Landed buildbot configs:

https://hg.mozilla.org/build/buildbot-configs/rev/0187778eccfb45e50fec09edaee145ca9387cf47

Comment 21

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #18)
> Comment on attachment 8917825 [details]
> Bug 1407398 - Add new talos heavy-user profile test jobs;
> 
> https://reviewboard.mozilla.org/r/188774/#review194116
> 
> still r+, the taskcluster configs look standard.  Ideally we will be able to
> test on try, look for noise, etc.

Thanks. I'll land the treeherder etl patch first; and once that's in, with the buildbot configs already landed, I'll be able to run the talos changes with taskcluster configs on try.

Comment 22

[GitHub Autolander Bot]

Attachment: [treeherder] rwood-moz:bug1407398 > mozilla:master

Comment 23

[Robert Wood [:rwood]]

Comment on attachment 8917922 [details] [review]
[treeherder] rwood-moz:bug1407398 > mozilla:master

Sending this your way :camd since :emorley is PTO, if you don't mind, thanks

Comment 24

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/buildbot-configs/rev/40289a691194e9bf5d43ff095512b328355dd48b

Comment 25

[Robert Wood [:rwood]]

Running a bunch of retriggers, with chromez used for 'h1' and dromaeojs used for 'h2':

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a5f66eab6c4ff30858e94d7dcb701a996f57e32

Comment 26

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/8ae17157aa5be86fe908c7998df985876ddd3a91
Bug 1407398 - Treeherder buildbot etl updates for new talos heavy-user profile jobs

Comment 27

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b0d77371e03f2b3a322a7fc21a2dcc386d275276

Comment 28

[Robert Wood [:rwood]]

Try run with the treeherder patch in production:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=328cfe913d5af13a1d2b43c9632d6593955da617

Comment 29

[Robert Wood [:rwood]]

Comment on attachment 8917825 [details]
Bug 1407398 - Run talos tp6 and ts_paint in duplication but with a heavy user-profile;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/188774/diff/3-4/

Comment 30

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=35fe3c57ab2ff5a727cb3599d3b56ad9ff8ec3e9

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/85e56b82f496
Run talos tp6 and ts_paint in duplication but with a heavy user-profile; r=jmaher

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/85e56b82f496