1504251 - Move Raptor jobs to tier 1, and turn off duplicate talos jobs

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Comment 1

[Robert Wood [:rwood]]

:jmaher, do we need to have Raptor support for stylo threads also (like the talos tp6-stylo-threads job)? Raptor doesn't have support for --stylo-threads. If we need that, I'll file a new bug for that.

I'm assuming we should leave the talos tp6-stylo-threads job running for now, and then turn it off after we add that to Raptor.

Comment 2

[Robert Wood [:rwood]]

Attachment: Bug 1504251 - Move Raptor jobs to tier 1, and turn off duplicate talos jobs; r?jmaher

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure about --stylo-threads, it appears to be added in bug 1383146, :cpeterson, do we still have a need to test talos with different stylo threads?

Comment 4

[Chris Peterson [:cpeterson]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> I am not sure about --stylo-threads, it appears to be added in bug 1383146,
> :cpeterson, do we still have a need to test talos with different stylo
> threads?

Cameron, do you still care about running Talos tp6 page load tests, AWSY, and web-platform-tests for both STYLO_THREADS=1 and STYLO_THREADS=4 (from bug 1383146)?

https://searchfox.org/mozilla-central/search?case=true&q=stylo-threads

https://searchfox.org/mozilla-central/search?case=true&q=STYLO_THREADS

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=56849ddc5ab9b3c2134902fb4bf09fb7dbbab2fe

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f9a9234959f
Move Raptor jobs to tier 1, and turn off duplicate talos jobs; r=jmaher

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/9f9a9234959f

Comment 8

[Cameron McCormack (:heycam)]

(In reply to Chris Peterson [:cpeterson] from comment #4)
> Cameron, do you still care about running Talos tp6 page load tests, AWSY,
> and web-platform-tests for both STYLO_THREADS=1 and STYLO_THREADS=4 (from
> bug 1383146)?

It would be good if we could keep running Talos and AWSY in both configurations, since they have some different characteristics and we don't want to regress either.

The chance of there being a correctness bug that would creep in with STYLO_THREADS=1 but not STYLO_THREADS=4 is less likely, so I wouldn't be as concerned about losing those.  (I wonder anyway if any of our platforms will naturally choose to use one styling thread?)

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

:heycam, talos tp6 stylo-threads=1 doesn't compare against anything else, we only run tp6 on raptor which is different pages, tools, and measurements. I only see value in correctness (i.e. not crashing) since talos failures will be lumped together into generic failures. Should we move this to raptor or a unittest? What difference do we get between AWSY and talos ss tp6 ?

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

:davehunt, in the previous comment (comment 9), I outline what we run and why I think we shouldn't be running tp6 on talos. Is this something we can move to raptor? I would like to simplify the sheriffs use case as much as possible and also allow for reducing the one reason for running mitmproxy in talos and the related intermittents.

Comment 11

[Intermittent Failures Robot]

38 failures in 178 pushes (0.213 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 38

Platform breakdown:

• linux64: 17
• linux64-qr: 14
• osx-10-10: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1504251&startday=2019-01-26&endday=2019-01-26&tree=all

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

Attachment: disable talos tp6_google

The sheriffs are concerned about going to merge with a near-permaorange here, even if it's going to be disabled RSN.  This should turn off just tp6_google.  (A better patch might be to disable tp6 on talos entirely, but I think that's not needed in order to get green for merge - try running at https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b309ddb8d3818930911483f81eeb886571673b3). Will take any r+

Comment 13

[Intermittent Failures Robot]

51 failures in 2383 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #17 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 5
• mozilla-inbound: 46

Platform breakdown:

• linux64: 27
• linux64-qr: 17
• osx-10-10: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1504251&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9039465 [details] [diff] [review]
disable talos tp6_google

Review of attachment 9039465 [details] [diff] [review]:
-----------------------------------------------------------------

I am fine with the talos.json changes; the timeoutmanager.cpp changes seem not so scary, but not in my wheelhouse.

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

Sorry, only the talos.json ones - I was tired and sick when I threw that up for review at 11pm

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8df4bb7de73f
disable talos tp6_google r=jmaher

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

Attachment: beta.patch

Beta version of the patch - and I didn't notice this bug was marked as resolved-fixed; I'll reopen since we hadn't fully disabled tp6 in talos with the initial landing. (If I'd notices, I'd have opened a second bug - sorry, it was late and I was sick).

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 9039566 [details] [diff] [review]
beta.patch

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1270059

User impact if declined

no user impact; near-permaorange on

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

Just turning off a test

String changes made/needed

none

Comment 19

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9039566 [details] [diff] [review]
beta.patch

Disable some tests. OK for beta uplift.

Comment 20

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/8df4bb7de73f

Comment 21

[Intermittent Failures Robot]

49 failures in 389 pushes (0.126 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 5
• mozilla-central: 8
• mozilla-beta: 6
• autoland: 24
• mozilla-inbound: 6

Platform breakdown:

• linux64: 23
• linux64-qr: 18
• osx-10-10: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1504251&startday=2019-01-28&endday=2019-01-28&tree=all

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Randell had landed this on Monday: https://hg.mozilla.org/releases/mozilla-beta/rev/01fb0fbf92d4c35b35bddd7708a92313e074cd10

Comment 23

[Intermittent Failures Robot]

49 failures in 2628 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #36 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 24
• mozilla-inbound: 6
• try: 5
• mozilla-beta: 6
• mozilla-central: 8

Platform breakdown:

• linux64-qr: 18
• osx-10-10: 8
• linux64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1504251&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 24

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We're tracking removing talos stylo sequential threads tp6 in bug 1508694.