1770562 - Perma raptor youtube OS X 10.15 WebRender Shippable failing as exception with deadline exceeded

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Iulian Moraru]

The failure has no failure log.

Last green jobs were on May 18th.

Push where ytp-h264-p job failed for the first time.
Push where ytp-v9-p job failed for the first time.

Comment 1

[Iulian Moraru]

Hi Andrew! Can you please take a look at this?
Thank you!

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Because these started to fail not with the same push but adjacent ones, could this be related to the macOS testing machines?

Comment 3

[Intermittent Failures Robot]

22 failures in 2694 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 22

Platform and build breakdown:

• macosx1015-64-shippable-qr: 22
  • opt: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 4

[Greg Mierzwinski [:sparky]]

:dhouse, it looks like the Mac power testing machines are unresponsive, could you take a look?

Comment 5

[Intermittent Failures Robot]

36 failures in 2624 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 36

Platform and build breakdown:

• macosx1015-64-shippable-qr: 36
  • opt: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-05-23&endday=2022-05-29&tree=all

Comment 6

[:dhouse]

generic-worker is failing because of errors from taskcluster like:

{
  "code": "ResourceNotFound",
  "message": "Worker pool releng-hardware/gecko-t-osx-1014-power does not exist\n\n---\n\n* method:     registerWorker\n* errorCode:  ResourceNotFound\n* statusCode: 404\n* time:       2022-06-01T15:23:45.161Z",

I'm reconfiguring them so they get connected.

Comment 7

[:dhouse]

:sparky they're claiming work again now. Please let me know if there are any problems.

Comment 8

[Greg Mierzwinski [:sparky]]

Thank you very much :dhouse! Will do :)

Comment 9

[Intermittent Failures Robot]

20 failures in 2917 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 20

Platform and build breakdown:

• macosx1015-64-shippable-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 10

[Andrej (:aglavic)]

Hi :dhouse, we are still having issues can you investigate?

Comment 11

[Intermittent Failures Robot]

40 failures in 3336 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #28 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 40

Platform and build breakdown:

• macosx1015-64-shippable-qr: 40
  • opt: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 12

[Intermittent Failures Robot]

26 failures in 3170 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

Repository breakdown:

• mozilla-central: 26

Platform and build breakdown:

• macosx1015-64-shippable-qr: 26
  • opt: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-06-13&endday=2022-06-19&tree=all

Comment 13

[:dhouse]

:andrej I fixed this. It was an issue with scopes for the workers to pull from the 1015 queue. I set some monitoring to check if this happens again.
Can you confirm if the mac power testers are working correctly now?

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

Hi :dhouse

Here is some mac power data. Is this what you were referring to? (as of posting this, looks like nothing since mid-May)

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&series=mozilla-central,3370249,1,10&timerange=7776000

Comment 15

[:dhouse]

(In reply to Kash [:kshampur] ⌚EST from comment #14)

Hi :dhouse

Here is some mac power data. Is this what you were referring to? (as of posting this, looks like nothing since mid-May)

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&series=mozilla-central,3370249,1,10&timerange=7776000

This task record is on the previous queue "releng-hardware/gecko-t-osx-1014-power"

:andrej can you switch these to run on the pool "releng-hardware/gecko-t-osx-1015-power" ?
If you want, I can just switch them to using the old queue name. Or do you want to rename the queue to remove the macos version like "gecko-t-osx-power"?

Comment 16

[:dhouse]

:sparky can you switch these to using the new pool name? This doesn't seem ideal to have the os version hard-coded. So if you want to change the pool name, I'm can switch it np (or switch back to using the 1014 pool name to just make this work).

Comment 17

[Intermittent Failures Robot]

14 failures in 3075 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14

Platform and build breakdown:

• macosx1015-64-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-06-20&endday=2022-06-26&tree=all

Comment 18

[Greg Mierzwinski [:sparky]]

:andrej, could you take care of making the change? We'd need to change these lines of code: https://searchfox.org/mozilla-central/search?q=t-osx-1014-power&path=&case=false&regexp=false

Comment 19

[Andrej (:aglavic)]

Got it, I'll get straight on this

Comment 20

[Intermittent Failures Robot]

12 failures in 3312 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12

Platform and build breakdown:

• macosx1015-64-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-06-27&endday=2022-07-03&tree=all

Comment 21

[Andrej (:aglavic)]

Attachment: Bug 1770562 - Use macosx-1015 for power tests instead of macosx-1014. r=#perftest

What we are doing:
Switching mac power data to run on osx-1015 from osx-1014

Why:
Generic workers were failing and attempting to pull from osx-1014, so we are setting them to pull from the osx-1015 worker pool

Comment 22

[Intermittent Failures Robot]

14 failures in 3072 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14

Platform and build breakdown:

• macosx1015-64-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-07-04&endday=2022-07-10&tree=all

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/016cbd986226
Use macosx-1015 for power tests instead of macosx-1014. r=dhouse

Comment 24

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/016cbd986226

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit auto_nag documentation.

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:dhouse, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox103 to wontfix.

For more information, please visit auto_nag documentation.

Comment 27

[:dhouse]

:andrej what do you think? I'm re-assigning to you since I do not know if it needs to be uplifted.

Comment 28

[Andrej (:aglavic)]

I don't think this needs to be uplifted

Comment 29

[Iulian Moraru]

This is still happening: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=OS%2CX%2C10.15%2CWebRender%2CShippable%2Craptor&tochange=2a77c9b52e2599a310a837e1907253d5ab271a99&selectedTaskRun=J48ecG15RjW2Eyxr9xhy1Q.0

Hi Andrej! Can you please take a look? Should we reopen this bug?
Thank you!

Comment 30

[Andrej (:aglavic)]

Hi dhouse,
Can you advise on the issue? You mentioned in my phabricator patch that the worker pool was not registered to taskcluster but that it should work now after you re-enabled the API key.

Comment 31

[:dhouse]

(In reply to Andrej (:andrej) from comment #30)

Hi dhouse,
Can you advise on the issue? You mentioned in my phabricator patch that the worker pool was not registered to taskcluster but that it should work now after you re-enabled the API key.

I got them working again last night: I checked and my api key was disabled again. I will check into it more this week. I think there is automation disabling the api key.

Comment 32

[Intermittent Failures Robot]

12 failures in 3341 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12

Platform and build breakdown:

• macosx1015-64-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2022-07-11&endday=2022-07-17&tree=all

Comment 33

[Andrej (:aglavic)]

No intermittent fails in the past 2 weeks, closing bug

Comment 34

[Iulian Moraru]

Hi Andrej! Can you please take a look at this? It is happening again, last green job was on Feb 10th: range from last green job.

Comment 35

[Intermittent Failures Robot]

6 failures in 3766 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform and build breakdown:

• macosx1015-64-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2023-02-06&endday=2023-02-12&tree=all

Comment 36

[Cosmin Sabou [:CosminS]]

The worker these ran on is offline and gecko-t-osx-1015-power pool lists no workers online. masterwayz, is this something you could look into as I see dhouse is no longer available? Thank you.

Comment 37

[Michelle Goossens [:masterwayz]]

Will look into this! Will probably end up being later today due to some bitbar access permissions that someone else first needs to fix for me.
And yes, I'll have to be pinged instead of dhouse.
(leaving NI in place)

Comment 38

[Cosmin Sabou [:CosminS]]

masterwayz, any updates on this kind of workers?

Comment 39

[Intermittent Failures Robot]

13 failures in 3856 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Platform and build breakdown:

• macosx1015-64-shippable-qr: 13
  • opt: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2023-02-13&endday=2023-02-19&tree=all

Comment 40

[Michelle Goossens [:masterwayz]]

I lack the access/context here so passing this along.

Comment 41

[Andrew Erickson [:aerickson]]

Michelle, I was able to ssh into these hosts.

Details at https://mozilla-hub.atlassian.net/wiki/spaces/ROPS/pages/179601939/Bitbar+Mac+Hosts.

Please let me know if you have any issues.

Comment 42

[Michelle Goossens [:masterwayz]]

Ah right, those documents didn't exist yet, thanks for making them!
I will do this once I get access to that SSH key...

Comment 43

[Michelle Goossens [:masterwayz]]

Jobs should be running now again!

Comment 44

[Cosmin Sabou [:CosminS]]

Jobs are picked up again, thank you for the fix.

Comment 45

[Intermittent Failures Robot]

13 failures in 3643 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Platform and build breakdown:

• macosx1015-64-shippable-qr: 13
  • opt: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770562&startday=2023-02-20&endday=2023-02-26&tree=all