1657084 - Linux asan gl2c webgl tasks fail permanently with exception

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This started with the change from bug 1616925.

See: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Csuperseded%2Cretry%2Cusercancel&searchStr=linux%2C18.04%2Cx64%2Casan%2Copt%2Cmochitests%2Ctest-linux1804-64-asan%2Fopt-mochitest-webgl2-core-e10s%2Cgl2c&fromchange=11f0f54c6e0a641470e6fba29733b3ab764df3ef&tochange=302bad8cf16d54cc4f30a13810d77b94b3903af6&selectedTaskRun=MvXT-kKQSESdWMFY4ejQNg.5

Comment 1

[Kelsey Gilbert [:jgilbert]]

That sure looks like an infra issue.
I'm not sure what I can do from a gecko webgl engineering side! We need logs at least, so we should figure out why we never get those.

I really don't see much likely in bug 1616925.
One thing worth checking: https://searchfox.org/mozilla-central/rev/a315a1a0f09550e23e4590a77e74f36543315da3/taskcluster/taskgraph/transforms/fetch.py#187

            task['scopes'] = ["secrets:get:" + job.get('secret')]

Other code generally adds to the list instead:
https://searchfox.org/mozilla-central/search?q=%27scopes%27&path=taskcluster%2Ftaskgraph%2Ftransforms&case=false&regexp=false

[taskcluster/taskgraph/transforms/github_sync.py]
23	task['scopes'].append('secrets:get:' + secret)

[taskcluster/taskgraph/transforms/job/common.py]
128	taskdesc['scopes'].append('secrets:get:project/taskcluster/gecko/hgfingerprint')
129	taskdesc['scopes'].append('secrets:get:project/taskcluster/gecko/hgmointernal')

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Docker and toolchain images got rebuilt by the push. It's possible this just picked changes like unpinned dependencies or other changes landed which didn't rebuilt these artifacts.

Comment 3

[Tom Ritter [:tjr]]

(In reply to Jeff Gilbert [:jgilbert] from comment #1)

That sure looks like an infra issue.

Yeah....

I'm not sure what I can do from a gecko webgl engineering side! We need logs at least, so we should figure out why we never get those.

I really don't see much likely in bug 1616925.
One thing worth checking: https://searchfox.org/mozilla-central/rev/a315a1a0f09550e23e4590a77e74f36543315da3/taskcluster/taskgraph/transforms/fetch.py#187

            task['scopes'] = ["secrets:get:" + job.get('secret')]

Other code generally adds to the list instead:

I'm skeptical this is the issue; in the other places, scopes was created as part of the task definition; this is the analogous location for the fetch task which is considerably simpler than the others. Also, this code isn't getting run as part of this job.

Comment 4

[Tom Ritter [:tjr]]

I captured the logs of an in-progress run. n=1, but it stops inside of test_2_conformance__textures__misc__texture-size-limit.html so I'm going to send in a try run omitting that test.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6c66ab73cf0f4074f4bc9909eb702704602466b

Comment 5

[Intermittent Failures Robot]

16 failures in 948 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 4

Platform breakdown:

• linux1804-64-asan: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-05&endday=2020-08-05&tree=all

Comment 6

[Tom Ritter [:tjr]]

So, it took a couple of runs, but then it succeeded? Except that the Tier-2 WebGL over IPC tests are not succeeding.

I'm still retriggering and watching logs (which must be done as they run, you can't get them afterwards....)

Comment 7

[Intermittent Failures Robot]

53 failures in 4618 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 35
• mozilla-central: 13
• try: 5

Platform breakdown:

• linux1804-64-asan: 53

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1657084 - Turn off webgl2-core tests on Linux asan until there is a fix for the permanent exception. r=jgilbert

Comment 9

[Intermittent Failures Robot]

60 failures in 3601 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 40
• try: 5
• mozilla-central: 15

Platform breakdown:

• linux1804-64-asan: 59
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 10

[Tom Ritter [:tjr]]

So I'm still trying to debug this, trying to find the test(s) that cause things to break. I'm still getting exceptions on the WebGL over IPC tests even after disabling two tests. it's slow debugging because I have to catch the jobs in-flight and then save the logs because they disappear.

Comment 11

[Intermittent Failures Robot]

16 failures in 811 pushes (0.02 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 12
• mozilla-central: 4

Platform breakdown:

• linux1804-64-asan: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-18&endday=2020-08-18&tree=all

Comment 12

[Intermittent Failures Robot]

69 failures in 4100 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 47
• mozilla-central: 17
• try: 5

Platform breakdown:

• linux1804-64-asan: 69

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 13

[Intermittent Failures Robot]

17 failures in 778 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 3
• mozilla-beta: 3

Platform breakdown:

• linux1804-64-asan: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-25&endday=2020-08-25&tree=all

Comment 14

[Andreea Pavel [:apavel]]

Hi Tom are there updates here?

There are 69 total failures in the last 7 days on linux1804-64-asan opt

Recent failure https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=56ce346234d1b5ec53b537ea22a9e545a21980ed&selectedTaskRun=MyBIKfOSSK6nmno6SGI5Sg.5 (log is unavailable)

Comment 15

[Intermittent Failures Robot]

74 failures in 4046 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 41
• mozilla-central: 15
• mozilla-beta: 14
• try: 4

Platform breakdown:

• linux1804-64-asan: 74

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 16

[Tom Ritter [:tjr]]

(In reply to Andreea Pavel [:apavel] from comment #14)

Hi Tom are there updates here?

There are 69 total failures in the last 7 days on linux1804-64-asan opt

Recent failure https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=56ce346234d1b5ec53b537ea22a9e545a21980ed&selectedTaskRun=MyBIKfOSSK6nmno6SGI5Sg.5 (log is unavailable)

We should definitely disable this stuff for now; I've been trying to get logs, but it's maddening, and transient.

Comment 17

[Intermittent Failures Robot]

18 failures in 809 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 7
• try: 2
• mozilla-beta: 5

Platform breakdown:

• linux1804-64-asan: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-09-01&endday=2020-09-01&tree=all

Comment 18

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1657084 - Skip huge-texture-allocation tests that stall Lin+ASAN jobs.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d22602f2feb7
Skip huge-texture-allocation tests that stall Lin+ASAN jobs. r=lsalzman

Comment 20

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/d22602f2feb7

Comment 21

[Intermittent Failures Robot]

57 failures in 2984 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 10
• autoland: 24
• try: 4
• mozilla-beta: 19

Platform breakdown:

• linux1804-64-asan: 57

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 22

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/releases/mozilla-beta/rev/3043baad48ff

Comment 23

[Intermittent Failures Robot]

8 failures in 3410 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 8

Platform breakdown:

• linux1804-64-asan: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 4130 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1657084&startday=2020-10-05&endday=2020-10-11&tree=all

Comment 25

[Intermittent Failures Robot]

1 failures in 1450 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux1804-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1657084&startday=2020-12-21&endday=2020-12-27&tree=all

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/97cd45ec5a15e4d2c8c257ad052f932511b4693f

Comment 27

[Intermittent Failures Robot]

1 failures in 3683 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• linux1804-64-asan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1657084&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 4255 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1657084&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 3224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1657084&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 30

[Kelsey Gilbert [:jgilbert]]

We can leave this for now, and handle re-enabling it in the future.