mochitest-webgl tests are retried too often

RESOLVED FIXED

Status

defect
RESOLVED FIXED
2 years ago
6 months ago

People

(Reporter: marco, Assigned: marco)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Assignee

Description

2 years ago
Most mochitest-webgl chunks are retried several times until they fail.
Assignee

Comment 1

2 years ago
(In reply to Marco Castelluccio [:marco] from comment #0)
> Most mochitest-webgl chunks are retried several times until they fail.

Well, the tests don't actually fail, they never start.
Assignee

Comment 2

2 years ago
Do you know what might be causing this?
Flags: needinfo?(rthijssen)
most likely bug 1372172
Flags: needinfo?(rthijssen)
Assignee

Updated

2 years ago
Depends on: 1372172
Assignee

Comment 4

2 years ago
Rob, are you still working on bug 1372172?
Any idea why this is happening so often with windows10-64-ccov/debug compared to other builds (e.g. windows10-64/debug)? Take a look at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=3&filter-tier=2&filter-tier=1&filter-searchStr=win%2010%2064%20debug.
Flags: needinfo?(rthijssen)
we don't know what causes it yet but when the gpu gets used by a test, the instance often hangs (impaired state in ec2). before i went on pto i thought it might be related to the theme change (windows falling back to classic theme in the middle of some gpu intensive task), from bug 1343049 but the patch there seems to have not had any effect. so yes, now that i'm back from pto i am looking at this again.
Flags: needinfo?(rthijssen)
Assignee

Comment 6

2 years ago
Posted patch PatchSplinter Review
Attachment #8934939 - Flags: review?(jmaher)
Assignee

Updated

2 years ago
Keywords: leave-open
Comment on attachment 8934939 [details] [diff] [review]
Patch

Review of attachment 8934939 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for the comment linking to the bug
Attachment #8934939 - Flags: review?(jmaher) → review+

Comment 8

2 years ago
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5d29d8150f97
Disable mochitest-webgl tests as they are retried too often, until the infrastructure problem is fixed. r=jmaher

Comment 9

2 years ago
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/eb74c83f92c2
Add another space before comment in yaml file to avoid linting failure. r=me
Assignee

Comment 11

a year ago
:grenade, mochitest-gpu are affected by the same problem (they never start). Do you think it's bug 1372172? Isn't bug 1372172 intermittent? This problem with mochitest-gl and mochitest-gpu is permanent with coverage builds.
i guess it could be the same issue if they're starting, but we lose contact with the instance before the task start is shown in treeherder. if you can link to a treeherder job where the tests should have started, but didn't, i can investigate.
Assignee

Comment 13

a year ago
(In reply to Rob Thijssen (:grenade UTC+2) from comment #12)
> i guess it could be the same issue if they're starting, but we lose contact
> with the instance before the task start is shown in treeherder. if you can
> link to a treeherder job where the tests should have started, but didn't, i
> can investigate.

Here it is: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=d75d996016dcf325c2db2ed8a47af512d07ffacd&filter-searchStr=win%20ccov%20gpu.
Assignee

Updated

a year ago
Duplicate of this bug: 1452507
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 17

a year ago
I've tried another try push today for mochitest-webgl, still failing: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c57c6fe3ba01f2e3e62b34d6d02ca77c573fe2ac.
Assignee

Comment 18

6 months ago
The suites are passing now that bug 1372172 has been fixed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d94fa5d9d9e6cbd707bbffb5d960e7d92a72ad4e.
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #9027519 - Flags: review?(jmaher)
Comment on attachment 9027519 [details] [diff] [review]
Re-enable test suites

Review of attachment 9027519 [details] [diff] [review]:
-----------------------------------------------------------------

great, I am excited to see all the jobs that failed on win10 getting enabled now:)
Attachment #9027519 - Flags: review?(jmaher) → review+

Comment 20

6 months ago
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/102c7135d952
Re-enable some mochitest suites using virtual-with-gpu that were failing with Windows ccov builds. r=jmaher
Assignee

Updated

6 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 6 months ago
Keywords: leave-open
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.