2018450 - Spike in failures on linux/win ccov

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[amarc]

Since this push from central, many jobs on linux/win ccov have started frequently failing, most of them are confirm failure jobs

Some examples: https://treeherder.mozilla.org/logviewer?job_id=549883508&repo=mozilla-central&task=WgXEQtpTS7uXgvrVI6Wx5A.0
https://treeherder.mozilla.org/logviewer?job_id=549883639&repo=mozilla-central&task=EBxMd4cpRzSo_SBU6UjxEg.0
https://treeherder.mozilla.org/logviewer?job_id=549924732&repo=mozilla-central&task=SetfD97-SwCEPkyCqIfOLQ.0

Comment 1

[amarc]

hi Aryx, do you know what's going on here?

Comment 8

[amarc]

hi Andrew, could these be caused by some kind of worker issue?

Comment 9

[Intermittent Failures Robot]

170 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 170

Table

	1proc-cf	1proc-swr-cf	async-cf	cf	cf-s	headless-cf	ioi-cf	no_variant	nofis-cf	nofis-swr-cf	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov			28	18	2	3	5	3	5	1		5	18
windows11-x86_64/ccov	4	2	16	23	2	2	2	1			2		7	7	4	8	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-21&endday=2026-02-21&tree=all

Comment 10

[Intermittent Failures Robot]

247 failures were associated with this bug in the last 7 days.
This is the #4 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 247

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr-cf	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov					40	23	2	3	5	13	1	7	1		9	27			1
windows10-x86_64/debug										1
windows11-x86_64/ccov	1	5	1	2	21	26	2	2	2	3				2	2	7	13	5	17	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-16&endday=2026-02-22&tree=all

Comment 12

[Andrew Erickson [:aerickson]]

The linux failures are on g-w cloud linux d2g hosts running the in-tree docker image gw-fxci-gcp-l1-2404-amd64-headless-googlecompute-2026-01-14.

Relops doesn't maintain the in-tree docker images. CC'ing :jcristau, who has been working on them a bit.

Comment 13

[Intermittent Failures Robot]

178 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 178

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless	no_variant	nofis-cf	nofis-swr-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig-cf
linux2404-x86_64/ccov					21	15			25	2	1				13	10
linux2404-x86_64/opt									1
windows11-x86_64/ccov	5	2	1	2	14	14	2	1	23			2	2	2	5	7	3	4	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-23&endday=2026-02-23&tree=all

Comment 14

[Julien Cristau [:jcristau]]

Are there specific failing tests (for each platform)? The linux worker image hasn't changed since mid-January.

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The count of browser-chrome failures on Linux CCov per push jumped and the tests listed in the duplicate bugs fail either frequently or permanently, e.g. High frequency ccov browser/base/content/test/tabcrashed/browser_aboutRestartRequired_buildid_false-positive.js | single tracking bug (bug 2018417).

Bug 2018418 failed - during the setup function of the test - before only (?) on Try pushes by :gsvelto (example) but no change by Gabriele is part of the changelog.

Gabriele, do you know what started these failures?

Comment 16

[Intermittent Failures Robot]

70 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 70

Table

	1proc	1proc-swr-cf	async-cf	cf	cf-s	msix	no_variant	nofis	s	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov	1		8	8			10	1			6	5
windows11-x86_64/ccov	3	1	8	4	1		3		1	1			3	2	1	2
windows11-x86_64/debug						1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-24&endday=2026-02-24&tree=all

Comment 17

[Gabriele Svelto [:gsvelto]]

I suspect it must have been because of a change in how these tasks are selected when running on try. I've got some pre-saved queries that I invoke with ./mach try fuzzy --preset <name> and they didn't include ccov tests, but didn't exclude them either. Recently the ccov tests are always running so I've been tweaking my queries to disable them explicitly unless I need them.

Comment 18

[Intermittent Failures Robot]

142 failures were associated with this bug yesterday.

Repository breakdown:

• autoland: 5
• mozilla-central: 137

Table

	1proc	async-cf	cf	cf-s	msix	no_variant	nofis	nofis-s	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc0-cf	wr-dc3-c-cf	xorig	xorig-cf
linux2404-x86_64/asan						1
linux2404-x86_64/ccov		12	21			16	1	1				10	9
windows11-x86/opt						4
windows11-x86_64/ccov	3	12	11	1		10			2	1	1	6	5	3	3	1	1	3	3
windows11-x86_64/debug					1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-25&endday=2026-02-25&tree=all

Comment 19

[Intermittent Failures Robot]

21 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 21

Table

	1proc	async-cf	cf	no_variant	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		4	4	4
windows11-x86_64/ccov	2		1	2	1	1	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-26&endday=2026-02-26&tree=all

Comment 20

[Intermittent Failures Robot]

189 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 189

Table

	1proc	async-cf	cf	ioi-cf	no_variant	nofis	nofis-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc2-o	xorig	xorig-cf
linux-x86_64/opt					1
linux2404-x86_64/ccov		12	19	7	18	2	2		1		10	12
windows11-x86_64/ccov	3	6	17	2	23			1	3	2	19	15	3	3	1	3	4

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-27&endday=2026-02-27&tree=all

Comment 21

[Intermittent Failures Robot]

127 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 127

Table

	1proc	async-cf	cf	cf-s	ioi-cf	no_variant	nofis	nofis-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc2-o-cf	xorig	xorig-cf
linux2404-x86_64/ccov		11	14		4	13	1	1				8	9
windows11-x86_64/ccov	3	8	14	1	1	13			1	2	2	7	7	2	2	1	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-28&endday=2026-02-28&tree=all

Comment 22

[Intermittent Failures Robot]

761 failures were associated with this bug in the last 7 days.
This is the #1 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• autoland: 5
• mozilla-central: 756

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless	ioi-cf	msix	no_variant	nofis	nofis-cf	nofis-s	nofis-swr-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc0-cf	wr-dc2-o	wr-dc2-o-cf	wr-dc3-c-cf	xorig	xorig-cf
linux-x86_64/opt											1
linux2404-x86_64/asan											1
linux2404-x86_64/ccov	1				72	90			11		94	5	5	1	1		1		52	52
linux2404-x86_64/opt											1
windows11-x86/opt											4
windows11-x86_64/ccov	19	2	1	3	48	61	5	1	3		75					7	8	8	37	34	15	15	1	1	1	1	9	12
windows11-x86_64/debug										2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-02-23&endday=2026-03-01&tree=all

Comment 23

[Intermittent Failures Robot]

133 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 133

Table

	async-cf	cf	ioi-cf	no_variant	nofis-cf	nofis-spi	nofis-spi-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf
linux2404-x86_64/ccov	13	19	9	11	1	1	1			1	12	10
windows11-x86_64/ccov	6	14	5	11				1	1	1	4	4	5	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-02&endday=2026-03-02&tree=all

Comment 25

[Intermittent Failures Robot]

127 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-esr140: 6
• mozilla-esr115: 1
• mozilla-central: 120

Table

	1proc	async-cf	cf	cf-s	ioi-cf	no_variant	nofis	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf
linux1804-x86_64/asan						1
linux1804-x86_64/opt									1
linux1804-x86_64/tsan						5
linux2404-x86_64/ccov		9	11		2	12	1				16	8
macosx1500-aarch64/opt						1
windows11-x86_64/ccov	5	5	14	1		13		1		1	3	14	2	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-03&endday=2026-03-03&tree=all

Comment 26

[Intermittent Failures Robot]

212 failures were associated with this bug yesterday.

Repository breakdown:

• autoland: 3
• mozilla-central: 209

Table

	1proc	async-cf	cf	ioi-cf	no_variant	nofis	nogpu	nogpu-cf	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		12	22	6	18	2					9	11			1
linux2404-x86_64/debug					2
linux2404-x86_64/tsan					1
macosx1500-aarch64/opt	2				5
windows11-x86_64/ccov	1	14	18		18		12	12	2	2	19	17	2	2	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-04&endday=2026-03-04&tree=all

Comment 27

[Intermittent Failures Robot]

378 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 378

Table

	1proc	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc3-c	xorig-cf
linux2404-x86_64/ccov				20	28		1	1	6	34	2	3					1	18	19
macosx1470-x86_64/debug										1
macosx1500-aarch64/opt										1
windows11-x86/opt										2
windows11-x86_64/ccov	8	1	3	25	45	2			3	39			22	25	1	4	4	23	28	3	3	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-05&endday=2026-03-05&tree=all

Comment 28

[Intermittent Failures Robot]

278 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 278

Table

	1proc	1proc-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig-cf
linux2404-x86_64/ccov			20	30	1	1	1	3	24	6	2	1			1			18	14
windows11-x86_64/ccov	5	1	18	25	1			1	24				15	16	1	3	3	17	17	4	2	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-06&endday=2026-03-06&tree=all

Comment 29

[Intermittent Failures Robot]

193 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 193

Table

	1proc-cf	async-cf	cf	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		9	16	2	2	14	4	2			1		8	9			1
windows11-x86_64/ccov	1	9	23		2	21			14	14	3	2	14	16	2	2	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-07&endday=2026-03-07&tree=all

Comment 30

[Intermittent Failures Robot]

1438 failures were associated with this bug in the last 7 days.
This is the #1 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-esr140: 6
• mozilla-esr115: 1
• autoland: 3
• mozilla-central: 1428

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-spi	nofis-spi-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	wr-dc3-c	xorig	xorig-cf
linux1804-x86_64/asan											1
linux1804-x86_64/opt																				1
linux1804-x86_64/tsan											5
linux2404-x86_64/ccov	1				89	136	1	2	4	34	121	16	8	1	2	1			1	1	2	87	83				2
linux2404-x86_64/debug											2
linux2404-x86_64/tsan											1
macosx1470-x86_64/debug											1
macosx1500-aarch64/opt	2										7
windows11-x86/opt											2
windows11-x86_64/asan											2						1			1
windows11-x86_64/ccov	19	2	1	3	85	148	5			13	135						69	74	5	14	14	87	102	20	15	1	2	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-02&endday=2026-03-08&tree=all

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I used the git revs to actually get a complete diff for all the changes of this merge to mozilla-central:
https://github.com/mozilla-firefox/firefox/compare/4a30b4e343d1..bfa2c2f2cb7c

My suspicion is that bug 2016051 actually caused this problem given that it touches quite a lot for CCOV builds and we only see these failures for CCOV builds.

Marco, can you please take a look?

Comment 33

[Marco Castelluccio [:marco]]

Bug 2016051 is adding a task at the end of all others, so it can't affect what previous tasks in the graph are doing.
The only regression it might have caused is how frequently we run the ccov tasks (see for example bug 2020718), but it can't affect the rate of failures.

Comment 34

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Marco Castelluccio [:marco] from comment #33)

The only regression it might have caused is how frequently we run the ccov tasks (see for example bug 2020718), but it can't affect the rate of failures.

What does frequent here mean? We did not run those jobs at all before, so what actually calculates the frequency?

Comment 35

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Note that especially the W-async-cf task (duped bug 2021821) is set to only run on try but not on any trunk branches. So as I read this is related to the frequency when the test is getting run, which jumped from never to always. And it's not related to the number of failures in such tasks - which are mostly expected due to underlying platform issues that need to get fixed.

Comment 36

[Marco Castelluccio [:marco]]

OK, I thought the bug was about a higher rate of failures of a task that was already being scheduled in the past. The solution to bug 2020718 would probably fix this too then.

Comment 37

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 2016051

Comment 38

[Intermittent Failures Robot]

167 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 166
• mozilla-beta: 1

Table

	1proc	async-cf	cf	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		7	13	1	2	17	9	2	1						10	16				1
linux2404-x86_64/opt						1
windows11-x86_64/asan										1
windows11-x86_64/ccov	3	4	14		1	16				11	11	1	1	1	10	9	2	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-09&endday=2026-03-09&tree=all

Comment 39

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Marco, do you think it might make sense to backout the regressor temporarily until the underlying taskgraph fix has landed and a new release it out? Or isn't it possible due to some other automation / tools depending on this jobs now?

Comment 40

[Marco Castelluccio [:marco]]

Unfortunately we can't because we already switched other automation to rely on that.

Comment 41

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Marco Castelluccio [:marco] from comment #40)

Unfortunately we can't because we already switched other automation to rely on that.

I thought so. Thanks for confirming. Let’s at least needinfo Andrew so he’s aware of this bug, as his taskgraph update work would likely help here significantly. Hopefully that can move forward quickly.

Comment 42

[Andrew Halberstadt [:ahal]]

Thanks, are you talking about https://github.com/taskcluster/taskgraph/pull/917 ? I'll try and get it ready for review

Comment 43

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrew Halberstadt [:ahal] from comment #42)

Thanks, are you talking about https://github.com/taskcluster/taskgraph/pull/917 ? I'll try and get it ready for review

Thank you! Yes, that's at least what Marco was pointing at, which should as well help for this bug (see comment 33).

Comment 44

[Intermittent Failures Robot]

214 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 214

Table

	1proc	1proc-swr	async-cf	cf	cf-s	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig
linux2404-x86_64/ccov	1		12	17	1	1	19	7	6		1	1	1	1	10	14
windows11-x86_64/ccov	1	1	6	27	1	1	26			8	11		2	2	16	13	2	2	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-10&endday=2026-03-10&tree=all

Comment 45

[Intermittent Failures Robot]

183 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 183

Table

	1proc	1proc-cf	async-cf	cf	cf-s	headless	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf
linux2404-x86_64/ccov	1	1	12	13		1	2	18	3	1	1			1			8	5
windows11-x86_64/ccov	1		7	17	1		2	24				15	14	2	2	3	10	14	2	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-11&endday=2026-03-11&tree=all

Comment 46

[Intermittent Failures Robot]

182 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 182

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	cf-vt	headless	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov	1	1			5	11		1	1	2	13	4	1	1					20	10			1	1
windows11-x86_64/ccov	2		1	1	10	18	1			2	17				14	12	2	2	13	6	2	2	2	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-12&endday=2026-03-12&tree=all

Comment 47

[Intermittent Failures Robot]

235 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 235

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov				1	16	18	1	1	5	27	4	2	1			1			13	15
windows11-x86_64/ccov	3	1	1	1	11	14	2		1	21				18	13	1	2	3	14	14	3	3	1	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-13&endday=2026-03-13&tree=all

Comment 48

[Intermittent Failures Robot]

147 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 144

Table

	1proc	1proc-swr	async-cf	cf	cf-s	headless	ioi-cf	no_variant	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/asan															3
linux2404-x86_64/ccov		1	8	10		2	1	16	2				1		8	10
windows11-x86_64/ccov	1	1	11	11	1		1	12		11	13	1	1	1	6	8	2	2	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-14&endday=2026-03-14&tree=all

Comment 49

[Intermittent Failures Robot]

1223 failures were associated with this bug in the last 7 days.
This is the #1 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-beta: 4
• mozilla-central: 1219

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	cf-vt	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nofis-swr-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/asan																						3
linux2404-x86_64/ccov	3	2	1	1	69	94	2	1	4	3	14	119	28	17	3	1		1	3	2	1	78	76			1	3
linux2404-x86_64/opt												1
windows11-x86_64/asan																	1
windows11-x86_64/ccov	12	1	4	2	52	112	7				9	119					80	80	5	10	13	71	73	14	13	8	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-09&endday=2026-03-15&tree=all

Comment 50

[Intermittent Failures Robot]

156 failures were associated with this bug yesterday.

Repository breakdown:

• autoland: 1
• mozilla-central: 155

Table

	1proc	1proc-swr	async-cf	cf	cf-s	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig-cf
linux2404-x86_64/ccov			6	13	1	2	12	2	3					11	11
linux2404-x86_64/opt							1
macosx1015-x86_64/debug							1
windows11-x86_64/ccov	2	1	9	15		1	20			11	11	1	1	11	6	1	2	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-16&endday=2026-03-16&tree=all

Comment 51

[Intermittent Failures Robot]

334 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 334

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-spi-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	vt	wmfme	wmfme-cf	wr-dc0-cf	wr-dc1-p-cf	wr-dc2-o-cf	xorig-cf
linux2404-x86_64/asan																				1
linux2404-x86_64/ccov					23	29	1	2	1	3	28	3	3	1			1			19	20	1						1
windows11-x86_64/ccov	6	1	1	1	17	30	2	1		2	38				20	18	3	3	4	22	18		2	3	1	1	1	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-17&endday=2026-03-17&tree=all

Comment 52

[Intermittent Failures Robot]

296 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 296

Table

	1proc	1proc-swr-cf	async-cf	cf	cf-s	headless	ioi-cf	msix	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
android-x86_64/opt										1
linux2404-x86_64/ccov			24	30		1	3		24	7	4			1		1	13	16			1
windows11-x86_64/ccov	3	1	16	18	2		1		27			18	17	2	3	3	20	22	3	3	2	2
windows11-x86_64/opt								1	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-18&endday=2026-03-18&tree=all

Comment 53

[Intermittent Failures Robot]

199 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 199

Table

	1proc	1proc-cf	async-cf	cf	cf-s	emewmf	emewmf-cf	headless-cf	ioi-cf	no_variant	nofis	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig
linux2404-x86_64/ccov			14	18	3			1	2	20	3	1		2	1		13	15
windows11-x86_64/ccov	3	1	8	19	1	1	1		1	21		11	11	1	2	2	10	8	2	2	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-19&endday=2026-03-19&tree=all

Comment 54

[Intermittent Failures Robot]

114 failures were associated with this bug yesterday.

Repository breakdown:

• autoland: 1
• mozilla-central: 113

Table

	1proc	async-cf	cf	headless	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr-cf	nogpu	nogpu-cf	s	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov	1	16	19	1	2	11	2	4	1			1	10	7
linux2404-x86_64/debug													1
windows11-x86_64/ccov	1	3	6			6				1	1		7	5	1	1	3	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-20&endday=2026-03-20&tree=all

Comment 55

[Intermittent Failures Robot]

130 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 130

Table

	1proc	1proc-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	no_variant	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig
linux2404-x86_64/ccov	1	1	8	9		2	1	2	16	4	1	1					14	15
windows11-x86_64/ccov	1		5	8	1				17			1	1	1	2	1	7	7	1	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-21&endday=2026-03-21&tree=all

Comment 56

[Intermittent Failures Robot]

1229 failures were associated with this bug in the last 7 days.
This is the #1 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 1227
• autoland: 2

Table

	1proc	1proc-cf	1proc-swr	1proc-swr-cf	async-cf	cf	cf-s	emewmf	emewmf-cf	headless	headless-cf	ioi-cf	msix	no_variant	nofis	nofis-cf	nofis-spi-cf	nofis-swr	nofis-swr-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	vt	wmfme	wmfme-cf	wr-dc0-cf	wr-dc1-p-cf	wr-dc2-o-cf	xorig	xorig-cf
android-x86_64/opt															1
linux2404-x86_64/asan																									1
linux2404-x86_64/ccov	2	1			91	118	5			6	3	14		111	17	18	1	1	1	2		5	1	1	80	84	1						1	1
linux2404-x86_64/debug																									1
linux2404-x86_64/opt														1
macosx1015-x86_64/debug														1
windows11-x86_64/ccov	16	2	2	2	58	96	6	1	1	1		5		129						62	59	7	11	11	77	66		10	12	1	1	1	7	8
windows11-x86_64/opt													1	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-16&endday=2026-03-22&tree=all

Comment 57

[Intermittent Failures Robot]

155 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 155

Table

	1proc	async-cf	cf	cf-s	cf-vt	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	vt	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		9	17	1		1	1	13	1	2						14	10
windows11-x86_64/ccov	2	10	11	1	1		7	19			1	2	1	1	1	12	12	1	1	1	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-23&endday=2026-03-23&tree=all

Comment 58

[Intermittent Failures Robot]

109 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 109

Table

	1proc	1proc-cf	1proc-swr	async-cf	cf	cf-s	cf-vt	ioi-cf	no_variant	nofis	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	vt	wmfme	wmfme-cf
linux2404-x86_64/ccov				3	6			3	7	2				1		5	9
windows11-x86_64/ccov	2	1	1	8	13	1	2	1	13		2	2	1	1	2	9	9	3	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-24&endday=2026-03-24&tree=all

Comment 59

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 2016051

Comment 60

[Intermittent Failures Robot]

391 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 391

Table

	1proc	1proc-swr	async-cf	cf	cf-s	headless	ioi-cf	no_variant	nofis	nofis-cf	nofis-spi-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	trainhop-rel	trainhop-rel-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov	1		22	36	2	1	6	39	13	6	1	1			2	1		25	28	2	2
windows11-x86_64/ccov	6	2	17	38	2		2	47					3	3	1	3	3	35	33			3	3	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-25&endday=2026-03-25&tree=all

Comment 61

[Intermittent Failures Robot]

151 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 151

Table

	1proc	1proc-swr	async-cf	cf	headless	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	vt	wmfme	wmfme-cf
linux2404-x86_64/ccov	1		14	17	2	1	15	4	5	2						12	16	1
windows11-x86_64/ccov	3	1	9	15		1	9				3	2	1	1	1	4	7		2	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-26&endday=2026-03-26&tree=all

Comment 62

[Intermittent Failures Robot]

128 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 128

Table

	1proc	async-cf	cf	cf-s	cf-vt	headless	ioi-cf	no_variant	nofis	nofis-swr	nogpu	nogpu-cf	spi	spi-cf	swr	swr-cf	trainhop-rel	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		9	13		1	2	2	14	2	1					7	8
linux2404-x86_64/opt									1						1
windows11-x86_64/ccov	1	7	10	1			2	14			1	2	1	1	8	10	1	1	1	3	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-27&endday=2026-03-27&tree=all

Comment 63

[Intermittent Failures Robot]

219 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 219

Table

	1proc	async-cf	cf	cf-s	headless	ioi-cf	no_variant	nofis	nofis-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		19	21		1	2	18	4	3						18	12
windows11-x86_64/ccov	4	10	22	1		3	19			9	11	1	2	4	13	15	1	2	3	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-28&endday=2026-03-28&tree=all

Comment 64

[Intermittent Failures Robot]

1283 failures were associated with this bug in the last 7 days.
This is the #1 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 1283

Table

	1proc	1proc-cf	1proc-swr	async-cf	cf	cf-s	cf-vt	headless	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-spi-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	trainhop-rel	trainhop-rel-cf	vt	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov	3		1	91	132	4	1	6	1	17	118	30	17	1	4			3	2		90	95	2	2	1				1
linux2404-x86_64/opt												1									1
windows11-x86_64/ccov	19	1	4	68	114	6	3			18	125					27	31	5	9	13	83	89	1		4	11	12	9	7

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-23&endday=2026-03-29&tree=all

Comment 65

[Intermittent Failures Robot]

221 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 221

Table

	1proc	1proc-swr-cf	async-cf	cf	cf-s	headless	ioi-cf	msix	no_variant	nofis	nofis-cf	nofis-swr	nofis-swr-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/ccov		1	12	21	1	1	3		25	8	4	3	1	1		1			16	19			1	1
windows11-x86_64/asan									1
windows11-x86_64/ccov	3		7	16	1		2		20					11	10		2	1	12	8	2	2	1	1
windows11-x86_64/opt								1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-30&endday=2026-03-30&tree=all

Comment 66

[Intermittent Failures Robot]

25 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 25

Table

	async-cf	cf	ioi-cf	no_variant	nofis	nofis-cf	swr	swr-cf
linux2404-x86_64/asan							2
linux2404-x86_64/ccov	2	5	1	5	2	1	3	2
windows11-x86_64/ccov		1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-31&endday=2026-03-31&tree=all

Comment 67

[amarc]

looks like it's fixed

Comment 68

[Intermittent Failures Robot]

124 failures were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 124

Table

	1proc	async-cf	cf	headless-cf	ioi-cf	no_variant	nofis	nofis-cf	nofis-swr	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig
linux2404-x86_64/ccov		5	11	1		10	4	2	1			4			8	5			1
windows11-x86_64/ccov	4	7	9		1	20				6	6	1	1	1	8	6	1	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-04-01&endday=2026-04-01&tree=all

Comment 69

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Amazing. Thanks a lot jcristau!

And yes, the number of jobs for CCOV builds drastically reduced:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=ccov&fromchange=81bd8e73cd80793f8d49b769880b6a954b8aa962&tochange=a5514d76e0e7d3f3b0a6afdee8b8bf8d29d2d129

I assume that with this change we still run all those jobs that we actually should run.

Marco, maybe you have the overview and could take a look?

Comment 70

[Intermittent Failures Robot]

377 failures were associated with this bug in the last 7 days.
This is the #2 most frequent failure this week.

This failure happened more than 75 times this week! Resolving this bug is a very high priority.

Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 377

Table

	1proc	1proc-swr-cf	async-cf	cf	cf-s	headless	headless-cf	ioi-cf	msix	no_variant	nofis	nofis-cf	nofis-swr	nofis-swr-cf	nogpu	nogpu-cf	s	spi	spi-cf	swr	swr-cf	wmfme	wmfme-cf	xorig	xorig-cf
linux2404-x86_64/asan																				2
linux2404-x86_64/ccov		1	19	37	1	1	1	4		40	14	7	4	1	1		5			27	26			2	1
windows11-x86_64/asan										1
windows11-x86_64/ccov	7		14	26	1			4		48					17	16	1	3	2	20	14	3	3	1	1
windows11-x86_64/opt									1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-03-30&endday=2026-04-05&tree=all

Comment 71

[Marco Castelluccio [:marco]]

We are looking good!

Comment 72

[Intermittent Failures Robot]

13 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Table

	1proc	no_variant	standalone
linux2404-x86_64/opt			1
windows11-x86_64/ccov	1	11

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2018450&startday=2026-04-08&endday=2026-04-14&tree=all