1608388 - Can we avoid running MinGW 32 bit builds on autoland?

Status: RESOLVED FIXED

(Testing :: General, enhancement, P3)

Description

[Marco Castelluccio [:marco]]

Are they providing more value compared to normal Windows 32 bit builds? Wouldn't running them on mozilla-central be enough?

They are taking ~50 minutes per push.

Comment 1

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

(In reply to Marco Castelluccio [:marco] from comment #0)

Are they providing more value compared to normal Windows 32 bit builds? Wouldn't running them on mozilla-central be enough?

They are taking ~50 minutes per push.

Can you explain the impact of this change to me? The MinGW builds frequently break when the Windows builds do not because they use different SDKs. So I think this would very likely cause a lot of build failures to be missed before we land patches to central and cause central backouts.

On a random autoland, I see
x64 debug 23 min
x64 opt 44 min
x86 debug 31 min
x86 opt 39 min

What we could do is only run the debug builds (and tests) on autoland and then run the opt builds on central. It's rare something would break opt and not debug. And presently we only run a single test on the opt build; the full test suite (which is a partial tests suite, everything we can enable right now) we run for MinGW is run for debug (which is for cost savings.)

Comment 2

[Marco Castelluccio [:marco]]

(In reply to Tom Ritter [:tjr] (ni for response to sec-[approval|rating|advisories|cve]) from comment #1)

Can you explain the impact of this change to me?

The idea is to reduce CI costs without increasing the backout rate (or increasing it only slightly).

The MinGW builds frequently break when the Windows builds do not because they use different SDKs. So I think this would very likely cause a lot of build failures to be missed before we land patches to central and cause central backouts.

Do the 32 bit and the 64 bit build both break in those cases? Or would running just the 64 bit configuration catch these kinds of failures?

On a random autoland, I see
x64 debug 23 min
x64 opt 44 min
x86 debug 31 min
x86 opt 39 min

What we could do is only run the debug builds (and tests) on autoland and then run the opt builds on central. It's rare something would break opt and not debug. And presently we only run a single test on the opt build; the full test suite (which is a partial tests suite, everything we can enable right now) we run for MinGW is run for debug (which is for cost savings.)

This sounds like a good option, we'd be saving almost 80 minutes per push. If we can also avoid running 32 bit debug build and tests it would be even better.

Comment 3

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

(In reply to Marco Castelluccio [:marco] from comment #2)

The MinGW builds frequently break when the Windows builds do not because they use different SDKs. So I think this would very likely cause a lot of build failures to be missed before we land patches to central and cause central backouts.

Do the 32 bit and the 64 bit build both break in those cases? Or would running just the 64 bit configuration catch these kinds of failures?

No actually the frequently only the x86 or the x64 build would break so I think we would want both. (I'd be fine being corrected with data though!)

What we could do is only run the debug builds (and tests) on autoland and then run the opt builds on central. It's rare something would break opt and not debug. And presently we only run a single test on the opt build; the full test suite (which is a partial tests suite, everything we can enable right now) we run for MinGW is run for debug (which is for cost savings.)

This sounds like a good option, we'd be saving almost 80 minutes per push. If we can also avoid running 32 bit debug build and tests it would be even better.

My impression is that we should keep both x86 and x64. I see most but not all autoland backouts of MinGW and it happens probably once every couple weeks. Somewhere we could probably dig up statistics... https://mozilla.logbot.info/?q=backed+MinGW&chs=%23developers&w=a is probably a good quick overview though.

Comment 4

[Marco Castelluccio [:marco]]

Unfortunately, because of the way autoland schedules tasks, we can't easily gather statistics. When you see a failure happening on x64 only, it might simply be because the x86 equivalent task was not scheduled. And the other way around.
We could collect statistics on full autoland pushes only (one every five pushes, recently one every ten pushes).

Anyway, simply disabling opt and keeping both x86 and x64 debug sounds fine for now.

Comment 5

[Marco Castelluccio [:marco]]

In the data I have, which is not fully complete (it will be hopefully in a couple of weeks), out of 2966 runs (71 failing) where both mingw32 and mingw64 run, there were 21 exclusive failures for mingw32 and 20 exclusive failures for mingw64. Once the data is complete, I will run an analysis checking if there are failures specific to mingw32 that wouldn't be caught by either mingw64 or normal win 32.

:bc, could you gather the costs for these tasks (possibly from November, so they don't include Christmas holidays or all hands), so we can better choose what to do?

Comment 6

[Marco Castelluccio [:marco]]

And I've filed bug 1613438 for running debug only and not opt.

Comment 7

[Bob Clary [:bc] (inactive)]

Costs didn't really start in a useful fashion until 2019-11-10. I do have data for try and autoland for the full Firefox 72 release cycle. This does include Christmas and New Years but can be scaled to a year if we treat it like a normal cycle. I also have full data for January which you might consider a proxy for a typical month.

Let's do January 1-31, 2020 for autoland, mingw, opt and debug:

provisionerId	workerType	project	tier	suite	groupSymbol	symbol	collection	cost	label
gecko-3	b-linux	autoland	2		WMC32	Bd	all	264.49	build-win32-mingwclang/debug
gecko-3	b-linux	autoland	2		WMC32	Bo	all	352.98	build-win32-mingwclang/opt
gecko-3	b-linux	autoland	2		WMC64	Bd	all	266.02	build-win64-mingwclang/debug
gecko-3	b-linux	autoland	2		WMC64	Bo	all	338.38	build-win64-mingwclang/opt
gecko-t	t-linux-xlarge	autoland	1		misc	mingw	opt	6.19	source-test-mozlint-mingw-cap
gecko-t	t-win10-64	autoland	2	cppunittest	nil	cppunit	debug	2.01	test-windows10-64-mingwclang/debug-cppunit-1proc
gecko-t	t-win10-64	autoland	2	firefox-ui-functional-local	Fxfn-l	en-US	debug	6.95	test-windows10-64-mingwclang/debug-firefox-ui-functional-local-e10s
gecko-t	t-win10-64	autoland	2	firefox-ui-functional-remote	Fxfn-r	en-US	debug	4.01	test-windows10-64-mingwclang/debug-firefox-ui-functional-remote-e10s, test-windows10-64/debug-firefox-ui-functional-remote-e10s
gecko-t	t-win10-64	autoland	2	mochitest-a11y	M-1proc	a11y	debug	5.44	test-windows10-64-mingwclang/debug-mochitest-a11y-1proc
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-plain-gpu	M	gpu	debug	8.27	test-windows10-64-mingwclang/debug-mochitest-gpu-e10s
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl1-core	M	gl1c	debug	104.14	test-windows10-64-mingwclang/debug-mochitest-webgl1-core-e10s
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl1-ext	M	gl1e	debug	39.90	test-windows10-64-mingwclang/debug-mochitest-webgl1-ext-e10s
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl2-core	M	gl2c	debug	26.11	test-windows10-64-mingwclang/debug-mochitest-webgl2-core-e10s
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl2-ext	M	gl2e1	debug	38.12	test-windows10-64-mingwclang/debug-mochitest-webgl2-ext-e10s-1
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl2-ext	M	gl2e2	debug	22.46	test-windows10-64-mingwclang/debug-mochitest-webgl2-ext-e10s-2
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl2-ext	M	gl2e3	debug	30.33	test-windows10-64-mingwclang/debug-mochitest-webgl2-ext-e10s-3
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgl2-ext	M	gl2e4	debug	31.59	test-windows10-64-mingwclang/debug-mochitest-webgl2-ext-e10s-4
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-webgpu	M	webgpu	debug	5.46	test-windows10-64-mingwclang/debug-mochitest-webgpu-e10s
gecko-t	t-win10-64-gpu-s	autoland	2	reftest	R	R1	debug	63.57	test-windows10-64-mingwclang/debug-reftest-e10s-1
gecko-t	t-win10-64-gpu-s	autoland	2	reftest	R	R2	debug	65.82	test-windows10-64-mingwclang/debug-reftest-e10s-2
gecko-t	t-win10-64-gpu-s	autoland	2	reftest	R	R3	debug	64.18	test-windows10-64-mingwclang/debug-reftest-e10s-3
gecko-t	t-win10-64-gpu-s	autoland	2	reftest	R	R4	debug	63.71	test-windows10-64-mingwclang/debug-reftest-e10s-4
gecko-t	t-win10-64	autoland	2	telemetry-tests-client	tt	c	debug	2.58	test-windows10-64-mingwclang/debug-telemetry-tests-client-e10s
gecko-t	t-win10-64	autoland	2	cppunittest	nil	cppunit	opt	1.87	test-windows10-64-mingwclang/opt-cppunit-1proc
gecko-t	t-win10-64-gpu-s	autoland	2	mochitest-plain-gpu	M	gpu	opt	6.93	test-windows10-64-mingwclang/opt-mochitest-gpu-e10s
gecko-t	t-win7-32	autoland	2	cppunittest	nil	cppunit	debug	5.43	test-windows7-32-mingwclang/debug-cppunit-1proc
gecko-t	t-win7-32	autoland	2	firefox-ui-functional-local	Fxfn-l	en-US	debug	24.10	test-windows7-32-mingwclang/debug-firefox-ui-functional-local-e10s
gecko-t	t-win7-32	autoland	2	firefox-ui-functional-remote	Fxfn-r	en-US	debug	10.37	test-windows7-32-mingwclang/debug-firefox-ui-functional-remote-e10s, test-windows7-32/debug-firefox-ui-functional-remote-e10s
gecko-t	t-win7-32	autoland	2	mochitest-a11y	M-1proc	a11y	debug	10.43	test-windows7-32-mingwclang/debug-mochitest-a11y-1proc
gecko-t	t-win7-32-gpu	autoland	2	mochitest-plain-gpu	M	gpu	debug	15.00	test-windows7-32-mingwclang/debug-mochitest-gpu-e10s
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl1-core	M	gl1c	debug	21.69	test-windows7-32-mingwclang/debug-mochitest-webgl1-core-e10s
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl1-ext	M	gl1e	debug	178.79	test-windows7-32-mingwclang/debug-mochitest-webgl1-ext-e10s
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl2-core	M	gl2c	debug	24.56	test-windows7-32-mingwclang/debug-mochitest-webgl2-core-e10s
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl2-ext	M	gl2e1	debug	30.93	test-windows7-32-mingwclang/debug-mochitest-webgl2-ext-e10s-1
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl2-ext	M	gl2e2	debug	22.70	test-windows7-32-mingwclang/debug-mochitest-webgl2-ext-e10s-2
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl2-ext	M	gl2e3	debug	27.11	test-windows7-32-mingwclang/debug-mochitest-webgl2-ext-e10s-3
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgl2-ext	M	gl2e4	debug	28.46	test-windows7-32-mingwclang/debug-mochitest-webgl2-ext-e10s-4
gecko-t	t-win7-32-gpu	autoland	2	mochitest-webgpu	M	webgpu	debug	11.32	test-windows7-32-mingwclang/debug-mochitest-webgpu-e10s
gecko-t	t-win7-32-gpu	autoland	2	reftest	R	R1	debug	51.69	test-windows7-32-mingwclang/debug-reftest-e10s-1
gecko-t	t-win7-32-gpu	autoland	2	reftest	R	R2	debug	199.88	test-windows7-32-mingwclang/debug-reftest-e10s-2
gecko-t	t-win7-32-gpu	autoland	2	reftest	R	R3	debug	51.65	test-windows7-32-mingwclang/debug-reftest-e10s-3
gecko-t	t-win7-32-gpu	autoland	2	reftest	R	R4	debug	52.31	test-windows7-32-mingwclang/debug-reftest-e10s-4
gecko-t	t-win7-32	autoland	2	telemetry-tests-client	tt	c	debug	5.71	test-windows7-32-mingwclang/debug-telemetry-tests-client-e10s
gecko-t	t-win7-32	autoland	2	cppunittest	nil	cppunit	opt	5.24	test-windows7-32-mingwclang/opt-cppunit-1proc
gecko-t	t-win7-32-gpu	autoland	2	mochitest-plain-gpu	M	gpu	opt	73.50	test-windows7-32-mingwclang/opt-mochitest-gpu-e10s
								2672.39

restricting it to debug only gives 1356.79

Comment 8

[Marco Castelluccio [:marco]]

(In reply to Bob Clary [:bc:] from comment #7)

Costs didn't really start in a useful fashion until 2019-11-10. I do have data for try and autoland for the full Firefox 72 release cycle. This does include Christmas and New Years but can be scaled to a year if we treat it like a normal cycle. I also have full data for January which you might consider a proxy for a typical month.

January was quite a peculiar month because of the All Hands + what happened before the All Hands (for the two weeks before the All Hands the code review bot found ~20% of the issues that it finds on a normal week).
2019-11-10 to 2019-12-10 might be a better period.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bugs fixed by week:

Last day of Su-Sa week, bugs fixed in Firefox-ish products
2/1/2020 137
1/25/2020 199
1/18/2020 208
1/11/2020 207
1/4/2020 96
12/28/2019 64
12/21/2019 225
12/14/2019 224
12/7/2019 218
11/30/2019 190
11/23/2019 204
11/16/2019 208
11/9/2019 251
11/2/2019 229
10/26/2019 221
10/19/2019 230
10/12/2019 273
10/5/2019 234

Comment 10

[Bob Clary [:bc] (inactive)]

I have no clue as to what is expected here. I suggest someone else who does, take over.

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #5)

Once the data is complete, I will run an analysis checking if there are failures specific to mingw32 that wouldn't be caught by either mingw64 or normal win 32.

I'm going to do this soon and then we can decide what to do.

Comment 12

[Marco Castelluccio [:marco]]

I've analyzed 22328 pushes (using the script from bug 1612544 1), checking if mingw32 was able to find any regression that couldn't already be found by either mingw64, win32 or win64. Out of 22328 pushes, there were 693 pushes with failures to analyze. We had 0 unique failures of mingw32.
I'd say we can move it to run on mozilla-central only.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

:ckerschb, as :marco points out in comment 12, we have very low risk of finding a mingw32 only regression- would it be ok to run this on mozilla-central only? Are there consumers of this build that in the rare case there is a regression could live with it for a day or two?

Comment 14

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #13)

:ckerschb, as :marco points out in comment 12, we have very low risk of finding a mingw32 only regression- would it be ok to run this on mozilla-central only? Are there consumers of this build that in the rare case there is a regression could live with it for a day or two?

Sounds fine. :)

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks Tom.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1608388 - Reduce MingW32 builds/tests to be run on m-c only. r=bc

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5631e8bb6f18
Reduce MingW32 builds/tests to be run on m-c only. r=bc

Comment 18

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/5631e8bb6f18

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

the tests stopped, but we still run:
build-win32-mingwclang/debug

we also run:
build-win64-mingwclang/debug

:marco, should the 64 bit debug build be run on autoland? If so we can do it on the backstop (10th push), if not, it can be m-c.

Comment 20

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #19)

the tests stopped, but we still run:
build-win32-mingwclang/debug

The build can be moved to mozilla-central too, it also didn't find any failure that the other builds couldn't find.

we also run:
build-win64-mingwclang/debug

:marco, should the 64 bit debug build be run on autoland? If so we can do it on the backstop (10th push), if not, it can be m-c.

I also couldn't find any failure of mingw64 that didn't also happen on win64 or win32. So maybe we can move it on the backstop to be safe, and move to m-c if the trend continues?

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1608388 - reduce frequency of mingw* builds. r=bc

reduce frequency of mingw* builds.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5b0db29d3faf
reduce frequency of mingw* builds. r=bc

Comment 23

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/5b0db29d3faf