1599197 - demote linux32 to tier-2, run builds not on autoland and try by default, only m-c

Status: RESOLVED FIXED

(Testing :: General, task, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently we do linux32 builds on try, autoland, m-c and on m-c run wpt tests. We should stop running builds on autoland and by default on try.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we spend about $2700/month on linux32 builds on autoland and another $1100/month on Try.

Assuming 95% reduction on Autoland and 50% reduction on try we should see ~$3100/month or $37000/year in savings.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

it appears we also have some unittests that run on autoland:

• linux32-shippable :: WPT()
• linux32-debug :: sm(arm)
• linux32-buildrusttest

I assume when the builds stop, the process for pgo, build-signing, repackage-linux, and upload-generated-sources will automatically stop.

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

Tasks:

• restrict linux32 builds to try and mozilla-central
• restrict linux32 test jobs to try and mozilla-central
• reduce linux32 test job to tier 2

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

:jmaher, I suspect the repackage tasks would need to have these lines removed?
https://searchfox.org/mozilla-central/source/taskcluster/ci/repackage/kind.yml#19-22

I think we can try landing a patch to remove linux32-shippable/opt builds from all repos other than mozilla-central and try, then see if we need to take the additional step of removing those lines.

Build-signing seems to be automatic if a shippable build is made, so presumably if the build no longer occurs on autoland then the signing should not occur.

Same as build-signing for upload-generated-sources.

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1599197 - demote linux32-shippable/opt tasks to tier 2 and limit builds to try and mozilla-central

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

Google group post: https://groups.google.com/forum/#!topic/mozilla.dev.platform/dsVeni30-38

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1410db0871ba
demote linux32-shippable/opt tasks to tier 2 and limit builds to try and mozilla-central r=jmaher,gbrown

Comment 8

[Andrei Ciure[:aciure]]

Backed out changeset 1410db0871ba (bug 1599197) for causing linting failures

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=283651362&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=1410db0871ba5a00a43906437e0050b19bc363bc

backout: https://hg.mozilla.org/integration/autoland/rev/5e47df675103bcf1914b61c2060b7c06ab31e290

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dfa46920fe00
demote linux32-shippable/opt tasks to tier 2 and limit builds to try and mozilla-central r=jmaher,gbrown

Comment 10

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/dfa46920fe00

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

we still do a lot of linux32 work:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=build%2Clinux&fromchange=0d79d00528a411a5bd3e8d11944fc1609dccb283&tochange=12b3c389fd6dc093a05a7f3f8f11c7cc094f0a8a

we have linux32 debug and pgo builds continuing, I think the rust stuff is taken care of in another patch.

:egao, can you make another pass at disabling the linux32 debug and pgo builds on autoland?

Comment 12

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #11)

we still do a lot of linux32 work:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=build%2Clinux&fromchange=0d79d00528a411a5bd3e8d11944fc1609dccb283&tochange=12b3c389fd6dc093a05a7f3f8f11c7cc094f0a8a

we have linux32 debug and pgo builds continuing, I think the rust stuff is taken care of in another patch.

:egao, can you make another pass at disabling the linux32 debug and pgo builds on autoland?

Yeah, I thought I'd need at least another pass or two to get everything down. Will do.

Comment 13

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1599197 - do not run linux32 on autoland or try by default

Comment 14

[Edwin Takahashi (:egao | infrequent contributor)]

One source of truth in this bug:

linux32 builds need to continue running on the following:

• mozilla-central
• beta
• release
• esr

linux32 builds will not run by default on:

• try
• autoland

Comment 15

[Edwin Takahashi (:egao | infrequent contributor)]

Question - how about the linux-gcp builds, should they also be restricted? Currently, linux-gcp/debug does not have any run-on-projects annotation.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/701fd0aec286
do not run linux32 on autoland or try by default r=jmaher,gbrown

Comment 17

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/701fd0aec286

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

gcp should be scheduled as tier-3 and on the same branches/frequencies as aws builds- I believe we are switching to gcp builds next week. So yes, lets move them to not run on autoland by default.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

In addition, I still see the PGO process (4 jobs) running for linux shippable and generating builds. In some cases there is a signing job as well. Almost there.

Comment 20

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1599197 - temporarily skip mfenced-12.html as it is interfering with the ubuntu1804 migration

Comment 21

[Phabricator Automation]

Comment on attachment 9119827 [details]
Bug 1599197 - temporarily skip mfenced-12.html as it is interfering with the ubuntu1804 migration

Revision D59384 was moved to bug 1602863. Setting attachment 9119827 [details] to obsolete.

Comment 22

[Edwin Takahashi (:egao | infrequent contributor)]

Strange. The changes I made to https://searchfox.org/mozilla-central/source/taskcluster/ci/instrumented-build/kind.yml#51 should have turned off PGO builds.

Comment 23

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1599197 - turn off linux 32bit jobs on autoland, part 3

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c079dea037f7
turn off linux 32bit jobs on autoland, part 3 r=jmaher

Comment 25

[Edwin Takahashi (:egao | infrequent contributor)]

I think the third patch did it.

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&tier=1%2C2%2C3&searchStr=linux&revision=c079dea037f7429780dc8c8818b610d5658150c5

Comment 26

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/c079dea037f7