1620166 - (ship-win-cross) Switch Windows shippable builds to cross-compiles

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Comment 1

[Masatoshi Kimura [:emk]]

Will Windows self-compiles no longer be tested on ci? (i.e. will it rapidly break?)

Comment 2

[Mike Hommey [:glandium]]

I think we're going to keep the plain builds as native. They're the closest to local builds.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1620166 - Switch shippable windows builds to cross builds.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1a54431e6d6
Switch shippable windows builds to cross builds. r=dmajor

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/a1a54431e6d6

Comment 6

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/9f851fcfe33f
Backed out changeset a1a54431e6d6 for causing beetmover exeptions Bug 1623616 a=backout

Comment 7

[Arthur Iakab [arthur_iakab]]

Backed out changeset a1a54431e6d6 for causing beetmover exeptions Bug 1623616 a=backout

Backout revision https://hg.mozilla.org/mozilla-central/rev/9f851fcfe33f9a13966a1bf59dc536ee2c8ccd33

Failed push https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=f985871b3e622041babc71f3770275f045ae7f33&searchStr=beetmover

Failure log https://firefox-ci-tc.services.mozilla.com/tasks/UVGsxMSHROaeLVwGI-DupQ/runs/1/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FUVGsxMSHROaeLVwGI-DupQ%2Fruns%2F1%2Fartifacts%2Fpublic%2Flogs%2Fchain_of_trust.log#L391

glandium or dmajor can you please take a look?

Comment 8

[(Away)]

I think we have a mar-versus-mar.exe mismatch in https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/manifests/firefox_candidates.yml#237-240,269-272 and/or a host-versus-target mismatch in https://searchfox.org/mozilla-central/source/toolkit/mozapps/installer/upload-files.mk#360-366.

Comment 9

[Tom Prince [:tomprince]]

We can probably consider not shipping the mar.exe/mbsdiff.exe at all, as we don't use them anywhere. (We still do use the published artifacts from linux64 build here though those could (should) be switched to the toolchains we build here.

Comment 10

[(Away)]

With the changes in comment 8, I got around the mar and mbsdiff problems, and now the beetmover task is complaining about a lack of public/build/target.normandy.json.

That file is gated behind ifndef CROSS_COMPILE: https://searchfox.org/mozilla-central/source/toolkit/mozapps/installer/upload-files.mk#397 and https://searchfox.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#128. From a skim of bug 1563564 it sounds like that was done due to problems with mac builds.

glandium, I had hoped to jump in with a quick fix while it was outside of work hours for you, but since this is becoming more involved, do you want to take it from here?

Maybe as a followup we should review all other uses of if[n]def CROSS_COMPILE too.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1620166 - Remove mar.exe, msbdiff.exe and target.normandy.json as artifacts of Windows builds.

The former two are not used anymore, and the latter can't be generated
on cross-builds, and it was agreed in bug 1563564 (and reconfirmed on
Matrix) that we can go without it until it's generated from separate
tasks.

More formally removing the former two will be the subject of a followup.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f1b532230821
Remove mar.exe, msbdiff.exe and target.normandy.json as artifacts of Windows builds. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/5c3aba6513e2
Switch shippable windows builds to cross builds. r=dmajor

Comment 13

[Florin Strugariu [:Bebe] (needinfo me)]

== Change summary for alert #25441 (as of Thu, 19 Mar 2020 19:43:51 GMT) ==

Improvements:

13% raptor-webaudio-firefox windows7-32-shippable opt 166.33 -> 145.17
4% raptor-webaudio-firefox windows10-64-shippable opt 130.67 -> 124.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25441

Comment 14

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/f1b532230821
https://hg.mozilla.org/mozilla-central/rev/5c3aba6513e2

Comment 15

[Marian Raiciof [:marauder]]

== Change summary for alert #25436 (as of Thu, 19 Mar 2020 07:52:28 GMT) ==

Improvements:

24% build times windows2012-32-shippable opt instrumented taskcluster-c5.4xlarge 2,639.03 -> 1,994.00
23% build times windows2012-aarch64 opt aarch64-no-eme nightly taskcluster-c5.4xlarge 2,818.62 -> 2,172.68
23% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,907.06 -> 2,244.43
22% build times windows2012-32-shippable opt nightly taskcluster-c5.4xlarge 2,883.64 -> 2,261.80

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25436

---

== Change summary for alert #25440 (as of Thu, 19 Mar 2020 18:43:48 GMT) ==

Regressions (were fixed by the last commit in 5c3aba6513e2):

31% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,253.20 -> 2,942.85
30% build times windows2012-32-shippable opt nightly taskcluster-c5.4xlarge 2,208.27 -> 2,867.96
26% build times windows2012-32-shippable opt instrumented taskcluster-c5.4xlarge 2,041.94 -> 2,580.14
26% build times windows2012-aarch64 opt aarch64-no-eme nightly taskcluster-c5.4xlarge 2,219.95 -> 2,787.19
22% build times windows2012-64-shippable opt instrumented taskcluster-c5.4xlarge 1,933.85 -> 2,365.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25440

---

== Change summary for alert #25456 (as of Fri, 20 Mar 2020 13:32:58 GMT) ==

Improvements:

23% build times windows2012-aarch64 opt aarch64-no-eme nightly taskcluster-c5.4xlarge 2,831.51 -> 2,180.86
23% build times windows2012-32-shippable opt instrumented taskcluster-c5.4xlarge 2,578.48 -> 1,995.98
21% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,900.37 -> 2,279.51

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25456

Comment 16

[Intermittent Failures Robot]

1 failures in 4955 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows2012-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1620166&startday=2020-03-16&endday=2020-03-22&tree=all