Closed Bug 1322402 Opened 8 years ago Closed 7 years ago

Drop support for universal Mac builds in Thunderbird 53

Categories

(Thunderbird :: Build Config, defect)

Product:
Thunderbird
Component:
Build Config
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 54.0

People

(Reporter: aleth, Assigned: aleth)

References

Details

(Whiteboard: [Thunderbird-testfailure:B Mac opt, see comment #7])

Attachments

(3 files, 4 obsolete files)

1322402-mac-build.patch (landed comment #17)
8.92 KB, patch
Details | Diff | Splinter Review
macosx64-config.patch
1.70 KB, patch
rail
: review+
Details | Diff | Splinter Review
[buildbotcustom] Swaps current beta, ESR45, and ESR52 back to universal
1.84 KB, patch
jlund
: review+
Details | Diff | Splinter Review 
+++ This bug was initially created as a clone of Bug #1295375 +++

Per bug 1183613 comment 16, we can drop universal mac builds in Firefox 53.

mozilla-central becomes Firefox 53 on November 7.
Attached patch dropUniversal.patch (obsolete) — — Splinter Review 
Could this be the way to go? Or is more needed?
Attachment #8821745 - Flags: review?(aleth)
Attached patch buildbot change (obsolete) — — Splinter Review 
I think this buildbot change is needed. But where is aurora, release etc?
Attachment #8821751 - Flags: feedback?(aleth)
(In reply to Richard Marti (:Paenglab) from comment #1)
> Could this be the way to go? Or is more needed?

Unfortunately much more will be needed, with lots of moving parts also in the build system.
Flags: needinfo?(aleth)
Comment on attachment 8821751 [details] [diff] [review]
buildbot change

:Paenglab, this is the same patch as the mozconfig one.
Attached patch buildbot change (obsolete) — — Splinter Review 
Correct patch
Attachment #8821751 - Attachment is obsolete: true
Attachment #8821751 - Flags: feedback?(aleth)
Comment on attachment 8825655 [details] [diff] [review]
buildbot change

Review of attachment 8825655 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/thunderbird_config.py
@@ +285,5 @@
>              'build_space': 12,
>              'upload_symbols': True,
>              'download_symbols': True,
>              'slaves': SLAVES['macosx64-lion'],
>              'platform_objdir': "%s/i386" % OBJDIR,

I think the "%s/i386" part should be changed to just OBJDIR.
Depends on: 1331297
Bug 1331297 has now busted OS X Opt Dailies, as was always going to happen sooner or later. Note we still have full test coverage on OS X debug, and local OS X Opt builds are of course not affected as they are not universal in the first place.
Flags: needinfo?(aleth)
Aleth, I also pushed this to try, https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=34ab1220161931cc7cdf379c05b39bd8beeed3b1

Changes compared to Richard's patch is that I removed the lightning universal stuff, in all the configs (including the l10n one)
Blocks: 1337350
build broke, https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=34ab1220161931cc7cdf379c05b39bd8beeed3b1&selectedJob=75134191, as it's using the old universal mozconfig. I wonder if we can do harm by landing this and the buildbotcustom changes, and then iterate if needed?

Or maybe at least the buildbotcustom one, so that we can push to try to test?
(In reply to Axel Hecht from comment #10)
> build broke,
> https://treeherder.mozilla.org/#/jobs?repo=try-comm-
> central&revision=34ab1220161931cc7cdf379c05b39bd8beeed3b1&selectedJob=7513419
> 1, as it's using the old universal mozconfig. I wonder if we can do harm by
> landing this and the buildbotcustom changes, and then iterate if needed?

I don't think Firefox is using buildbot any more on dailies, so it can't get more broken ;)

Having said that, iirc there's a lot of os x build special casing in the buildbot code. I don't think the buildbot patch is correct as is, but if you want to r+ it and do it in stages, that's fine.
No longer blocks: 1337350
Whiteboard: [Thunderbird-testfailure][see comment #7]
Whiteboard: [Thunderbird-testfailure][see comment #7] → [Thunderbird-testfailure:B Mac opt, see comment #7]
All the talk about updates and stuff in bug 1295375 concern me. Plus, I never understood buildbot-configs, so having to touch that would also be quite a bit out of my comfort zone.
Assignee: nobody → aleth
I'm a little confused here. SM have landed:
https://hg.mozilla.org/comm-central/rev/becf847f864ca2765095ea5f22a4cd6f99d35729
which is not a big patch.

There are a couple of patches attached here and then Aleth took the bug. So what's going on? We've been without Mac coverage for a while now.
Comment on attachment 8825655 [details] [diff] [review]
buildbot change

Rail, could this work, when we have created the files in mail/config/mozconfigs/macosx64?
Attachment #8825655 - Flags: review?(rail)
In bug 1339182 [Remove OSX universal support in the build system]
https://hg.mozilla.org/mozilla-central/rev/d88a5673b1f7b5e3c8c37383e43c5c1b599ed89c
M-C removed build/macosx/universal

So I'm getting build oranges since these files are still in C-C. This patch removes those directory, and also includes the combined patches of Richard and Pike to fix files in mail/config/mozconfigs/macosx-universal, in fact I remove the whole thing since there was only one file left after moving release and nightly.

With this patch I get a green debug build, the opt build still doesn't work due to a build bot problem:
cp: mail/config/mozconfigs/macosx-universal/nightly: No such file or directory
Attachment #8821745 - Attachment is obsolete: true
Attachment #8834390 - Attachment is obsolete: true
Attachment #8821745 - Flags: review?(aleth)
Attachment #8834390 - Flags: review?(aleth)
https://hg.mozilla.org/comm-central/rev/6ee27a7af6cd3e7bd25f392048fd40160910275a

Due to lack of movement here and the current build oranges, I had to land this as a bustage fix. However, reading the comments above, was the agreed process anyway.

Let's now move over to the build bot changes.
Comment on attachment 8825655 [details] [diff] [review]
buildbot change

Review of attachment 8825655 [details] [diff] [review]:
-----------------------------------------------------------------

I think :ewong is correct, let's drop i386 from platform_objdir
Attachment #8825655 - Flags: review?(rail) → review-
Removed the i386 from platform_objdir.

Rail, if this patch should get r+, please can you land it then? Thank you.
Attachment #8825655 - Attachment is obsolete: true
Attachment #8838599 - Flags: review?(rail)
Comment on attachment 8838599 [details] [diff] [review]
macosx64-config.patch

Thank you!
Attachment #8838599 - Flags: review?(rail) → review+
Rail, could you please get this into production. Neither Richard nor myself are build specialists. The action here was triggered by the landing of bug 1339182 (see comment #16). So we hope we made the necessary changes in C-C and with some luck we'll get a build.
Flags: needinfo?(rail)
Will do in a bit.
Flags: needinfo?(rail)
https://hg.mozilla.org/comm-central/rev/6ee27a7af6cd3e7bd25f392048fd40160910275a (comment #17)
https://hg.mozilla.org/build/buildbot-configs/rev/a7218494e505 (comment #24)

Successful build:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=6ee27a7af6cd3e7bd25f392048fd40160910275a&selectedJob=78392016

Thanks to all involved. If you find more problems, reopen this bug or file another one.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 54.0
FWIW, I just triggered a new nightly from that revision, let's see what happens.
Updated to new Daily and it is working. :-)
When TB 54 will enter beta and later ESR59 the buildbot-config files need to be adapted.

DXR search where macosx-universal is referenced: https://dxr.mozilla.org/build-central/search?q=mail%2Fconfig%2Fmozconfigs%2Fmacosx-universal&redirect=false
(In reply to Richard Marti (:Paenglab) from comment #28)
> When TB 54 will enter beta and later ESR59 the buildbot-config files need to
> be adapted.
Does that mean that we need to uplift this to Aurora since otherwise the changed build bot file will not allow an Aurora build?

(In reply to Axel Hecht from comment #29)
> There don't seem to be l10n repacks, I don't find any public traces of
> errors, sadly.
As I said, I'm not the author here. I merely fixed some orange builds and while I was there landed other people's patches which made it (semi-)work. Without those changes the files in mail/config/mozconfigs/macosx64/ would have referenced files I had to remove in build/macosx/universal/mozconfig (since FF removed them and we must keep the builds in sync).
(In reply to Jorg K (GMT+1) from comment #30)
> (In reply to Richard Marti (:Paenglab) from comment #28)
> > When TB 54 will enter beta and later ESR59 the buildbot-config files need to
> > be adapted.
> Does that mean that we need to uplift this to Aurora since otherwise the
> changed build bot file will not allow an Aurora build?

I see no aurora file. So I guess aurora uses the nightly config too. Only beta and esr have their own files.
So using the same build bot file, Aurora will be busted tomorrow unless this gets uplifted and then breaks l10n on Aurora.
When my assumption is correct, yes,
(In reply to Jorg K (GMT+1) from comment #32)
> So using the same build bot file, Aurora will be busted tomorrow unless this
> gets uplifted and then breaks l10n on Aurora.

wrt l10n..  is it working now?
(In reply to Edmund Wong (:ewong) from comment #34)
> wrt l10n..  is it working now?
Whether l11n is working on Aurora? According to the announcement
https://mail.mozilla.org/pipermail/tb-planning/2016-October/004954.html
and the evidence:
http://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora-l10n/
Yes!

I'm still trying to see what Aleth did in October 2016 to get them working.

Th only thing I noticed that mail/config/mozconfigs/macosx-universal/l10n-mozconfig which I removed
https://hg.mozilla.org/comm-central/rev/6ee27a7af6cd3e7bd25f392048fd40160910275a#l5.2
didn't have |. $topsrcdir/build/mozconfig.common| but its replacement does:
https://dxr.mozilla.org/comm-central/source/mail/config/mozconfigs/macosx64/l10n-mozconfig

Would removing that line make a difference (for the better)?
Flags: needinfo?(ewong)
Could someone please compare
deleted file mode 100644
--- a/mail/config/mozconfigs/macosx-universal/l10n-mozconfig
with
https://dxr.mozilla.org/comm-central/source/mail/config/mozconfigs/macosx64/l10n-mozconfig

and tell me which of the marked differences
. $topsrcdir/build/mozconfig.common (**)
ac_add_options --with-l10n-base=../../l10n (**)

are likely to make it work again. Also note that for Windows, |disable-compile-environment| was removed here:
https://hg.mozilla.org/comm-central/rev/09ece7c086487cd738adb91669a43ad106a11443

For Mac it got added here:
https://hg.mozilla.org/comm-central/rev/6bbdff770fd6
(In reply to Jorg K (GMT+1) from comment #36)
> Could someone please compare
> deleted file mode 100644
> --- a/mail/config/mozconfigs/macosx-universal/l10n-mozconfig
> with
> https://dxr.mozilla.org/comm-central/source/mail/config/mozconfigs/macosx64/
> l10n-mozconfig
> 
> and tell me which of the marked differences
> . $topsrcdir/build/mozconfig.common (**)
> ac_add_options --with-l10n-base=../../l10n (**)

I would say first try to change 
ac_add_options --with-l10n-base=../../l10n
to ac_add_options --with-l10n-base=../../../l10n

Because this looks like a copy of FX and on C-C we have mozilla a directory deeper.
OK, Aurora just got busted, so let's uplift this and add Richards suggestion ;-)

Aurora (TB 53):
https://hg.mozilla.org/releases/comm-aurora/rev/6c3bbdce6b7b2a45f4afa44d3db3165c4cc239fb
https://hg.mozilla.org/releases/comm-aurora/rev/b2892e9b6a35fa8801218c511e3ead629508afa9

Note that in the file 'nightly' I added |ac_add_options --enable-profiling| to make it match the C-C file. No idea why that wasn't there in the first place.

Let's trigger a build on Aurora now.
Hmm, removing the build/macosx/universal/ files on aurora will likely give check-sync-dir oranges because they are only removed in M-C.
Richard, can you please try to trigger a N on Mac opt only on C-A. When I click "Add new jobs" I get "Error fetching runnable jobs".
I filed bug 1340787 for the problem in comment #40. Who should I CC this to?
I get the same error.
I tried to trigger a new nightly via https://secure.pub.build.mozilla.org/buildapi/self-serve/comm-aurora.
It said OK, and that works. I'll cancel the all but Mac.
Oops, grammar derailed: I'll cancel all but Mac.
BTW, did the same thing you did, that's why there are two now. Cancel one of the macs, too?
I cancelled the second lot completely.
(In reply to Richard Marti (:Paenglab) from comment #39)
> Hmm, removing the build/macosx/universal/ files on aurora will likely give
> check-sync-dir oranges because they are only removed in M-C.
The N is green. Mac binaries here:
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-02-40-05-comm-aurora/
but not here:
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-02-40-05-comm-aurora-l10n/
So that tweak
https://hg.mozilla.org/releases/comm-aurora/rev/b2892e9b6a35fa8801218c511e3ead629508afa9
didn't work.
Then | . $topsrcdir/build/mozconfig.common | probably needs to be removed too as the error says: "mozbuild.configure.options.InvalidOptionError: RUSTC is not available in this configuration" and mozconfig.common relates to | . "$topsrcdir/build/mozconfig.rust" |. And it seems it fails before it wants to go to l10n and we don't know which is the correct path. But I would leave the longer path in it at first because the old l10n file had it too.
OK, last try, Aleth will hate me for this trial-and-error approach:
https://hg.mozilla.org/releases/comm-aurora/rev/616d41945914fffcdcd57e87f5bd303544ade7e4

I'll trigger a build to see what that does.
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-07-32-20-comm-aurora/
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-07-32-20-comm-aurora-l10n/ didn't work.
ERROR: Invalid value --with-l10n-base, ../../../l10n doesn't exist

So I'll do one last one with |ac_add_options --with-l10n-base=../../l10n|
Blocks: 1340829
Damn, I was too impatient here:
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-09-30-48-comm-aurora/
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-09-30-48-comm-aurora-l10n/
did get created. So summarising:
|ac_add_options --with-l10n-base=../../l10n| is correct and 
|. $topsrcdir/build/mozconfig.common| needs to go.

I'll do this quickly in bug 1340829.
Flags: needinfo?(ewong)
(In reply to Jorg K (GMT+1) from comment #54)
> Damn, I was too impatient here:
> https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-09-30-
> 48-comm-aurora/
> https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-09-30-
> 48-comm-aurora-l10n/
> did get created. So summarising:
> |ac_add_options --with-l10n-base=../../l10n| is correct and 
> |. $topsrcdir/build/mozconfig.common| needs to go.
> 
> I'll do this quickly in bug 1340829.

Yeah. sorry for the delay in response.

../../l10n is correct because the objdir no longer includes the
platform (otherwise, it'd require ../../../l10n).  

Also, why does |. $topsrcdir/build/mozconfig.common| need to go?
(In reply to Edmund Wong (:ewong) from comment #55)
> Also, why does |. $topsrcdir/build/mozconfig.common| need to go?
I really don't know. I tried with and without and without worked, see bug 1340829. Also comment #48 here.
The file I removed didn't have it:
https://hg.mozilla.org/comm-central/rev/6ee27a7af6cd3e7bd25f392048fd40160910275a#l5.2
Sorry, they let the apprentice (me) do the job and due to complete ignorance he could only do a trial-and-error job, which succeeded on the 4th try!
ESR 45:
https://hg.mozilla.org/releases/comm-esr45/rev/f8478f0eaea4de365bf126c4805c3b31f001e8d0
Fingers crossed. Since the offending line |. $topsrcdir/build/mozconfig.common| wasn't in
mail/config/mozconfigs/macosx64/l10n-mozconfig to start with, the uplift from bug 1340829 wasn't necessary.
Why are you pushing this to TB 52 and 45? FX has it only in 53.
Because builds are failing there due to our build bot changes:
cp: mail/config/mozconfigs/macosx64/nightly: No such file or directory

Check for example

ESR 45
https://archive.mozilla.org/pub/thunderbird/nightly/2017/02/2017-02-18-03-02-11-comm-esr45/comm-esr45-macosx64-nightly-bm82-build1-build5.txt.gz

Beta TB 52:
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-macosx64/1487706688/comm-beta-macosx64-bm86-build1-build0.txt.gz

With the patch uplifted, it builds again. As I said, they called the apprentice to do the job :-(
Mac OS X Snow Leopard (10.6) and later will only run on Intel Macs and doesn't need universal/fat binaries according to https://en.wikipedia.org/wiki/Universal_binary.

Since according to https://www.mozilla.org/en-US/thunderbird/45.7.1/system-requirements/ TB 45.x only runs on 10.6 and later, it makes no sense to ship universal binaries for any of the versions we support.

Note that TB 52 (https://www.mozilla.org/en-US/thunderbird/52.0beta/system-requirements/) only supports Mac OS X 10.9 and later.

Anyway, I'm prepared to be wrong and leave it to the build peer to figure it out.
(In reply to Jorg K (GMT+1) from comment #61)
> Mac OS X Snow Leopard (10.6) and later will only run on Intel Macs and
> doesn't need universal/fat binaries according to
> https://en.wikipedia.org/wiki/Universal_binary.
> 
> Since according to
> https://www.mozilla.org/en-US/thunderbird/45.7.1/system-requirements/ TB
> 45.x only runs on 10.6 and later, it makes no sense to ship universal
> binaries for any of the versions we support.

But such changes shouldn't be done in a point release, especially on a stable ESR release.

What if you use the macosx-universal configs in the macos64 directories? then they call still the universal config ($topsrcdir/build/macosx/universal/mozconfig). maybe also the removed files in build/macosx/universal/ are needed again.
I won't touch it any more. The build failed anyway at the packaging stage:
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr45-macosx64/1487873962/comm-esr45-macosx64-bm84-build1-build1.txt.gz
with the weirdest of errors.

Re. your suggestion: The build bot error is:
cp: mail/config/mozconfigs/macosx64/nightly: No such file or directory

So you're suggesting to copy the "universal" nightly, release and l10n files from macosx-universal to macosx64? Nice hack. We can back out what I landed and see whether an official build can still be done.
(In reply to Jorg K (GMT+1) from comment #61)
> Since according to
> https://www.mozilla.org/en-US/thunderbird/45.7.1/system-requirements/ TB
> 45.x only runs on 10.6 and later, it makes no sense to ship universal
> binaries for any of the versions we support.
>
The universal binaries currently built are not intel/ppc but 32-bit/64-bit binaries. IIRC the 32-bit ones are needed in order to run some of the supported plugins.
OK, the problem came from the build bot change in comment #24. Nick (:nthomas) has kindly offered to fix the mess we made.

I will then backout the beta (TB 52) and ESR 45 changesets. We can still build TB 52 beta 4 without those changes.
Comment on attachment 8840637 [details] [diff] [review]
[buildbotcustom] Swaps current beta, ESR45, and ESR52 back to universal

should do the trick
Attachment #8840637 - Flags: review?(jlund) → review+
Pardon my ignorance..  if the macosx64's src_mozconfig is set to the nightly 
mozconfig,  where in the infra, during the release, would the src_mozconfig be
set to the release one?  (I do apologize if it's apparent.  I'm like a bit
overwhelmed with a lot of different fires, of which this mac-universal is 
one of 'em.)
(In reply to Edmund Wong (:ewong) from comment #70)
> Pardon my ignorance..  if the macosx64's src_mozconfig is set to the nightly 
> mozconfig,  where in the infra, during the release, would the src_mozconfig
> be
> set to the release one?  (I do apologize if it's apparent.  I'm like a bit
> overwhelmed with a lot of different fires, of which this mac-universal is 
> one of 'em.)

ah nevermind..  I found it.  it's in the release-thunderbird-*.py
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: