967055 - Investigate standing up a Mac OSX Mozmill-CI node to test 32-bit mode

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect, P2)

Description

[u279076]

On January 16, 2014 it was discovered by Henrik that Firefox builds were busted on Mac OSX when running in 32-bit mode (bug 960509). I think we were lucky that Henrik was testing this manually and that it was caught in Nightly. It would be good to be running Mozmill tests daily on at least one Mac OSX node where 32-bit mode is forced.

Another approach might be to double up the testing on existing Mac OSX nodes (ie. run once in default 64-bit mode and repeat the tests in 32-bit mode using 'arch -i386'.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I would say we could run all the tests on each OS X nodes twice, once for 64bit mode and afterward for 32bit mode. But then I would limit this to Nightly only. That's the branch where we only run tests for en-US and no other locale. So it wouldn't hurt us.

Anthony, what do you think?

There will be a problem with a restart test, which exactly is flipping already between 32bit and 64bit mode. So it would have to be updated.

Lifting the priority of this bug up to P1, given the bustage of the 32bit mode on OS X.

Comment 2

[Andrei Eftimie]

Do we have any data on how popular 32bit mode is on OSX?
The 64bit kernel was first shipped with OSX 10.6 (which is the oldest version we still support).
Conversely 10.6 is the _last_ OS version to support a 32bit kernel.

We might assume from this that 32bit mode and 10.6 are tightly coupled, but I have no numbers to  confirm or deny this.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #2)
> Do we have any data on how popular 32bit mode is on OSX?
> The 64bit kernel was first shipped with OSX 10.6 (which is the oldest
> version we still support).
> Conversely 10.6 is the _last_ OS version to support a 32bit kernel.

Huh what makes you think of that? 10.6 is the last version of OS X, which supported PPC. So its support dropped. But no operating system stopped supporting 32bit applications. That wouldn't still work given the high amount of 32bit systems.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As simplified variation of a test, we might be able to run it as part of the already existent test:
http://hg.mozilla.org/qa/mozmill-tests/file/0afa1489e922/firefox/tests/functional/restartTests/testRestartChangeArchitecture

So when restarting Firefox in 32bit mode, we would have to do some specific checks to ensure if Firefox started up correctly.

But I would like to let Anthony decide what's best for the time being.

Comment 5

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> (In reply to Andrei Eftimie from comment #2)
> > Do we have any data on how popular 32bit mode is on OSX?
> > The 64bit kernel was first shipped with OSX 10.6 (which is the oldest
> > version we still support).
> > Conversely 10.6 is the _last_ OS version to support a 32bit kernel.
> 
> Huh what makes you think of that? 10.6 is the last version of OS X, which
> supported PPC. So its support dropped. But no operating system stopped
> supporting 32bit applications. That wouldn't still work given the high
> amount of 32bit systems.

Yeah, I've read that wrong. 10.6 shipped with a 32bit kernel by default (and had support for a 64bit kernel), while all newer versions shipped with a 64bit kernel by default.

Do we have some telemetry data on how many people are running a 32bit kernel on OSX? As said above I would assume _all_ of them are still on 10.6. And it would be interesting to know out of all 10.6 users, how many are running a 32bit kernel (I would assume most of them since the ones that could update their OS, probably did)

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I have no idea, but metrics people might be able to get this information from us.

Comment 7

[sci-fi]

Hello,

If I may inject my own "food for thoughts" here about this topic please.  This might be "old news" but I am feeling the need to repeat.

I am one of those strugglin' 10.6/SL users.
I am using a model "iMac6,1" which is sort-of related to the first MacPro towers, all of us have 32-bit EFI/BIOS, mine has a Core-2-Duo CPU, which means SL usually runs tasks in 64-bit mode (if available within the app) but _must_ run the kernel code in 32-bit mode.

I know I can run 10.7/Lion (which is _really_ the last version I am officially allowed to run) but I will lose several functions, most notably Rosetta, and acquire several pesky gizmos including "recovery partition", the newer Gatekeeper, and MAS (Mac App Store) for updates etc.  (So I wonder how 32-bit EFI/BIOS can run with Lion if Lion has no 32-bit kernel?)

I also know I could try the "rEFInd" 64-bit fake calls to make it seem as tho my iMac has 64-bit EFI.  But at the time I fetched their file, I believe it was tailored for those early tower models and not for the iMac/mini/etc models such as mine.  So I haven't tried it yet.

At any rate, I've been testing the (latest) tinderbox builds with a shell proc CLI (Terminal.app) thus:
> #!/bin/sh
> export DYLD_LIBRARY_PATH="/Applications/Nightly/FirefoxNightly.app/Contents/MacOS"
> export ffn="$DYLD_LIBRARY_PATH/firefox"
> export MOZ_DISABLE_CONTENT_SANDBOX=1
> arch -i386 $ffn 
including many other settings from /etc/bashrc and friends.

Sometimes I'll try adding this code to the above proc to extend the DYLD path:
> export DYLD_LIBRARY_PATH="/usr/local/ssl/lib:/usr/local/lib/libquicktime:$DYLD_LIBRARY_PATH"
and use the full path to firefox executable:
> arch -i386 "/Applications/Nightly/FirefoxNightly.app/Contents/MacOS/firefox"
to try pointing to my own build of OpenSSL (which comes from their github master tree) etc since The Fruits never did provide suitable updates there to us; this is just an educated guess tho.  ;)

I've just-now been in active contact with the author of "SixtyFour.app"
<http://getsixtyfour.com/>
which had a very-recent update (I guess I'm trying to 'help plug' his work).
I've invited him to this bugreport.
Please see the image that explains why we ought to run in 32-bit mode:
At the link just above,
click on the word "significantly" hilited in blue
in the paragraph under the "Download -or- Purchase" link. 
(I don't know how to give a URL for that "Google Chart".)

Please pardon the personal stuff below.

Why am I 'griping' about this here?
Why don't I simply buy a newer model?
I've briefly told my reason on other bugreports here and at other projects.
Here's why:
I have been on disability for over 7-years with no end in sight.
I am also eligible for full retirement with over 4-decades work on record.
I CANNOT CURRENTLY AFFORD any extra computer, new or used.
My main paid job was in a mainframe shop as a technical guru of sorts; we have a state-wide private wired network connecting different division HQs (grouped counties) all across the state.
They also let me build FreeBSD system albeit on antiquated IBM PCs (they seemed to want me to fail with such equipment, but I made 'em work really well).

I have joined several projects because mainly I am very interested in testing etc, possibly write a bit of code, and also because it 'tickles' my grey noodles that can keep me busy and hopefully generally useful.

So I feel very qualified with the expertise to Jump This Fruity Ship _completely_ and go with 100% full F/OSS hardware and software
if I had the right kind of help.
(currently I wonder if I should order Kevin Trudeau's book "Free Money" and pursue the avenues therein? e.g. get a Grant, instead of a loan at my credit union)

And then y'all can go ahead with supporting all the people that _have_ money.

That's the bottom line with this particular topic with me.

We still have people with purely 32-bit machines still out there.
And quite likely they won't upgrade by buying a new machine either with similar problems as mine.
So how are _they_ going to keep up with the security updates and other internet gizmos etc.?  (The Fruits won't care to answer this either.)
Yes I know y'all "need usage numbers" to justify supporting them.

The platform doesn't really matter ---
if you want to drop 32-bitters on the OSX system
then also drop the 32-bitters on Wynderz™
and also drop the 32-bitters on other *ix systems e.g. Linux
and etc etc etc.

To me, after being in the computer engineering market in many many forms for many many years (I actually started with OS-9/6809 on the Tandy CoCo as a personal hobby, I even ran a Fido™ node on a modem dialup line), not being able to continue supporting the older systems smells of nothing but Lazyness to me.
I know I just said a crewl thing to all the unpaid volunteers here, but if you are paid in any way/shape/form then yes I would want you to feel insulted.

It should not hurt me per sé since I should be able to run 10.7 and can run 64-bit in userland, if I am forced to.

So I guess I am "A voice for the 32-bitters" and thought this needs to be relayed.

(I do note that the PPC OSX machines are supported by a non-Mozilla group.  Perhaps we need to start a non-Mozilla group for all purely 32-bit Intel® etc machines also?)

Thanks for reading.  I'll stay tuned (even via email private if needed).

Comment 8

[Andrei Eftimie]

We're actually not debating Firefox support, but test infrastructure support for these older builds. Which would hopefully catch regressions for them.

(My personal opinion is more forward-pushing, thus probably more non-inclusive of older hardware, but as said, it is personal and doesn't have any sway on the official Firefox hardware/software support)

I would still argue whether or not this is worth doing. Software need to go forward.
And while I do sympathise with your old machine, at one point it will stop working - your computer is 8 years old now (which in today's fickle computer life I do find impressive).

Comment 9

[u279076]

As long as Mozilla officially supports Mac OS X 10.6 I think we need to support all that entails, which means checking periodically that 32-bit mode has not broken. Of course, we have to balance our attention  (and limited resources) toward those areas of most need.

I don't have firm numbers on how many users actually use Firefox in 32-bit mode on Mac OS X 10.6 but I would guess it's one of our lower user bases. That said, we have to support them one way or another. When it comes down to it, it's a question of cost.

A) The cost of setting up, running, and maintaining automation for a declining platform
 - vs -
B) The cost of running a manual smoketest run once in a while until we desupport the platform

My guess is that A costs more in the short-term, B costs more in the long-term. 

I'm fine going down the B path if Automation Development doesn't have the resources to support this. But we need to acknowledge the risk that opens us up to. Automation running on Nightly will catch bugs quicker and earlier than running manual smoketests once in a while, thereby greatly decreasing the risk of shipping a broken release to these users.

Ultimately, I think the decision lies with Automation Development, whether it's something they want to invest in and support. If not, the Firefox QE team will have to adjust our strategy accordingly.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Before we continue here, I posted a question of usefulness to the dev.platform group:
https://groups.google.com/forum/#!topic/mozilla.dev.platform/z_Ty34fSD6g

Once we know what to do, we will continue here.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Anthony, what about running 32bit tests for all supported versions of Firefox on a 10.9 host? I don't see a reason for 10.6 specifically. A host with 10.9 should simply live longer, and should show the same bustage if exists. What do you think?

Comment 12

[u279076]

I don't have data to support this but I suspect the user base running 32-bit gets much much smaller as we look at more modern OSX versions. I'd be in favour of just deploying this for 10.6 as long as we plan to support it.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Mozmill-CI is going down soon and 32bit mode is even not supported anytime longer.