Closed Bug 327092 Opened 19 years ago Closed 18 years ago

Mac OS X tinderboxes must meet minimum build requirements

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Other
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jaas, Assigned: preed)

References

Details

(Whiteboard: ETA 2/21/07)

Attachments

(2 files)

tinderbox upgrade plan, v1.0
2.15 KB, text/plain
Details
mac-tinderbox-inventory
1.32 KB, text/plain
Details
I'm not sure if filing an IT request is the right way to go about this, but here goes... :) The minimum build requirements for Mac OS X are Mac OS X 10.3.9 and Xcode 1.5. Any tinderboxes that do not meet these requirements will burn very soon. Any machines producing production builds must be updated to Mac OS X 10.4.4 (Tiger) and Xcode 2.2.1. This is so they can produce PPC/x86 cross builds. This must be done soon in order to be ready for an Universal Binary Mac OS X release of Firefox 1.5.0.2. If you do a fresh install of Mac OS X on any machines (which I highly recommend when doing major version updates, e.g. 10.2->10.3, 10.3->10.4), please use darwinports to install dependencies, not fink. Fink does not work on Intel Macs, and is painful compared to darwinports. Sorry if this seems sudden, but I have been trying to get this going for some time without success. Please let me know if an IT request is a good or bad way to do this. I'm not sure if IT covers tboxes.
Doesn't seem sudden to me. Gory details over in bug 299214.
I always found darwinports painful because it made you actually compile everything. Fink at least lets you use apt-get to grab binaries of things. Of course, if Fink doesn't work on Intel then it doesn't matter.
bz has closed the tree because of the bustage, which we didn't expect. He did it on the grounds that he doesn't want to get perf regressions. So now this request is a lot more urgent.
Severity: critical → blocker
You can still use fink on ppc hardware if you prefer it, but for the time being, it won't work on x86. DarwinPorts works on both CPUs.
Assignee: server-ops → build
Group: mozillaorgconfidential
Component: Server Operations → Build & Release
QA Contact: justin → preed
Tree is open. This is still critical, but not a blocker.
Severity: blocker → critical
Assignee: build → aravind
Please add specific host names and schedules. I will need to know the host name of the box that will be upgraded, the software you need on it (and where to get it from). It will also be useful to have priorities set on these (like this host needs to be upgraded first, etc.) Since most of this will probably need me to go to the colo, I plan on doing this one day a week. I will try to get out there every wednesday.
Status: NEW → ASSIGNED
To add on to what Josh said: Machines used to build Camino must be upgraded to at least 10.4.4/2.1. Currently, this includes binus, pawn, and boxset. pawn at least may not have sufficient hardware to run the new configuration. Machines used to produce universal binaries must be upgraded to at least 10.4.4/2.2.1. Which machines are to be used for this task is not my call. All other machines must be upgraded to at least 10.3.9/1.5. There is currently only one tinderbox with 10.4.4/2.2.1, maya, and it's building Camino.
Aravind, first priority is probably monkey. It's a SeaMonkey tinderbox running perf tests under 10.2.8. After that, I imagine it's most important to set up something able to build a universal Firefox. It's been suggested that almost all of the tinderboxen be upgraded to 10.4.4/2.2.1, maybe leaving only one or two boxes to monitor the 10.3.9/1.5 build. I support doing this whereever possible.
This text file describes what tinderboxes we need to upgrade, and how to do it. Feedback would be great, but there isn't much time to wait for it. This is a lot of machines to upgrade, but all release machines for all products need to produce Universal Binaries and some of the non-release machines should too. There will be a couple machines that still run Mac OS X 10.3.9 since we support building on that OS version.
Before doing "erase and install" it's worth making sure that these machines don't have things like critical diffs needed for doing release builds in their tinderbox scripts or any other critical stuff. (Does it delete the whole disk or just the OS?)
I share dbaron's concerns. Does an "erase and install" include formatting the disk? If so, why is this preferable to an upgrade? Given the state of certain areas of our build/release farm, I'm loathe to start blowing away machines and reinstalling from scratch.
(In reply to comment #9) > Created an attachment (id=211944) [edit] > tinderbox upgrade plan, v1.0 Josh: I read through the plan and have a couple of questions: 1. Could you put what each machine's expected role is going to be (which products they'll build on which branches?) Maybe they won't change? 2. Could you include a list of the necessary compilers/ tools that are needed, how/where to get them, and (if it's non-obvious) how to install them? We're trying to move in the direction of getting the tinderbox/official build environments more documented, stablized, and less ad hoc, so it'd be good to move in this direction while working on the tinderboxen (especially if the machines have to be re-built from scratch).
> 1. Could you put what each machine's expected role is going to be (which > products they'll build on which branches?) Maybe they won't change? I don't know of any reason that the roles need to change. > 2. Could you include a list of the necessary compilers/ tools that are > needed,how/where to get them, and (if it's non-obvious) how to install them? We need to use Xcode 2.2.1 on all Tiger machines, and Xcode 1.5 on all 10.3.9 machines. You can download these from developer.apple.com if you have a free account there. When installing Xcode, we need to make sure that all SDKs get installed. This means doing a custom install of Xcode and selecting all SDKs. For package management (libidl, that kind of thing) we should be using darwinports. See the bug summary about this. Erase and install erases the whole disk, not just the OS. The reason I suggest doing this is that there is a higher risk of something going wrong when you upgrade major OS version. Especially if you're messing around in the BSD subsystem with Terminal, doing things like installing fink and changing permissions. We're going to have to go through that twice for any machines we want to move from 10.2 to 10.4 (many of them). I don't think it is worth the risk, and I doubt it will be faster to upgrade than it will be to just start clean unless there is something crazy going on with tbox configs. Which I doubt because Chase was able to set up tinderboxes in about 3-5 minutes and never mentioned anything special that had to be done to me. Better check anyway though, as dbaron suggested. Also, it would be comforting to me to know exactly what tools are on the machine and how they are set up. I don't like the idea of inheriting any "interesting" setups for our build environment when I'm not aware of any need for anything special. A clean slate will assure us of a standard, conflict-free build environment. An example would be that we suspect there is something wrong with the toolchain on at least one of the Camino tinderboxes (something funny with "make"?). I envision this working in 3 steps - someone wipes the machine and puts the target OS version on it, then sets up ssh and VNC. Then I will install Xcode and darwinports, set up a build environment. I can do that quickly, and I do it all the time so we can avoid having to debug problems. Then someone can set up the tinderbox however they like and it should build just fine. Chase had a very standard way of setting up tbox configs, I don't know what that is and I've never successfully set up a tinderbox myself.
(In reply to comment #13) > I don't know of any reason that the roles need to change. Great; so do can you add to the list which machines are currently responsible for which builds? > We need to use Xcode 2.2.1 on all Tiger machines, and Xcode 1.5 on all 10.3.9 > machines. You can download these from developer.apple.com if you have a free > account there. When installing Xcode, we need to make sure that all SDKs get > installed. This means doing a custom install of Xcode and selecting all SDKs. I know I don't have a download of these packages; I'll see if IT does. > For package management (libidl, that kind of thing) we should be using > darwinports. See the bug summary about this. Ok; do you have a specific list of packages that need to be installed from darwinports? > Also, it would be comforting to me to know exactly what tools are on the > machine and how they are set up. I don't like the idea of inheriting any > "interesting" setups for our build environment when I'm not aware of any need > for anything special. A clean slate will assure us of a standard, conflict-free > build environment. An example would be that we suspect there is something wrong > with the toolchain on at least one of the Camino tinderboxes (something funny > with "make"?). I agree that a clean build environment is preferable, but I know some of these machines have been used for other tasks (l10n repackaging, patch creation, etc.), and so there may be tools/configurations that need to be preserved (or at least accounted for) on any of the build machines (this isn't a Mac-specific problem). I'm worried about inadvertently deleting something important. > I envision this working in 3 steps - someone wipes the machine and puts the > target OS version on it, then sets up ssh and VNC. Then I will install Xcode > and darwinports, set up a build environment. I can do that quickly, and I do it > all the time so we can avoid having to debug problems. Then someone can set up > the tinderbox however they like and it should build just fine. Chase had a very > standard way of setting up tbox configs, I don't know what that is and I've > never successfully set up a tinderbox myself. Those steps sound good, but instead of having a black box where ssh/VNC is setup, and then "something" happens to get a build environment, I'd like get documented a specific set of instructions/requirements necessary to set up the build environment. And then, to verify the instructions, have someone follow them to get the environment set up. Let's try and pick one or two machines to completely wipe that we know (hopefully?) haven't been used for anything else, and were only ever tinderbox machines. That'd be a good place to start.
preed, most of your questions about setting up the dev environment should be answered here: http://developer.mozilla.org/en/docs/Mac_OS_X_Build_Prerequisites#Software_Requirements There are some system-level improvements that are kind of difficult to get without wiping everything clean. Probably the best way to go is to save copies of whatever you find in /build, except for source and obj trees. Speaking from experience, upgrading from one major OS version to another in-place is possible and works for the most part, but you won't wind up with something as "standard" as a clean installation, and you might have to go on a few wild goose chases to fix things that the upgrade couldn't handle intelligently.
Both of the first two tinderboxen on the list are only on build duty as far as I know. Here's what they're up to: monkey PMG4 1x933MHz G4 512MB 10.2.8 SeaMonkey (test) barcelona PMG4 1x867MHz G4 512MB 10.2.8 SeaMonkey (rel), Mozilla1.7 (test) I've taken inventory and have a more complete list at home, but those two at least should be safe to wipe.
(In reply to comment #16) > monkey PMG4 1x933MHz G4 512MB 10.2.8 SeaMonkey (test) > barcelona PMG4 1x867MHz G4 512MB 10.2.8 SeaMonkey (rel), Mozilla1.7 (test) These are good candidates for round 1; I haven't seen either of those machines used for anything aside from tinderbox since late June 2005.
I am going to completely wipe out and install fresh OS's on monkey and barcelona this afternoon. Please let me know if this is NOT okay.
Will not be updating these boxes (monkey and barcelona) today since Paul needs to make backups of the boxes and this is taking longer than expected (older hardware, etc.).
Is it time to add another XServ to the build farm? Do we have room at the colo?
Room yes, power - questionable. What I would like to see is xserve's replace the 3-4 desktops we have before adding more machines. Not sure if there is a need for increased capacity or just OS upgrades - preed? -Justin (In reply to comment #20) > Is it time to add another XServ to the build farm? Do we have room at the > colo? >
We could also perhaps move some desktops from the colo into the machine room at the office if that would help; we never set up most of the stuff that was going to go in there, so there should be room unless somebody filled it with something.
(In reply to comment #19) > Will not be updating these boxes (monkey and barcelona) today since Paul needs > to make backups of the boxes and this is taking longer than expected (older > hardware, etc.). This is done; feel free to re-install/wipe monkey and barcelona.
Will try to upgrade Barcelona to 10.3.* today. The other box (monkey) only has a cdrom drive and OS X tiger ships in a DVD format only. To get CDs for it, we have to ship back the original DVD, the receipt and $10. Marcia is looking into this.
Aravind - as for only having a CD drive... What are the specs on some of these older boxes? I know Pawn is pretty old (B&W G3?), maybe an Xserver could replace multiple older boxes. If we're going to do that, you don't need to send in the DVD.
If monkey only has a CD drive, it wouldn't be surprising if none of the older G3/G4 units could accomodate DVDs.
barcelona has been updated to 10.3 today. We installed panther with all the security updates. We have enabled ssh and installed vnc on it. This is now over to preed to install Xcode and whatever else is needed for builds.
Mac SeaMonkey builds don't have canvas and svg enabled. Perhaps that could be fixed at the same time (while setting up the tinderbox scripts)? At least for barcelona?
We will try to install os X on monkey today. Let me know if you want us to hold off on this.
Don't hold off, please, do as many boxes as you can!
(In reply to comment #30) > Don't hold off, please, do as many boxes as you can! As someone who has to get these machines working again after they've been upgraded, I'd much prefer it on some sort of schedule, actually.
Paul, Please back up the remaining boxes in the list and let us know when you are ready for us to wipe them out and reinstall the O.S on them.
We've ordered a couple of new Xserves; it's likely that we'll be moving builds to them, as opposed to wiping things out (to start, anyway; we only have two, so that juggling act only works for so long). I've got a copy of XCode 1.5 for monkey, and barcelona will (hopefully) get re-installed today with Tiger today.
I want to make sure that nobody tosses out the boxes that aren't on the list. Some, which are already running 10.3, should stay that way to handle building 1.7-based products, which won't build on 10.4 out of the box. That was sort of implicit in the upgrade plan, but I don't know if anyone's said it aloud or given a reason.
monkey has been upgraded to 10.4.*. Over to paul to get the rest of the environment ready for builds.
(In reply to comment #35) > monkey has been upgraded to 10.4.*. Over to paul to get the rest of the > environment ready for builds. Monkey and Barcelona were the first on the list, so that's what we did; but in talking with Mark on IRC, we're going to defer getting monkey and barcelona up, since we've got two xserves on their way as replacements.
Per Apple: Due to an unexpected delay, we were unable to ship your product(s) by the date originally quoted to you. We now anticipate shipping the following item(s) as follows: Z0BH, XSERVE G5 2.3GHZ DP CTO will now ship by Mar 03, 2006 and deliver by Mar 06, 2006
Blocks: 328854
Blocks: 329150
Apple now says: Estimated Shipped By Estimated Delivered By Mar 9, 2006 Mar 10, 2006 One is already racked, We'll get this one racked ASAP.
(In reply to comment #38) > Apple now says: > > Estimated Shipped By Estimated Delivered By > Mar 9, 2006 Mar 10, 2006 > > One is already racked, We'll get this one racked ASAP. > Justin, did it arrive? When can we expect a new nightly for Mac?
The first box arrived, was racked and built (OS and XCode) mid last week (handed off to Paul). The second one will be racked today, but you shouldn't need two to get builds working. Paul could give you updated status on where the build process is... -Justin (In reply to comment #39) > (In reply to comment #38) > > Apple now says: > > > > Estimated Shipped By Estimated Delivered By > > Mar 9, 2006 Mar 10, 2006 > > > > One is already racked, We'll get this one racked ASAP. > > > > Justin, > did it arrive? When can we expect a new nightly for Mac? >
(In reply to comment #40) > Paul could give you updated status on where the build process is... This bug is blocked by getting bug 329533 fixed. I don't have a concrete ETA.
(In reply to comment #41) > This bug is blocked by getting bug 329533 fixed. I don't have a concrete ETA. 329533 is 1.7-only, the requiements haven't changed on that branch. Since there are now new Xserves, I don't care if barcelona stays 10.2 for all of eternity.
(In reply to comment #42) > (In reply to comment #41) > > This bug is blocked by getting bug 329533 fixed. I don't have a concrete ETA. > > 329533 is 1.7-only, the requiements haven't changed on that branch. Since > there are now new Xserves, I don't care if barcelona stays 10.2 for all of > eternity. Yes, I understand. My point is that the Suite 1.7 builds come before upgrades necessary for universal builds. The Suite 1.7 builds would have taken zero time to deliver, but we blew barcelona away (as requested by the upgrade plan), and had to spend 3+ days finding hardware, re-installing, configuring, etc., etc., etc. to re-create an environment that could build the Suite 1.7 builds. That is all time that could've been used for getting the new Xserves ready for universal binaries, but it had to be diverted to fixing Suite builds. Also, isn't bug 329801 blocking universal binary support?
Justin and Paul, what does that mean now? There are several extremely annoying bugs that make printing impossible and are already fixed (e.g. Bug 324141 , Bug 326363 , Bug 327492 ) but no new nightlies were released since Feb. 15th ( Bug 328854 ). Please don't tell us that 95% of Mac users with PPCs have to wait for almost a month now for universal binaries because 5% of the users who have MacIntels already do not want to use Rosetta. Some users here put great effort in pushing the problem and responsibility to someone else and around in circles, but nobody seems to have any interest in informing us about progress and timeframe. Any guess when we will be able to print again (with correct paper orientation)?
> Some users here put great effort in pushing the problem and responsibility to > someone else and around in circles The Mozilla IT staff and developers are doing all we can to get this done as soon as possible. Nobody is pushing off problems or responsibility. > Any guess when we will be able to > print again (with correct paper orientation)? Please file a bug or take this discussion to a bug that has already been filed.
(In reply to comment #45) > > > Any guess when we will be able to > > print again (with correct paper orientation)? > > Please file a bug or take this discussion to a bug that has already been filed. > Josh, I did file a bug concerning this problem! Your cheerleader status obviously prevented you from clicking on the bugs I posted before. All of them deal with this problem and they all have the status "RESOLVED". However this does not help unless new nightlies get posted.
(In reply to comment #46) > I did file a bug concerning this problem! Your cheerleader status obviously > prevented you from clicking on the bugs I posted before. All of them deal with > this problem and they all have the status "RESOLVED". However this does not > help unless new nightlies get posted. I will ping preed to see if there is anything I can do help get this resolved sooner, but I can pretty much guarantee that name-calling will not improve your chances.
(In reply to comment #46) > > I did file a bug concerning this problem! Your cheerleader status obviously > prevented you from clicking on the bugs I posted before. All of them deal with > this problem and they all have the status "RESOLVED". However this does not > help unless new nightlies get posted. > One of the things that I like so much about this project is that you're not tied down to anyone else's schedule. You don't have to wait for ppc nightlies to come out, nor do you have to wait for universal binaries. If you've got the patches you need then all you have to do is build your own. Read the mozilla dev pages or check out mozillazine if you need help getting started.
Universal binaries won't happen until bug 329801 gets in. Taking, since (the first part of) this is on my plate now.
Assignee: aravind → preed
Status: ASSIGNED → NEW
Depends on: 329801
*** Bug 299214 has been marked as a duplicate of this bug. ***
Bah, the old bug had all the deps.
Blocks: 214164, 298047, 300047, 300947, 313309, 322578, 325065
Any news about when new nightlies will be available?
Not a lot of action here recently. The new Xserves lighten the load as far as producing universal binaries goes, but many of the other boxes still need to be upgraded. Let me know if you think the upgrade plan needs to be revised.
Hilo trunk builds (Sunbird & Lightning) are currently in flames since they have the old XCode. Sunbird is hoping to cut a release off the trunk soon, so this is a relatively high priority for us. Is there anything we can do to help get it upgraded?
(In reply to comment #54) > Hilo trunk builds (Sunbird & Lightning) are currently in flames since they have > the old XCode. Hilo is back in business. Thanks!
No longer blocks: sunbird-0.3a2
Earlier today, I upgraded hilo, boxset, and pawn to 10.3.9/Xcode 1.5, the new minimum. boxset is still on the list for an upgrade to 10.4. hilo isn't, but maybe we should revisit that if calendar is interested in doing universal releases.
We've lost the seamonkey nightly trunk builds that was produced by barcelona. Any chance getting a replacement machine with (at least) 10.3.9 for those?
something that can produce universal builds of seamonkey would be nice too :-)
In case it comes back to bite us, the affected builds are: hilo: Lt-Mozilla1.8 Lt-Trunk Sb-Trunk Tb-Trunk boxset: Camino Thunderbird pawn: camino
Will we be using one of the new Xserves for the missing SeaMonkey Mac build, or will we be repurposing one of the other Mac machines once their builds are migrated to the Xserves? (In reference to bug 328854)
Could somebody explain the reason why there hasn't been any noticable development on the issue in two weeks, and how we/I could help to finally get new Mac nightlies of SeaMonkey?
The build team's been putting in 240% for the 1.8.0.2 and 1.7.13 releases. (Thanks, Paul!) Nobody's going to forget about this bug.
Depends on: 334456
(In reply to comment #62) > Nobody's going to forget about this bug. > Are you sure? No news since more then a month now. No Seamonkey nightlies since January 5th!!!! ANYBODY working on these problems????
(In reply to comment #63) > No Seamonkey nightlies since January 5th Your problem is bug 328854 and not the one here.
(In reply to comment #64) > (In reply to comment #63) > > No Seamonkey nightlies since January 5th > > Your problem is bug 328854 and not the one here. > Robert, I may remind you of your post in that bug: https://bugzilla.mozilla.org/show_bug.cgi?id=328854#c6 . There you state that that bug depends on this one here - which is also documented in the dependency tree. So, what's going on? Comments always point to the "other" bug and nobody seems to work on either...
*** Bug 335506 has been marked as a duplicate of this bug. ***
Blocks: 307699
An update on this (long-standing) bug: We have two PPC Xserves that will be joining the community network shortly; one will be Lightning/Sunbird's, to build whatever they need built, and one will be Camino's. The Xserve already in the community network will be used for Seamonkey builds. All of these machines meet the minimal requirements. We're planning on doing this within the next couple of weeks.
Whiteboard: ETA 2/21/07
I finally get to resolve this bug; Camino, Sunbird, and Seamonkey all have a Mac with a sufficiently-recent version of OS X on them in the colo for use. As a reminder, hilo, pawn, binus, boxset, and maya will be turned off at the end of April. If we need to ship them to someone, we can look into what would be required to do that.
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: