Closed Bug 552924 Opened 12 years ago Closed 12 years ago
Allow distinguishing Universal ppc/i386 from Universal i386/x86
_64 builds in AUS request
1.81 KB, patch
|Details | Diff | Splinter Review|
1.99 KB, patch
|Details | Diff | Splinter Review|
2.19 KB, patch
|Details | Diff | Splinter Review|
Currently for any OS X Universal build we hardcode "Universal-gcc3" as the ABI portion of the AUS request: http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#170 It's my understanding that bug 519060 intends to have us shipping Universal i386/x86_64 builds as a separate binary alongside our existing Universal ppc/i386 builds. In order for updates to work for both builds, we're going to have to sort this out somehow.
Since the OS_ABI part of the url is app update specific I see no reason why we shouldn't be able to extend that to include whether it is universal. Instead of Darwin_Universal-gcc3 it could become the real OS and ABI with either a trailing -u or _u or adding it to the middle to denote that it is universal. cc'ing a couple of people that are interested in the update ping url
That would duplicate the %OS_VERSION% part of the URL. Counter proposal: * %BUILD_TARGET% is Darwin_Universal-gcc3 or Darwin_Universal-gcc42 (or whatever the Build guys decide is appropriate; we're not guaranteeing binary compat for the two Universal builds, right ?) * %OS_VERSION% is extended to include the CPU architecture, so that we can avoid updating existing PPC users to a 386 Universal That's much better from an AUS & RelEng point of view, because AUS is looking up data in a directory tree in the form .../Firefox/%VERSION%/%BUILD_TARGET%/%BUILD_ID% .... If we have to track multiple OS versions at the %BUILD_TARGET% level then that's a major maintenance hassle.
Nick, %BUILD_TARGET% *is* OS and ABI http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#1984 and it is always Universal-gcc3 for universal builds currently http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#171
But you really mean the build systems OS_ARCH when you say OS in comment #3, eg WINNT, Linux, Darwin. What should I take "real OS" to mean in comment #1 ?
Sure... I was just using the term as provided by XULAppInfo, etc. which is OS. There is ambiguity in terms like OS, BUILD_TARGET and without looking at the source both can be confusing.
We'll need to fix this before enabling nightly 10.6 builds, otherwise we'll end up downgrading x86_64 users to ppc/i386 via nightly updates.
Is there a date yet for when updates will be offered? I planned on looking at this in the next week or two after I finish up my Q1 deliverables.
CC'ing Mike Taylor and Ben Hearsum, who are working on that.
(In reply to comment #6) > We'll need to fix this before enabling nightly 10.6 builds, otherwise we'll end > up downgrading x86_64 users to ppc/i386 via nightly updates. AFAIK, the plan was to do just straight x86-64 OS X builds, which shouldn't have this same problem. I think we should keep this bug open until we're completely sure we never want to ship an x86-64/i386 build, but I don't think it needs to block shipping nightlies.
Ted is quite correct, as ever. Here's the update query from one of the test builds bear has made: https://aus2.mozilla.org/update/3/Firefox/3.7a4pre/20100326142052/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2010.2.0/default/default/update.xml?force=1 The Darwin_x86_64-gcc3 for %BUILD_TARGET% distinguishes use from the existing ppc/i386 universal build. Please ignore comment #6.
Josh asked me to fix this since it blocks the work in bug 571367... any reason not to go with what I outlined in comment #1?
The caveat there would be that we'd hit a different URL depending on what arch you were running from the binary, no? So for our existing Universal ppc/i386 builds, when run from a PPC machine you'd get "ppc-gcc3-u" and when run from an x86 machine you'd get "i386-gcc3-u".
Exactly. This would just work if / when there are similar changes in the future and I believe AUS could serve up the appropriate update xml for these without having to hardcode the (incorrect) ABI in the client as we do today.
nthomas, this will likely require work on the AUS side so I'm running this by you first... are you ok with changing AUS to handle the Mac universal updates in this manner?
Attachment #450709 - Flags: review?(nrthomas)
btw: I went with -universal so the meaning is better understood.
Attachment #450709 - Flags: review?(nrthomas) → feedback?(nrthomas)
Comment on attachment 450709 [details] [diff] [review] patch rev1 I made this change on a ppc/i386 universal build, and the AUS query used 'Darwin_x86-gcc3-universal'. The current 64bit builds use 'Darwin_x86_64-gcc3'. So it looks like a i386/x86_64 universal build running on i386 will say the same as a ppc/i386 build on i386, which doesn't allow us to provide the correct update file to each.
Attachment #450709 - Flags: feedback?(nrthomas) → feedback-
bah... didn't realize both dmg's were going to contain the same ABI and essentially the same build.
There is no way that I know of for app update to directly figure out which is which under this scenario. You could change Firefox's app update url preference during build time so the two are distinguishable as a short term fix or somehow add this to the MacUtils implementation.
I think it'd be trivial to change the macutils impl to distinguish these two cases, since it already reads the fat Mach-O binary and looks for both architectures: http://mxr.mozilla.org/mozilla-central/source/xpcom/base/nsMacUtilsImpl.cpp#113 In fact, as written, a universal i386/x86_64 binary would have .isUniversal == false, since it doesn't contain a PPC binary. :)
Thanks Ted, filed bug 572125 for this
Nick, the value that will be added to the update url using Josh's patch in bug 572125 to distinguish what the universal contains will be as follows ppc-i386 and i386-x86_64 I'd like to also include the currently running architecture so there is an upgrade path for those running i386 with a ppc-i386 universal binary to the i386-x86_64 universal binary. Any reason we can't use Darwin_x86-gcc3-u-ppc-i386, Darwin_x86-gcc3-u-i386-x86_64, etc. or something to that affect? https://aus2.mozilla.org/update/3/Firefox/3.6.3/20100401064631/Darwin_x86-gcc3-u-ppc-i386/en-US/release/Darwin%2010.4.0/default/default/update.xml?force=1 https://aus2.mozilla.org/update/3/Firefox/3.6.3/20100401064631/Darwin_x86-gcc3-u-i386-x86_64/en-US/release/Darwin%2010.4.0/default/default/update.xml?force=1 etc.
Does this capture the proposed changes correctly ? Darwin_Universal-gcc3 (existing universal build) will start reporting as --> Darwin_ppc-gcc3-u-ppc-i386 (running on ppc) --> Darwin_i386-gcc3-u-ppc-i386 (running on i386, most of our mac users) Unchanged: Darwin_x86_64-gcc3 Linux_x86_64-gcc3 Linux_x86-gcc3 WINNT_x86_64-msvc WINNT_x86-msvc When the existing 64bit build (Darwin_x86_64-gcc3) becomes universal we have Darwin_i386-gcc3-u-i386-x86_64 (running on i386) Darwin_x86_64-gcc3-u-i386-x86_64 (running on x86_64) And in future we will want to offer an update path from Darwin_i386-gcc3-u-ppc-i386 to the i386/x86_64 universal build. The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or possibly 10.6.2). Can we have a universal binary with an i386 part supporting 10.5 and up, combined with the x86_64 for 10.6 only ?
(In reply to comment #22) > The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or > possibly 10.6.2). Can we have a universal binary with an i386 part supporting > 10.5 and up, combined with the x86_64 for 10.6 only ? Yes, we can set mozconfig options per-architecture (including the SDK).
Nick, this implements what has been discussed in the bug.
rs, Can I get a reply to comment #22 ? Then I can evaluate how much pain snippet generation is going to cause us.
Sorry about that... I mistakenly thought Ted had answered it. (In reply to comment #22) > Does this capture the proposed changes correctly ? > > Darwin_Universal-gcc3 (existing universal build) will start reporting as > --> Darwin_ppc-gcc3-u-ppc-i386 (running on ppc) > --> Darwin_i386-gcc3-u-ppc-i386 (running on i386, most of our mac users) Correct > Unchanged: > Darwin_x86_64-gcc3 > Linux_x86_64-gcc3 > Linux_x86-gcc3 > WINNT_x86_64-msvc > WINNT_x86-msvc Correct > When the existing 64bit build (Darwin_x86_64-gcc3) becomes universal we have > Darwin_i386-gcc3-u-i386-x86_64 (running on i386) > Darwin_x86_64-gcc3-u-i386-x86_64 (running on x86_64) Correct Just for completeness, we'll also have the previous two you mentioned Darwin_ppc-gcc3-u-ppc-i386 Darwin_i386-gcc3-u-ppc-i386 > And in future we will want to offer an update path from > Darwin_i386-gcc3-u-ppc-i386 to the i386/x86_64 universal build. Correct. After ppc support is dropped we'll want to migrate Darwin_i386-gcc3-u-ppc-i386 to the Darwin i386 / x86_64 universal. > The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or > possibly 10.6.2). Can we have a universal binary with an i386 part supporting > 10.5 and up, combined with the x86_64 for 10.6 only ? Ted answered this in comment #23 Additional note: there have been bug reports from people that have stripped the unused architecture from the universal. Would you be ok with updating them to the appropriate build? If so, I will file a followup.
(In reply to comment #26) > Additional note: there have been bug reports from people that have stripped the > unused architecture from the universal. Would you be ok with updating them to > the appropriate build? If so, I will file a followup. That bears on the changes we'd to support this bug. I think you mean that we should update (with completes only) from Darwin_ppc-gcc3 --> ppc/i386 universal build Darwin_i386-gcc3 --> ppc/i386 universal build for now, and later Darwin_ppc-gcc3 --> nowhere after ppc is dropped Darwin_i386-gcc3 --> i386/x86_64 universal build Sound right ? For us this bug boils down to adding a level of abstraction between the builds we have have and how we update people. I'm trying to figure out the right place to do this.
(In reply to comment #27) > (In reply to comment #26) > > Additional note: there have been bug reports from people that have stripped the > > unused architecture from the universal. Would you be ok with updating them to > > the appropriate build? If so, I will file a followup. > > That bears on the changes we'd to support this bug. I think you mean that we > should update (with completes only) from > Darwin_ppc-gcc3 --> ppc/i386 universal build > Darwin_i386-gcc3 --> ppc/i386 universal build > for now, and later > Darwin_ppc-gcc3 --> nowhere after ppc is dropped > Darwin_i386-gcc3 --> i386/x86_64 universal build > Sound right ? Correct with the addition of Darwin_x86_64-gcc3 --> i386/x86_64 universal build > For us this bug boils down to adding a level of abstraction between the builds > we have have and how we update people. I'm trying to figure out the right place > to do this. Might be best to do the stripped universal build to the appropriate universal complete mar in a separate bug since bug 519060 is blocking2.0=BetaN+ which this bug blocks.
(In reply to comment #28) > Might be best to do the stripped universal build to the appropriate universal > complete mar in a separate bug since bug 519060 is blocking2.0=BetaN+ which > this bug blocks. OK. It looks like only 1 in a 1000 mac users are stripping their binary anyway.
Comment on attachment 458773 [details] [diff] [review] patch - uses macutils.architecturesInBinary Nick, I'm going to get review for this so it can be checked in as soon as the aus side is ready. Just let me know in this bug when things are ready. Thanks.
Attachment #458773 - Flags: review?(dtownsend)
Comment on attachment 458773 [details] [diff] [review] patch - uses macutils.architecturesInBinary Sorry for the delay in responding to the feedback request. I'm a little nervous about combining the arch that's running with the fixed value which describes the build-type the user has. In a perfect world we might move the running arch somewhere else, like %OS_VERSION%. However AUS doesn't have any means to support this - all it currently does with %OS_VERSION% is a simple blocklist of major updates for de-supported OSes. Adding better support for OS_VERSION, or adding a new parameter, significantly increases the scope of work required. Even if we did that we'd still have to enhance the utilities we use to generate snippets, adding a new mechanism to map the varied arch requests to a smaller set of builds, so we might as well do all the work there. Apache rewrites or symlinks can help us in the short term. Anyway, that's a long way to say feedback+. bug 583671 to track the RelEng work to support this.
Attachment #458773 - Flags: feedback?(nrthomas) → feedback+
Would it be easier if it used the following formatting? Darwin_ppc-i386-gcc3 Darwin_x86_64-gcc3 and add a param to the url for the current architecture being used in the binary? I believe the stripped universal binaries then have. Darwin_ppc-gcc3 Darwin_x86-gcc3 Darwin_x64-gcc3 The extra param for the current could only be added when it is universal or always for Mac OS X... whichever is easier.
Attachment #458773 - Flags: review?(dtownsend) → review+
(In reply to comment #32) > Would it be easier if it used the following formatting? It would be easier in the sense that it requires only a small change to snippet generation to continue the current arrangement of builds. But it doesn't help much when we come to transition users from one universal to another, and adds the burden of changing AUS too. Lets just go ahead with what you have and we'll figure something out our side.
Are any of these patches landed in nightlies or betas yet? I was just doing some searching in our blocklist and aus data and I don't see anything new in blocklist. I found one ping in AUS.. We are trying to calculate some breakdowns of PPC vs Intel users for Beltzner and having trouble figuring out where to extract the data. # select product_build_target, sum(requests) from aus_requests_by_day where utc_year = 2010 and utc_month = 8 and product_build_target like 'Darwin%' group by 1 order by 2 desc limit 20 product_build_target | sum ----------------------------------+----------- Darwin_Universal-gcc3 | 121044309 Darwin_x86-gcc3 | 1798368 Darwin_ppc-gcc3 | 181007 Darwin_x86_64-gcc3 | 16586 Darwin_Universal-gcc2 | 1 Darwin_Universal-gcc4 | 1 Darwin | 1 Darwin_x86_64-gcc3-u-i386-x86_64 | 1 (8 rows)
(In reply to comment #34) > Are any of these patches landed in nightlies or betas yet? No. Bug 583671 will need to be fixed first and I'll resolve this bug as fixed when this bug lands (it will be a single patch landing for this bug).
Please go ahead and land before bug 583671 is done.
Pushed to mozilla-central http://hg.mozilla.org/mozilla-central/rev/0e40a49c27bb There are already tests for this so marking in-testsuite+
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Looks like I introduced a red-herring at comment #22. It should be Darwin_x86-gcc3-u-ppc-i386 not Darwin_i386-gcc3-u-ppc-i386, ie it's x86 not i386 for the 32bit Intel case.
How will we support major updates from the 3.5 and 3.6 branches ? We will want to block PPC updates from there to 4.0 as a result of dropping PPC as a supported arch. We could backport this bug and bug 572125, or come up with an alternative version of bug 588412 that does the blocking in AUS. What do you think Rob ?
We could get away with prepending the real abi so we know what they are currently running to the update url without backporting bug 572125 since we only have i386/ppc universals on 3.6.
Nick, I am fairly certain this is all that is needed for 1.9.1 (if we want to update users from 1.9.2 to 2.0) and 1.9.2. I went with the same format as we use on trunk now. We'll need to let 3rd parties (e.g. Songbird, etc.) about this change as well. Does this look good to you?
Attachment #478623 - Flags: feedback?(nrthomas)
Comment on attachment 478623 [details] [diff] [review] patch for 1.9.2 Looks fine to me, as it gives us the same values we have for 4.0 (eg Darwin_x86-gcc3-u-ppc-i386 on Intel).
Attachment #478623 - Flags: feedback?(nrthomas) → feedback+
Attachment #478623 - Flags: review?(dtownsend)
Nick, since this will likely require changes to aus by releng when can this land?
Should just be changes to patcher, similar to what we need for 4.0, so no restriction from us.
Attachment #478623 - Flags: review?(dtownsend) → review+
(In reply to comment #38) > Pushed to mozilla-central > http://hg.mozilla.org/mozilla-central/rev/0e40a49c27bb ... to be in b5, but not the earlier 4.0 betas or 3.7a* devpreviews.
tracking-fennec: --- → ?
Target Milestone: --- → mozilla2.0b5
Requesting blocking on 1.9.2. This is needed on at least 1.9.2 (e.g. Firefox 3.6.x) so we can distinguish between systems running i386 vs ppc so we can update the systems running i386 to the i386/x86_64 universal binaries and not update the ppc systems. Do we want the ability to update from 1.9.1 (e.g. Firefox 3.5.x) to 2.0 (e.g. Firefox 4.0.x)?
blocking1.9.2: --- → ?
tracking-fennec: ? → ---
We've been doing updates from 3.0.19 to 3.6.x, as well as from 3.5.x. So I suspect we do, even if 3.5 soon reaches end-of-life.
We'll want this on both branches
When it goes to branches, please make sure deispanjer and the metrics team are ready for it!
I cc'd him to this bug a long time ago for just that reason but I'll email him just to make sure
cc'ing a few people that might be interested (already spoke with Mook on irc).
(In reply to comment #52) > cc'ing a few people that might be interested (already spoke with Mook on irc). Thanks for the heads up; the corresponding Songbird bug is http://bugzilla.songbirdnest.com/show_bug.cgi?id=22610 (but there's nothing to see there, it's just a note that we might have to think about it).
Comment on attachment 478623 [details] [diff] [review] patch for 1.9.2 a=LegNeato for 188.8.131.52
Attachment #478623 - Flags: approval184.108.40.206+
Comment on attachment 478623 [details] [diff] [review] patch for 1.9.2 ...and a=220.127.116.11 as well
Attachment #478623 - Flags: approval18.104.22.168+
Landed on mozilla-1.9.1 and mozilla-1.9.2 Pushed to mozilla-1.9.2 http://hg.mozilla.org/releases/mozilla-1.9.2/rev/5e157173397f Pushed to mozilla-1.9.1 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/d16805f53fa1
Comment on attachment 478623 [details] [diff] [review] patch for 1.9.2 > if (macutils.isUniversalBinary) >- abi = "Universal-gcc3"; >+ abi += "-u-ppc-i386"; > #endif This patch hasn't been updated this line correctly on 1.9.1 and has changed gAbi to abi, which probably causing bug 610075.
Pushed a followup bustage fix to 1.9.1 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/24b02f04fb6d
The followup fix needs to be pushed to 1.9.2 as well. The same problem is happening there. Users are stranded in 1.9.1 and 1.9.2 on pre-fix builds right now too.
I don't think we need it on 1.9.2. I took the 2010-11-05 build and didn't get any errors in the console about gAbi/abi. Once I helped AUS find the update snippets for the new %BUILD_TARGET% (bug 583671 comment #5) I was able to update to 2010-11-07 without issues. Also verified on 1.9.1 the 2010-11-06 build updates to -07 OK. Both of those for Intel.
You need to log in before you can comment on or make changes to this bug.