552924 - Allow distinguishing Universal ppc/i386 from Universal i386/x86_64 builds in AUS request

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[(not currently active) Ted Mielczarek]

Currently for any OS X Universal build we hardcode "Universal-gcc3" as the ABI portion of the AUS request:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#170

It's my understanding that bug 519060 intends to have us shipping Universal i386/x86_64 builds as a separate binary alongside our existing Universal ppc/i386 builds. In order for updates to work for both builds, we're going to have to sort this out somehow.

Comment 1

[Robert Strong (they/them - no direct email)]

Since the OS_ABI part of the url is app update specific I see no reason why we shouldn't be able to extend that to include whether it is universal.

Instead of Darwin_Universal-gcc3 it could become the real OS and ABI with either a trailing -u or _u or adding it to the middle to denote that it is universal.

cc'ing a couple of people that are interested in the update ping url

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That would duplicate the %OS_VERSION% part of the URL. Counter proposal:
* %BUILD_TARGET% is Darwin_Universal-gcc3 or Darwin_Universal-gcc42 (or whatever the Build guys decide is appropriate; we're not guaranteeing binary compat for the two Universal builds, right ?)
* %OS_VERSION% is extended to include the CPU architecture, so that we can avoid updating existing PPC users to a 386 Universal

That's much better from an AUS & RelEng point of view, because AUS is looking up data in a directory tree in the form
  .../Firefox/%VERSION%/%BUILD_TARGET%/%BUILD_ID% ....
If we have to track multiple OS versions at the %BUILD_TARGET% level then that's a major maintenance hassle.

Comment 3

[Robert Strong (they/them - no direct email)]

Nick, %BUILD_TARGET% *is* OS and ABI
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#1984

and it is always Universal-gcc3 for universal builds currently
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#171

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

But you really mean the build systems OS_ARCH when you say OS in comment #3, eg WINNT, Linux, Darwin. What should I take "real OS" to mean in comment #1 ?

Comment 5

[Robert Strong (they/them - no direct email)]

Sure... I was just using the term as provided by XULAppInfo, etc. which is OS. There is ambiguity in terms like OS, BUILD_TARGET and without looking at the source both can be confusing.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

We'll need to fix this before enabling nightly 10.6 builds, otherwise we'll end up downgrading x86_64 users to ppc/i386 via nightly updates.

Comment 7

[Robert Strong (they/them - no direct email)]

Is there a date yet for when updates will be offered? I planned on looking at this in the next week or two after I finish up my Q1 deliverables.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

CC'ing Mike Taylor and Ben Hearsum, who are working on that.

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to comment #6)
> We'll need to fix this before enabling nightly 10.6 builds, otherwise we'll end
> up downgrading x86_64 users to ppc/i386 via nightly updates.

AFAIK, the plan was to do just straight x86-64 OS X builds, which shouldn't have this same problem. I think we should keep this bug open until we're completely sure we never want to ship an x86-64/i386 build, but I don't think it needs to block shipping nightlies.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Ted is quite correct, as ever. Here's the update query from one of the test builds bear has made:
https://aus2.mozilla.org/update/3/Firefox/3.7a4pre/20100326142052/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2010.2.0/default/default/update.xml?force=1

The Darwin_x86_64-gcc3 for %BUILD_TARGET% distinguishes use from the existing ppc/i386 universal build. Please ignore comment #6.

Comment 11

[Robert Strong (they/them - no direct email)]

Josh asked me to fix this since it blocks the work in bug 571367... any reason not to go with what I outlined in comment #1?

Comment 12

[(not currently active) Ted Mielczarek]

The caveat there would be that we'd hit a different URL depending on what arch you were running from the binary, no? So for our existing Universal ppc/i386 builds, when run from a PPC machine you'd get "ppc-gcc3-u" and when run from an x86 machine you'd get "i386-gcc3-u".

Comment 13

[Robert Strong (they/them - no direct email)]

Exactly. This would just work if / when there are similar changes in the future and I believe AUS could serve up the appropriate update xml for these without having to hardcode the (incorrect) ABI in the client as we do today.

Comment 14

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

nthomas, this will likely require work on the AUS side so I'm running this by you first... are you ok with changing AUS to handle the Mac universal updates in this manner?

Comment 15

[Robert Strong (they/them - no direct email)]

btw: I went with -universal so the meaning is better understood.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 450709 [details] [diff] [review]
patch rev1

I made this change on a ppc/i386 universal build, and the AUS query used 'Darwin_x86-gcc3-universal'. The current 64bit builds use 'Darwin_x86_64-gcc3'. So it looks like a i386/x86_64 universal build running on i386 will say the same as a ppc/i386 build on i386, which doesn't allow us to provide the correct update file to each.

Comment 17

[Robert Strong (they/them - no direct email)]

bah... didn't realize both dmg's were going to contain the same ABI and essentially the same build.

Comment 18

[Robert Strong (they/them - no direct email)]

There is no way that I know of for app update to directly figure out which is which under this scenario. You could change Firefox's app update url preference during build time so the two are distinguishable as a short term fix or somehow add this to the MacUtils implementation.

Comment 19

[(not currently active) Ted Mielczarek]

I think it'd be trivial to change the macutils impl to distinguish these two cases, since it already reads the fat Mach-O binary and looks for both architectures:
http://mxr.mozilla.org/mozilla-central/source/xpcom/base/nsMacUtilsImpl.cpp#113

In fact, as written, a universal i386/x86_64 binary would have .isUniversal == false, since it doesn't contain a PPC binary. :)

Comment 20

[Robert Strong (they/them - no direct email)]

Thanks Ted, filed bug 572125 for this

Comment 21

[Robert Strong (they/them - no direct email)]

Nick, the value that will be added to the update url using Josh's patch in bug 572125 to distinguish what the universal contains will be as follows
ppc-i386
and
i386-x86_64

I'd like to also include the currently running architecture so there is an upgrade path for those running i386 with a ppc-i386 universal binary to the i386-x86_64 universal binary.

Any reason we can't use Darwin_x86-gcc3-u-ppc-i386, Darwin_x86-gcc3-u-i386-x86_64, etc. or something to that affect?

https://aus2.mozilla.org/update/3/Firefox/3.6.3/20100401064631/Darwin_x86-gcc3-u-ppc-i386/en-US/release/Darwin%2010.4.0/default/default/update.xml?force=1

https://aus2.mozilla.org/update/3/Firefox/3.6.3/20100401064631/Darwin_x86-gcc3-u-i386-x86_64/en-US/release/Darwin%2010.4.0/default/default/update.xml?force=1

etc.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Does this capture the proposed changes correctly ?

Darwin_Universal-gcc3 (existing universal build) will start reporting as
 --> Darwin_ppc-gcc3-u-ppc-i386      (running on ppc)
 --> Darwin_i386-gcc3-u-ppc-i386     (running on i386, most of our mac users)

Unchanged: 
 Darwin_x86_64-gcc3
 Linux_x86_64-gcc3
 Linux_x86-gcc3
 WINNT_x86_64-msvc
 WINNT_x86-msvc

When the existing 64bit build (Darwin_x86_64-gcc3) becomes universal we have 
 Darwin_i386-gcc3-u-i386-x86_64      (running on i386)
 Darwin_x86_64-gcc3-u-i386-x86_64    (running on x86_64)

And in future we will want to offer an update path from Darwin_i386-gcc3-u-ppc-i386 to the i386/x86_64 universal build. 

The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or possibly 10.6.2). Can we have a universal binary with an i386 part supporting 10.5 and up, combined with the x86_64 for 10.6 only ?

Comment 23

[(not currently active) Ted Mielczarek]

(In reply to comment #22)
> The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or
> possibly 10.6.2). Can we have a universal binary with an i386 part supporting
> 10.5 and up, combined with the x86_64 for 10.6 only ?

Yes, we can set mozconfig options per-architecture (including the SDK).

Comment 24

[Robert Strong (they/them - no direct email)]

Attachment: patch - uses macutils.architecturesInBinary

Nick, this implements what has been discussed in the bug.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

rs, Can I get a reply to comment #22 ? Then I can evaluate how much pain snippet generation is going to cause us.

Comment 26

[Robert Strong (they/them - no direct email)]

Sorry about that... I mistakenly thought Ted had answered it.

(In reply to comment #22)
> Does this capture the proposed changes correctly ?
> 
> Darwin_Universal-gcc3 (existing universal build) will start reporting as
>  --> Darwin_ppc-gcc3-u-ppc-i386      (running on ppc)
>  --> Darwin_i386-gcc3-u-ppc-i386     (running on i386, most of our mac users)
Correct

> Unchanged: 
>  Darwin_x86_64-gcc3
>  Linux_x86_64-gcc3
>  Linux_x86-gcc3
>  WINNT_x86_64-msvc
>  WINNT_x86-msvc
Correct

> When the existing 64bit build (Darwin_x86_64-gcc3) becomes universal we have 
>  Darwin_i386-gcc3-u-i386-x86_64      (running on i386)
>  Darwin_x86_64-gcc3-u-i386-x86_64    (running on x86_64)
Correct

Just for completeness, we'll also have the previous two you mentioned
Darwin_ppc-gcc3-u-ppc-i386
Darwin_i386-gcc3-u-ppc-i386

> And in future we will want to offer an update path from
> Darwin_i386-gcc3-u-ppc-i386 to the i386/x86_64 universal build. 
Correct. After ppc support is dropped we'll want to migrate Darwin_i386-gcc3-u-ppc-i386 to the Darwin i386 / x86_64 universal.

> The existing Darwin_x86_64-gcc3 build is targeted is built for 10.6 (or
> possibly 10.6.2). Can we have a universal binary with an i386 part supporting
> 10.5 and up, combined with the x86_64 for 10.6 only ?
Ted answered this in comment #23

Additional note: there have been bug reports from people that have stripped the unused architecture from the universal. Would you be ok with updating them to the appropriate build? If so, I will file a followup.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #26)
> Additional note: there have been bug reports from people that have stripped the
> unused architecture from the universal. Would you be ok with updating them to
> the appropriate build? If so, I will file a followup.

That bears on the changes we'd to support this bug. I think you mean that we should update (with completes only) from
  Darwin_ppc-gcc3  -->  ppc/i386 universal build
  Darwin_i386-gcc3 -->  ppc/i386 universal build
for now, and later
  Darwin_ppc-gcc3  -->  nowhere after ppc is dropped
  Darwin_i386-gcc3 -->  i386/x86_64 universal build
Sound right ?

For us this bug boils down to adding a level of abstraction between the builds we have have and how we update people. I'm trying to figure out the right place to do this.

Comment 28

[Robert Strong (they/them - no direct email)]

(In reply to comment #27)
> (In reply to comment #26)
> > Additional note: there have been bug reports from people that have stripped the
> > unused architecture from the universal. Would you be ok with updating them to
> > the appropriate build? If so, I will file a followup.
> 
> That bears on the changes we'd to support this bug. I think you mean that we
> should update (with completes only) from
>   Darwin_ppc-gcc3  -->  ppc/i386 universal build
>   Darwin_i386-gcc3 -->  ppc/i386 universal build
> for now, and later
>   Darwin_ppc-gcc3  -->  nowhere after ppc is dropped
>   Darwin_i386-gcc3 -->  i386/x86_64 universal build
> Sound right ?
Correct with the addition of
Darwin_x86_64-gcc3 -->  i386/x86_64 universal build

> For us this bug boils down to adding a level of abstraction between the builds
> we have have and how we update people. I'm trying to figure out the right place
> to do this.
Might be best to do the stripped universal build to the appropriate universal complete mar in a separate bug since bug 519060 is blocking2.0=BetaN+ which this bug blocks.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #28)
> Might be best to do the stripped universal build to the appropriate universal
> complete mar in a separate bug since bug 519060 is blocking2.0=BetaN+ which
> this bug blocks.

OK. It looks like only 1 in a 1000 mac users are stripping their binary anyway.

Comment 30

[Robert Strong (they/them - no direct email)]

Comment on attachment 458773 [details] [diff] [review]
patch - uses macutils.architecturesInBinary

Nick, I'm going to get review for this so it can be checked in as soon as the aus side is ready. Just let me know in this bug when things are ready. Thanks.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 458773 [details] [diff] [review]
patch - uses macutils.architecturesInBinary

Sorry for the delay in responding to the feedback request. I'm a little nervous about combining the arch that's running with the fixed value which describes the build-type the user has. In a perfect world we might move the running arch somewhere else, like %OS_VERSION%. However AUS doesn't have any means to support this - all it currently does with %OS_VERSION% is a simple blocklist of major updates for de-supported OSes. Adding better support for OS_VERSION, or adding a new parameter, significantly increases the scope of work required.  Even if we did that we'd still have to enhance the utilities we use to generate snippets, adding a new mechanism to map the varied arch requests to a smaller set of builds, so we might as well do all the work there. Apache rewrites or symlinks can help us in the short term.

Anyway, that's a long way to say feedback+. bug 583671 to track the RelEng work to support this.

Comment 32

[Robert Strong (they/them - no direct email)]

Would it be easier if it used the following formatting?

Darwin_ppc-i386-gcc3
Darwin_x86_64-gcc3

and add a param to the url for the current architecture being used in the binary?

I believe the stripped universal binaries then have.
Darwin_ppc-gcc3
Darwin_x86-gcc3
Darwin_x64-gcc3

The extra param for the current could only be added when it is universal or always for Mac OS X... whichever is easier.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #32)
> Would it be easier if it used the following formatting?

It would be easier in the sense that it requires only a small change to snippet generation to continue the current arrangement of builds. But it doesn't help much when we come to transition users from one universal to another, and adds the burden of changing AUS too. Lets just go ahead with what you have and we'll figure something out our side.

Comment 34

[Daniel Einspanjer [:dre] [:deinspanjer]]

Are any of these patches landed in nightlies or betas yet?

I was just doing some searching in our blocklist and aus data and I don't see anything new in blocklist. I found one ping in AUS..
We are trying to calculate some breakdowns of PPC vs Intel users for Beltzner and having trouble figuring out where to extract the data.

# select product_build_target, sum(requests) from aus_requests_by_day where utc_year = 2010 and utc_month = 8 and product_build_target like 'Darwin%' group by 1 order by 2 desc limit 20

       product_build_target       |    sum    
----------------------------------+-----------
 Darwin_Universal-gcc3            | 121044309
 Darwin_x86-gcc3                  |   1798368
 Darwin_ppc-gcc3                  |    181007
 Darwin_x86_64-gcc3               |     16586
 Darwin_Universal-gcc2            |         1
 Darwin_Universal-gcc4            |         1
 Darwin                           |         1
 Darwin_x86_64-gcc3-u-i386-x86_64 |         1
(8 rows)

Comment 35

[Robert Strong (they/them - no direct email)]

(In reply to comment #34)
> Are any of these patches landed in nightlies or betas yet?
No. Bug 583671 will need to be fixed first and I'll resolve this bug as fixed when this bug lands (it will be a single patch landing for this bug).

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Please go ahead and land before bug 583671 is done.

Comment 37

[Robert Strong (they/them - no direct email)]

Attachment: patch for checkin

Comment 38

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-central
http://hg.mozilla.org/mozilla-central/rev/0e40a49c27bb

There are already tests for this so marking in-testsuite+

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Looks like I introduced a red-herring at comment #22. It should be 
Darwin_x86-gcc3-u-ppc-i386 not Darwin_i386-gcc3-u-ppc-i386, ie it's x86 not i386 for the 32bit Intel case.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

How will we support major updates from the 3.5 and 3.6 branches ? We will want to block PPC updates from there to 4.0 as a result of dropping PPC as a supported arch. We could backport this bug and bug 572125, or come up with an alternative version of bug 588412 that does the blocking in AUS. What do you think Rob ?

Comment 41

[Robert Strong (they/them - no direct email)]

We could get away with prepending the real abi so we know what they are currently running to the update url without backporting bug 572125 since we only have i386/ppc universals on 3.6.

Comment 42

[Robert Strong (they/them - no direct email)]

Attachment: patch for 1.9.2

Nick, I am fairly certain this is all that is needed for 1.9.1 (if we want to update users from 1.9.2 to 2.0) and 1.9.2. I went with the same format as we use on trunk now. We'll need to let 3rd parties (e.g. Songbird, etc.) about this change as well. Does this look good to you?

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 478623 [details] [diff] [review]
patch for 1.9.2

Looks fine to me, as it gives us the same values we have for 4.0 (eg Darwin_x86-gcc3-u-ppc-i386 on Intel).

Comment 44

[Robert Strong (they/them - no direct email)]

Nick, since this will likely require changes to aus by releng when can this land?

Comment 45

[Nick Thomas [:nthomas] (UTC+12)]

Should just be changes to patcher, similar to what we need for 4.0, so no restriction from us.

Comment 46

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #38)
> Pushed to mozilla-central
> http://hg.mozilla.org/mozilla-central/rev/0e40a49c27bb

... to be in b5, but not the earlier 4.0 betas or 3.7a* devpreviews.

Comment 47

[Robert Strong (they/them - no direct email)]

Requesting blocking on 1.9.2.

This is needed on at least 1.9.2 (e.g. Firefox 3.6.x) so we can distinguish between systems running i386 vs ppc so we can update the systems running i386 to the i386/x86_64 universal binaries and not update the ppc systems.

Do we want the ability to update from 1.9.1 (e.g. Firefox 3.5.x) to 2.0 (e.g. Firefox 4.0.x)?

Comment 48

[Nick Thomas [:nthomas] (UTC+12)]

We've been doing updates from 3.0.19 to 3.6.x, as well as from 3.5.x. So I suspect we do, even if 3.5 soon reaches end-of-life.

Comment 49

[LegNeato]

We'll want this on both branches

Comment 50

[Mike Beltzner [:beltzner, not reading bugmail]]

When it goes to branches, please make sure deispanjer and the metrics team are ready for it!

Comment 51

[Robert Strong (they/them - no direct email)]

I cc'd him to this bug a long time ago for just that reason but I'll email him just to make sure

Comment 52

[Robert Strong (they/them - no direct email)]

cc'ing a few people that might be interested (already spoke with Mook on irc).

Comment 53

[Mook (songbird dead account)]

(In reply to comment #52)
> cc'ing a few people that might be interested (already spoke with Mook on irc).

Thanks for the heads up; the corresponding Songbird bug is http://bugzilla.songbirdnest.com/show_bug.cgi?id=22610 (but there's nothing to see there, it's just a note that we might have to think about it).

Comment 54

[LegNeato]

Comment on attachment 478623 [details] [diff] [review]
patch for 1.9.2

a=LegNeato for 1.9.2.13

Comment 55

[LegNeato]

Comment on attachment 478623 [details] [diff] [review]
patch for 1.9.2

...and a=1.9.1.16 as well

Comment 56

[Robert Strong (they/them - no direct email)]

Landed on mozilla-1.9.1 and mozilla-1.9.2

Pushed to mozilla-1.9.2
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/5e157173397f

Pushed to mozilla-1.9.1
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/d16805f53fa1

Comment 57

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 478623 [details] [diff] [review]
patch for 1.9.2

>   if (macutils.isUniversalBinary)
>-    abi = "Universal-gcc3";
>+    abi += "-u-ppc-i386";
> #endif

This patch hasn't been updated this line correctly on 1.9.1 and has changed gAbi to abi, which probably causing bug 610075.

Comment 58

[Robert Strong (they/them - no direct email)]

Pushed a followup bustage fix to 1.9.1
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/24b02f04fb6d

Comment 59

[Al Billings [:abillings - ex-MoCo]]

The followup fix needs to be pushed to 1.9.2 as well. The same problem is happening there. Users are stranded in 1.9.1 and 1.9.2 on pre-fix builds right now too.

Comment 60

[Nick Thomas [:nthomas] (UTC+12)]

I don't think we need it on 1.9.2. I took the 2010-11-05 build and didn't get any errors in the console about gAbi/abi. Once I helped AUS find the update snippets for the new %BUILD_TARGET% (bug 583671 comment #5) I was able to update to 2010-11-07 without issues.

Also verified on 1.9.1 the 2010-11-06 build updates to -07 OK. Both of those for Intel.