1071576 - unstrand users who have stripped an architecture away from our official releases

Status: VERIFIED FIXED

(Release Engineering :: Release Requests, defect)

Description

[[e-mail] (:bhearsum)]

While investigating bug 1069454 I discovered that we're getting thousands of requests per day from seemingly invalid build targets, like Darwin_x86_64-gcc3. While investigating them I found reference to users stripping out the unneeded arch from their Firefox install: http://superuser.com/questions/581334/automatic-updates-of-firefox-stable-and-beta-dont-work-on-os-x - "Ah that's a good point again! Indeed I stripped the i386 part away in all (stable, beta, aurora, nightly) of my Firefox editions."

This results in the BUILD_TARGET changing to a non-universal one, which we don't serve updates for. I wish people didn't do this, but I think it would still be good to try to serve updates to them - right now we have tons of users stuck on old versions because of this.

The simple way to do this would be to add these build targets as alias'. The only downside to this (AFAIK) is that if a partial is offered to that user, it will fail and then they'll download the complete. It's not impossible to stop offering partials to these users, but I don't think it's worth the effort to implement.

Comment 1

[[e-mail] (:bhearsum)]

I filed bug 1071580 to try to guard against this on the platform side.

Comment 2

[Johnathan Nightingale [:johnath]]

Cc'ng Benjamin and Robert as FYI even though I know this is a server-side fix.

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

Great find Ben. Is there a way to tell how many instances are impacted by "thousands of requests per day"? If it's a significant amount, it would seem that our users our telling us what they want (not universal installs). It may be worth talking about how we can deliver that to them.

(In reply to Ben Hearsum [:bhearsum] from comment #0)
> The simple way to do this would be to add these build targets as alias'. The
> only downside to this (AFAIK) is that if a partial is offered to that user,
> it will fail and then they'll download the complete. It's not impossible to
> stop offering partials to these users, but I don't think it's worth the
> effort to implement.

I take it that this will result in these instances adding back the deleted i386 part of the installation that the user manually removed. I wonder how happy they'll be about that.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Lawrence Mandel [:lmandel] from comment #3)
> Great find Ben. Is there a way to tell how many instances are impacted by
> "thousands of requests per day"?

I think we'd probably be best to ask Metrics about that - I don't think I have a way to query the data I have like this. I can give them the particularls, though.

> (In reply to Ben Hearsum [:bhearsum] from comment #0)
> > The simple way to do this would be to add these build targets as alias'. The
> > only downside to this (AFAIK) is that if a partial is offered to that user,
> > it will fail and then they'll download the complete. It's not impossible to
> > stop offering partials to these users, but I don't think it's worth the
> > effort to implement.
> 
> I take it that this will result in these instances adding back the deleted
> i386 part of the installation that the user manually removed. I wonder how
> happy they'll be about that.

Yeah, good question. Assuming most of them don't know that they've made themselves insecure (which I suspect is the case), I'd argue that giving them an update to a secure build is the right thing to do -- they can re-strip later if they so choose.

Comment 5

[:Gavin Sharp [email: [e-mail]]]

If this is happening in a widespread way, it's probably due to "Clean your Mac" apps like https://cloudup.com/co3WoXjRTZq which do this to reclaim hard drive space.

Comment 6

[Robert Strong (they/them - no direct email)]

I believe we've worked around this before by serving a full update to these users or at the very least talked about just serving a full update for them.

Comment 7

[Robert Strong (they/them - no direct email)]

Here it is bug 652058

Comment 8

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> I filed bug 1071580 to try to guard against this on the platform side.
So, sending the universal build target would in some circumstances give the user a partial update when they must get the complete update (at least without somewhat complex changes on the client side) and require changes to all Mac Firefox clients which is more risky than changing the server side. It will also be trivial to track how often this happens using this method. I suggest this is entirely done on the aus side and that bug 1071580 is wontfix'd.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #8)
> (In reply to Ben Hearsum [:bhearsum] from comment #1)
> > I filed bug 1071580 to try to guard against this on the platform side.
> So, sending the universal build target would in some circumstances give the
> user a partial update when they must get the complete update (at least
> without somewhat complex changes on the client side) and require changes to
> all Mac Firefox clients which is more risky than changing the server side.
> It will also be trivial to track how often this happens using this method. I
> suggest this is entirely done on the aus side and that bug 1071580 is
> wontfix'd.

I don't think I feel as strongly as you do about only serving the completes. I definitely think we should make an effort to support these users, but I'm not particularly concerned about wasting their bandwidth. Is there a downside to serving the partial+complete other than the user ending up doing a fallback update?

Either way, I guess I'm fine WONTFIXing bug 1071580 - whatever we do here should be easy enough to continue doing going forward.

Comment 10

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #9)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #8)
> > (In reply to Ben Hearsum [:bhearsum] from comment #1)
> > > I filed bug 1071580 to try to guard against this on the platform side.
> > So, sending the universal build target would in some circumstances give the
> > user a partial update when they must get the complete update (at least
> > without somewhat complex changes on the client side) and require changes to
> > all Mac Firefox clients which is more risky than changing the server side.
> > It will also be trivial to track how often this happens using this method. I
> > suggest this is entirely done on the aus side and that bug 1071580 is
> > wontfix'd.
> 
> I don't think I feel as strongly as you do about only serving the completes.
> I definitely think we should make an effort to support these users, but I'm
> not particularly concerned about wasting their bandwidth. Is there a
> downside to serving the partial+complete other than the user ending up doing
> a fallback update?
Besides the user in some cases being notified that the partial failed not really.

> 
> Either way, I guess I'm fine WONTFIXing bug 1071580 - whatever we do here
> should be easy enough to continue doing going forward.
Exactly and an overall better path forward.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #10)
> (In reply to Ben Hearsum [:bhearsum] from comment #9)
> > (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> > comment #8)
> > > (In reply to Ben Hearsum [:bhearsum] from comment #1)
> > > > I filed bug 1071580 to try to guard against this on the platform side.
> > > So, sending the universal build target would in some circumstances give the
> > > user a partial update when they must get the complete update (at least
> > > without somewhat complex changes on the client side) and require changes to
> > > all Mac Firefox clients which is more risky than changing the server side.
> > > It will also be trivial to track how often this happens using this method. I
> > > suggest this is entirely done on the aus side and that bug 1071580 is
> > > wontfix'd.
> > 
> > I don't think I feel as strongly as you do about only serving the completes.
> > I definitely think we should make an effort to support these users, but I'm
> > not particularly concerned about wasting their bandwidth. Is there a
> > downside to serving the partial+complete other than the user ending up doing
> > a fallback update?
> Besides the user in some cases being notified that the partial failed not
> really.

In that case, I'm going to suggest that we just serve the partial+complete, and let people fall back, unless you (or someone else) feels strongly otherwise.

I know that people aren't conciously changing their build targets, but given the limited drawbacks, I don't think it's worthwhile special casing this to serve completes only.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: add single arch build targets as alias'

Both the tools that create snippets AUS2/3 and the ones that generate metadata for Balrog use this mapping to figure out build targets for any given platform. This should fix this on all channels.

It will slowdown pushsnip for the release channel a bit, but that seems like a worthwhile trade off.

And as a reminder: this will cause any users who are on a version that gets a partial to fail to apply it, and fallback to a complete.

Comment 14

[Robert Strong (they/them - no direct email)]

Comment on attachment 8493909 [details] [diff] [review]
add single arch build targets as alias'

I think that should be fine. It would be good to also get Gavin's buy in regarding user's seeing the failed for the partial and to let support know about it.

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #14)
> Comment on attachment 8493909 [details] [diff] [review]
> add single arch build targets as alias'
> 
> I think that should be fine. It would be good to also get Gavin's buy in
> regarding user's seeing the failed for the partial and to let support know
> about it.

Gavin?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8493909 [details] [diff] [review]
add single arch build targets as alias'

Would could symlink for AUS2 snippets, but can eat that pain until Balrog saves us.

Comment 17

[:Gavin Sharp [email: [e-mail]]]

(In reply to Ben Hearsum [:bhearsum] from comment #15)
> > I think that should be fine. It would be good to also get Gavin's buy in
> > regarding user's seeing the failed for the partial and to let support know
> > about it.
> 
> Gavin?

I'm not sure what the question is. Seeing failed partials and successful full updates is better than "no updates available".

Comment 18

[Robert Strong (they/them - no direct email)]

(In reply to :Gavin Sharp [email: [e-mail]] from comment #17)
> (In reply to Ben Hearsum [:bhearsum] from comment #15)
> > > I think that should be fine. It would be good to also get Gavin's buy in
> > > regarding user's seeing the failed for the partial and to let support know
> > > about it.
> > 
> > Gavin?
> 
> I'm not sure what the question is. Seeing failed partials and successful
> full updates is better than "no updates available".
With this simple fix these users will be seeing the failed update for the case where a partial is also served vs. a more complex fix where only a complete update is served so the user won't see the failure.

Comment 19

[:Gavin Sharp [email: [e-mail]]]

Right, I see now. I don't have any idea how hard the "special casing" would be, but that seems fine.

Comment 20

[Ed Morley [:emorley]]

(In reply to Lawrence Mandel [:lmandel] from comment #3)
> Great find Ben. Is there a way to tell how many instances are impacted by
> "thousands of requests per day"? If it's a significant amount, it would seem
> that our users our telling us what they want (not universal installs). It
> may be worth talking about how we can deliver that to them.

Is there a bug filed for shipping separate installers vs universal binaries? I know the topic has come up several times when talking about installer size reductions, but I can't find any related bugs.

Comment 21

[Robert Strong (they/them - no direct email)]

I believe the main thing preventing us from just shipping an Mac 64 bit build is that netflix uses silverlight and there is only an 32 bit version of Mac silverlight. Not that it wouldn't be possible to offer a single architecture dmg along with the universal 32 and 64 bit along with updates, etc. but I hope we can just go with a 64 bit dmg and no longer offer the universal dmg.

Comment 22

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #21)
> I believe the main thing preventing us from just shipping an Mac 64 bit
> build is that netflix uses silverlight and there is only an 32 bit version
> of Mac silverlight. Not that it wouldn't be possible to offer a single
> architecture dmg along with the universal 32 and 64 bit along with updates,
> etc. but I hope we can just go with a 64 bit dmg and no longer offer the
> universal dmg.

10.7 and up only support 64-bit processors (http://en.wikipedia.org/wiki/Mac_OS_X_Lion#Hardware_support), so once we de-support 10.6 and don't need it for plugins, we can stop shipping 32-bit, I think. I doubt there's a reason to bother with decoupling them in the meantime (unless we think the plugin support will take years).

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 8493909 [details] [diff] [review]
add single arch build targets as alias'

Thanks folks! This will take effect with next releases on each channel.

Comment 24

[[e-mail] (:bhearsum)]

URLs like https://aus3.mozilla.org/update/3/Firefox/32.0/20140807212602/Darwin_x86_64-gcc3/en-US/beta/Darwin%2013.1.0/default/default/update.xml are now returning updates, yay.