271473 - (decouple) decouple services on update.mozilla.org

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

The app update service (AUS), plugin finder service (PFS), and extension/theme
directory (ETD) all live on the update.mozilla.org domain name.  They should
each live on their own domain names so we can allocate hardware resources to
them independently and isolate them from each other.

We need to do this in the most minimally disruptive way to existing Firefox
clients while still meeting the goal of putting each service on its own domain
name and making update.mozilla.org only redirect to each service.

AUS and PFS can both handle arbitrary HTTP redirects without breaking. 
Extension/theme installation whitelists update.mozilla.org explicitly, but the
whitelisting may be subdomain-recursive, in which case we could relocate the
site to foo.update.mozilla.org without breaking the whitelisting.

foo.update.mozilla.org is clumsy for long-term use, however, so we should pick a
better long-term name, whitelist both, redirect users to foo.update.mozilla.org
in the short-term, and then switch to the long-term name once we're satisfied
that the vast majority of users have upgraded to versions that whitelist the
long-term name.

We should get this into not only the next major version of Firefox but also the
next maintenance release, if any, to propagate this change as quickly as possible.

Possible names for these services:

AUS: aus.mozilla.org
PFS: pfs.mozilla.org
ETD: addons|extras.(update.)mozilla.org

Note that AUS and PFS domain names should be invisible to the end user, so it's
not critical for their names to make sense to someone unfamiliar with the
services.  ETD's domain name, on the other hand, should make sense to regular
end users.

Comment 1

[Wolf [:wolf]]

Perhaps something simple. mozillaupdate.mozilla.org. or mozillaupdate.org.
I'd discourage breaking the site's current name (by picking a DNS host that just
mismatches entirely), as most of the new Firefox 1.0 converts have been
introduced to it by now as "Mozilla Update" (Beta).

Comment 2

[Myk Melez [:myk] [@mykmelez]]

>Perhaps something simple. mozillaupdate.mozilla.org. or mozillaupdate.org.
>I'd discourage breaking the site's current name (by picking a DNS host that just
>mismatches entirely), as most of the new Firefox 1.0 converts have been
>introduced to it by now as "Mozilla Update" (Beta).

"update" doesn't match the function of the site very well (it works better for
the app update service, which umo was originally coupled with), and since we're
still in beta, now is a reasonable time to change it to something that
represents the content the directory is serving.  Note that users will still get
redirected from the old URL to the new one, so nothing will break with a new URL.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

aus.mozilla.org, pfs.mozilla.org, and alternatives addons(.update).mozilla.org
and extras(.update).mozilla.org have all been added to DNS, and iguana has been
configured to serve them.  The server-side work on this is thus mostly done,
except that we need to update iguana's certificate so that it covers these new
domains.  In the meantime, moving this bug to a client Bugzilla component so I
can request review on the patch I'm about to attach.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v1: changes update.mozila.org to aus|pfs|addons.mozilla.org

This patch changes references to update.mozilla.org in Aviary applications to
aus.mozilla.org for the application finder service, pfs.mozilla.org for the
plugin finder service, and addons.mozilla.org for the extension/theme
directory.

It also whitelists installation of add-ons from addons.update.mozilla.org,
which is where we'll be redirecting older clients hitting update.mozilla.org so
that we don't break their ability to install add-ons.

I've also registered extras.mozilla.org as an alternative to
addons.mozilla.org.

Comment 5

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v2: correct xpinstall.whitelist.add usage

This version corrects my usage of xpinstall.whitelist.add and removes
addons.update.mozilla.org from the list, since that domain is covered by the
update.mozilla.org whitelisting.

Ben says r=ben, but I'll wait to check this in until the server certificate on
iguana is updated to take these new domains into account.

Comment 6

[Wolf [:wolf]]

I think I prefer extras.mozilla.org to addons.. Seems to have a better sound to
it, and the two terms basically mean the same thing with extras likely to make
better since for themes. (addons might get confused with the plugin/extension
confusion that already exists.)

Comment 7

[Myk Melez [:myk] [@mykmelez]]

> I think I prefer extras.mozilla.org to addons.

I asked Ben which one he wanted, and he said addons, since it's his collective
term for the extensions and themes the directory makes available.

> (addons might get confused with the plugin/extension
> confusion that already exists.)

Plugins are likely to eventually make their way into the directory, in some form
or another, and they're also add-ons, so this shouldn't present a problem.

Comment 8

[Wolf [:wolf]]

My thought with picking extras over addons was more site naming (or renaming in
this case) than the actual DNS. Since it's unlikely that calling the site
"Mozilla Update" will make much sense in the near future, I'm thinking if
renamed, which Comment #2 suggested that should happen, that "Mozilla Extras"
sounds better than "Mozilla Addons". In theory the name and DNS don't have to be
linked, but for end-users, it makes more sense to.

Interestingly enough, update-beta has this line on the front page, written by
Daniel Burka, "Mozilla Update is the place to get extras for your Mozilla
products." which suggests to me that the name change from Mozilla Update to
Mozilla Extras would make the most sense.

Comment 9

[Myk Melez [:myk] [@mykmelez]]

>My thought with picking extras over addons was more site naming (or renaming in
>this case) than the actual DNS.

Sure, that makes sense, and it does make sense for DNS and the site to match. 
Either name works for me, however, both in DNS and on the site, and it sounds
like you feel the same, even if you prefer extras over addons, so let's go with
what Ben, as reviewer, wants us to do here.

Ok, certificate installed, server config polished, and patch checked in to trunk.

Checking in browser/app/profile/firefox.js;
/cvsroot/mozilla/browser/app/profile/firefox.js,v  <--  firefox.js
new revision: 1.35; previous revision: 1.34
done
Checking in browser/components/help/locale/en-US/customization.xhtml;
/cvsroot/mozilla/browser/components/help/locale/en-US/customization.xhtml,v  <--
 customization.xhtml
new revision: 1.11; previous revision: 1.10
done
Checking in browser/locales/en-US/profile/bookmarks.html;
/cvsroot/mozilla/browser/locales/en-US/profile/bookmarks.html,v  <--  bookmarks.html
new revision: 1.3; previous revision: 1.2
done
Checking in mail/app/profile/all-thunderbird.js;
/cvsroot/mozilla/mail/app/profile/all-thunderbird.js,v  <--  all-thunderbird.js
new revision: 1.13; previous revision: 1.12
done
Checking in toolkit/locales/en-US/chrome/mozapps/extensions/extensions.dtd;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/extensions/extensions.dtd,v
 <--  extensions.dtd
new revision: 1.3; previous revision: 1.2
done
Checking in toolkit/locales/en-US/chrome/mozapps/extensions/extensions.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/extensions/extensions.properties,v
 <--  extensions.properties
new revision: 1.3; previous revision: 1.2
done
Checking in toolkit/locales/en-US/chrome/mozapps/plugins/plugins.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/plugins/plugins.properties,v
 <--  plugins.properties
new revision: 1.3; previous revision: 1.2
done
Checking in toolkit/locales/en-US/chrome/mozapps/update/update.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/update/update.properties,v
 <--  update.properties
new revision: 1.3; previous revision: 1.2
done
Checking in toolkit/mozapps/plugins/content/pluginInstallerWizard.js;
/cvsroot/mozilla/toolkit/mozapps/plugins/content/pluginInstallerWizard.js,v  <--
 pluginInstallerWizard.js
new revision: 1.4; previous revision: 1.3
done
Checking in toolkit/mozapps/plugins/content/pluginInstallerWizard.xul;
/cvsroot/mozilla/toolkit/mozapps/plugins/content/pluginInstallerWizard.xul,v 
<--  pluginInstallerWizard.xul
new revision: 1.3; previous revision: 1.2
done
Checking in browser/components/help/locale/en-US/customization.xhtml;
/cvsroot/mozilla/browser/components/help/locale/en-US/customization.xhtml,v  <--
 customization.xhtml
new revision: 1.12; previous revision: 1.11
done

Comment 10

[Myk Melez [:myk] [@mykmelez]]

Hmm, there are two more server-side changes to do for this:

Redirect permanent /update/firefox https://aus.mozilla.org/update/firefox
Redirect permanent /plugins https://pfs.mozilla.org/plugins

(in the vhost config for update.mozilla.org)

Comment 11

[Wolf [:wolf]]

All of /update/ can redirect to aus. Theres nothing in there that is not part of
the Application Update service. Just like /update.rdf is part of aus for Firefox
0.9.x and not addons[.update].mozilla.org.

Comment 12

[Steffen Wilberg]

I updated the links on www.mozilla.org, most of them to addons.mozilla.org. I
also changed references to "update.mozilla.org" to "addons.mozilla.org". I
didn't change the term "Mozilla Update".

Comment 13

[Myk Melez [:myk] [@mykmelez]]

>All of /update/ can redirect to aus. Theres nothing in there that is not part of
>the Application Update service. Just like /update.rdf is part of aus for Firefox
>0.9.x and not addons[.update].mozilla.org.

The rest of /update uses the MySQL database, so it's more difficult to decouple
from the directory, but we should do so at some point.  I'm not sure it should
go on AUS, however, since it's PHP-based, while AUS is static files, and it'd be
good to keep it that way to make hosting it redundantly easier.

Ok, Apache config updated.

Comment 14

[alanjstr]

Just a note that we want this for FX 1.0.1, too.

Comment 15

[Dave Miller [:justdave]]

fwiw, the redirects were all undone last night to help alleviate bandwidth
congestion during the DDoS.  Everything is pulling directly from
update.mozilla.org again (or addons.mozilla.org, or whichever domain they choose
to hit).

Comment 16

[Asa Dotzler [:asa]]

Comment on attachment 168738 [details] [diff] [review]
patch v2: correct xpinstall.whitelist.add usage

a=asa for 1.0.1

Comment 17

[Myk Melez [:myk] [@mykmelez]]

Committing to branch...

Checking in browser/app/profile/firefox.js;
/cvsroot/mozilla/browser/app/profile/firefox.js,v  <--  firefox.js
new revision: 1.7.4.40.2.1; previous revision: 1.7.4.40
done
Checking in browser/components/help/locale/en-US/customization.xhtml;
/cvsroot/mozilla/browser/components/help/locale/en-US/customization.xhtml,v  <--
 customization.xhtml
new revision: 1.1.4.1.2.16.2.1; previous revision: 1.1.4.1.2.16
done
Checking in browser/locales/en-US/profile/bookmarks.html;
/cvsroot/mozilla/browser/locales/en-US/profile/bookmarks.html,v  <--  bookmarks.html
new revision: 1.1.2.8.2.1; previous revision: 1.1.2.8
done
Checking in mail/app/profile/all-thunderbird.js;
/cvsroot/mozilla/mail/app/profile/all-thunderbird.js,v  <--  all-thunderbird.js
new revision: 1.1.2.11.2.1; previous revision: 1.1.2.11
done
Checking in toolkit/locales/en-US/chrome/mozapps/extensions/extensions.dtd;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/extensions/extensions.dtd,v
 <--  extensions.dtd
new revision: 1.1.2.5.2.1; previous revision: 1.1.2.5
done
Checking in toolkit/locales/en-US/chrome/mozapps/extensions/extensions.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/extensions/extensions.properties,v
 <--  extensions.properties
new revision: 1.1.2.4.2.1; previous revision: 1.1.2.4
done
Checking in toolkit/locales/en-US/chrome/mozapps/plugins/plugins.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/plugins/plugins.properties,v
 <--  plugins.properties
new revision: 1.1.2.7.2.1; previous revision: 1.1.2.7
done
Checking in toolkit/locales/en-US/chrome/mozapps/update/update.properties;
/cvsroot/mozilla/toolkit/locales/en-US/chrome/mozapps/update/update.properties,v
 <--  update.properties
new revision: 1.1.2.6.2.1; previous revision: 1.1.2.6
done
Checking in toolkit/mozapps/plugins/content/pluginInstallerWizard.js;
/cvsroot/mozilla/toolkit/mozapps/plugins/content/pluginInstallerWizard.js,v  <--
 pluginInstallerWizard.js
new revision: 1.1.2.11.2.1; previous revision: 1.1.2.11
done
Checking in toolkit/mozapps/plugins/content/pluginInstallerWizard.xul;
/cvsroot/mozilla/toolkit/mozapps/plugins/content/pluginInstallerWizard.xul,v 
<--  pluginInstallerWizard.xul
new revision: 1.1.2.11.2.1; previous revision: 1.1.2.11
done

Comment 18

[Steffen Wilberg]

So we've changed a bunch of files in /locales/en-US/. I guess we want to do the
same to the rest of our 27 official locales, or at least some of them.

Comment 19

[Axel Hecht]

(In reply to comment #18)
> So we've changed a bunch of files in /locales/en-US/. I guess we want to do the
> same to the rest of our 27 official locales, or at least some of them.

We cannot change the locales without owner-approval. Please file bugs on the 
locales so that we can do this for 1.1. Or at least file a meta bug in Mozilla
Localisations / Others and assign it to mlp-staff, CC me and gandalf.

The scheme for 1.0.1 doesn't really allow this change. Unless you have singled out
builds that you want to change. But I don't see us doing that for 35 builds.
Asa?

Comment 20

[Chase Phillips]

(In reply to comment #19)
> We cannot change the locales without owner-approval. Please file bugs on the 
> locales so that we can do this for 1.1. Or at least file a meta bug in Mozilla
> Localisations / Others and assign it to mlp-staff, CC me and gandalf.

I realize this comment is almost two months old, but this is the first time I'm
reading it.  I disagree.  This is a service-level change that is being made to
the application and there's no realistic way that we can walk everyone through
this type of alteration and have everyone get it right on their own without it
being a time sink.

Instead, we should move to a model where mass-changes to locales should not
require locale owner approval.  This allowed one of the copyright fixes that
made it into 1.0.2 to also make it into each of our locales, and to do so in a
relatively short amount of time.

Comment 21

[Dave Miller [:justdave]]

We completed the planned decoupling this last week by removing the redirect to
addons.update.mozilla.org from addons.mozilla.org if the user is not using
Firefox 1.0 (Firefox 1.0 still gets addons.update.mozilla.org).  However, we
subsequently got several complaints from people that addons.mozilla.org was not
in their whitelist and extension installs were being blocked.  There is some
conjecture that this is because the whitelist change only affected new installs,
and didn't affect people who upgraded from 1.0, but I have yet to see any proof
from anyone.

Comment 22

[Dave Miller [:justdave]]

my own copy of FF 1.0.2 does not have addons.mozilla.org whitelisted.

Comment 23

[alanjstr]

(In reply to comment #22)
> my own copy of FF 1.0.2 does not have addons.mozilla.org whitelisted.

1) Where would I go to see the whitelist
2) What was your update method

I was unable to install FlashGot using Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.7.6) Gecko/20050328 Firefox/1.0.2 (MOOX M3)

Comment 24

[Myk Melez [:myk] [@mykmelez]]

I installed Firefox 1.0, created a new profile, then installed Firefox 1.0.1 and
started it with the profile I had created in Firefox 1.0.  My profile did not
whitelist addons.mozilla.org.  I then created a profile in Firefox 1.0.1.  It
did whitelist addons.mozilla.org.

So Dave's conjecture is correct: Firefox does not update the whitelist when you
upgrade to Firefox 1.0.1 but rather continues to use an existing profile.

It looks like the code in nsInstallTrigger populates the whitelist (which lives
in the hostperm.1 file in the profile dir) from the xpinstall.whitelist.add
preference, but then it sets that preference to the empty string in the
profile's prefs.js file.  Since profile preferences take precedence over app
preferences, the empty profile preference overrides the updated 1.0.1 app
preference, and the whitelist never gets updated with the new value of the
preference.

http://lxr.mozilla.org/mozilla/source/xpinstall/src/nsInstallTrigger.cpp#349

I don't know how to fix this problem.  Perhaps there's a way to force a
preference to get reset to the application default upon upgrade?

Comment 25

[Myk Melez [:myk] [@mykmelez]]

Ok, according to bsmedberg I have to include addons.mozilla.org in a new
uniquely-named preference.  Working on a patch now.

Comment 26

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v3: whitelists addons.mozilla.org for users with existing profiles too

This patch adds a new preference (xpinstall.whitelist.add.103) containing
addons.mozilla.org, patches nsInstallTrigger.cpp to whitelist entries in both
the original preference (xpinstall.whitelist.add) and the new one, and removes
addons.mozilla.org from the original preference (since that would be redundant
with the new one).

Comment 27

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v4: updates all-thunderbird.js too

The preferences in all-thunderbird.js also need updating.

Comment 28

[Chase Phillips]

Won't block 1.0.3.  Nominating for 1.0.4.

Comment 29

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 179811 [details] [diff] [review]
patch v4: updates all-thunderbird.js too

verbal a=chofmann for check-in to 1.0 branch

Comment 30

[Dave Miller [:justdave]]

OK, here's the rest of the story here, since I gather there's still some confusion.

We have an SSL cert for *.mozilla.org.  The X.509 spec says that a * in a
certificate can only replace one chunk of the domain name.  This means that the
*.mozilla.org certificate is technically invalid for the
addons.update.mozilla.org domain because there are too many parts in the domain
to match the certificate. (see RFC 2818 section 3.1)

Firefox itself seems to accept the certificate (in violation of the spec), but
nothing else does.  Safari and most RSS newsreaders (what I have to test with)
all choke on the certificate on addons.update.mozilla.org complaining of a
domain name mismatch.  All of these useragents handle addons.mozilla.org just fine.

While we could in theory get a second certificate for addons.update.mozilla.org
that specifically spelled out the entire thing, the SSL negotiation happens
before the hostname is sent, so there is no way to choose a certificate based on
the requested hostname.  Since it's sitting behind a squid proxy, we also don't
have the capability to pick which domain they get based on the UserAgent string
(which would also solve this problem). We initially got around this because FF
1.0 is hitting the update.mozilla.org domain, and that domain redirects to
addons.update.mozilla.org, but if you go to addons.mozilla.org directly (which
1.0.1 and up do) you stay there.

Comment 31

[Myk Melez [:myk] [@mykmelez]]

Second fix checked in to trunk and branch.  Resolving fixed again.  Note that
the second fix, unlike the first, does not affect l10n, so the "[affects l10n]"
message in the status whiteboard does not apply to this subsequent change.

The checkin messages for the branch got lost off the end of my terminal buffer,
but here are the messages for the trunk:

Checking in browser/app/profile/firefox.js;
/cvsroot/mozilla/browser/app/profile/firefox.js,v  <--  firefox.js
new revision: 1.41; previous revision: 1.40
done
Checking in mail/app/profile/all-thunderbird.js;
/cvsroot/mozilla/mail/app/profile/all-thunderbird.js,v  <--  all-thunderbird.js
new revision: 1.26; previous revision: 1.25
done
Checking in xpinstall/public/nsISoftwareUpdate.h;
/cvsroot/mozilla/xpinstall/public/nsISoftwareUpdate.h,v  <--  nsISoftwareUpdate.h
new revision: 1.38; previous revision: 1.37
done
Checking in xpinstall/src/nsInstallTrigger.cpp;
/cvsroot/mozilla/xpinstall/src/nsInstallTrigger.cpp,v  <--  nsInstallTrigger.cpp
new revision: 1.74; previous revision: 1.73
done