798486 - Stub installer builds being served on http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly are using the wrong build date

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Jason Smith [:jsmith]]

Steps:

1. Install the stub installer win32 build from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central
2. Launch firefox and select about firefox

Expected:

The build date should match today's date, as this is the latest build.

Actual:

For today (10/5), the build I got was from 10/3. This isn't right.

Comment 1

[[e-mail] (:bhearsum)]

Hmmm, that build should be pulling http://download.mozilla.org/?product=firefox-nightly-latest&os=win&lang=en-US, which ends up at http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-18.0a1.en-US.win32.installer.exe

which has a buildid of BuildID=20121005030609

I'm pretty sure this is CDN caching screwing us. I don't know what we can do about it, besides not serve these files off of the CDN. CC'ing Jake and Corey for their input. Please note that this isn't a blocker, as it's not critical to this weekends tests.

Comment 2

[Jake Maul [:jakem]]

Yep, this is likely a CDN caching issue. There's no way for the CDN to know when the files should expire... the filename doesn't change like it does with release and beta builds.

Note the same problem will happen with Aurora when it goes through bouncer, and again with Beta when we start doing daily beta builds with an unchanging filename.

Lots of potential ways to go about fixing this. The best way is generally to include a query string when hitting the CDN, where the query string changes with each new version. that may not be feasible here though without code changes to bouncer and nightly.mozilla.org.


Another solution would be to have a .htaccess file that sets intelligent Cache-Control headers. For example, we can do this:

ExpiresByType <mime-type> "modification + 24 hours"

Other feasible options here: http://httpd.apache.org/docs/2.2/mod/mod_expires.html

Basically, we can have these files cached for some time interval based on the modification time.

To do this, we would need a comprehensive list of files or directories where this should occur, and what timing intervals should be used.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Jake Maul [:jakem] from comment #2)
> Other feasible options here:
> http://httpd.apache.org/docs/2.2/mod/mod_expires.html
> 
> Basically, we can have these files cached for some time interval based on
> the modification time.
> 
> To do this, we would need a comprehensive list of files or directories where
> this should occur, and what timing intervals should be used.

We could do this, roughly. Our interval is normally 24h but new builds can be triggered my hand at any time.

It seems to me that the best option here is not to serve stub installer for Nightly or Aurora from the CDN. Really, we probably shouldn't be serving any files that are regularly overwritten from it.

Unassigning myself because it's doubtful I'll be doing any work on this. Catlee might have some clever ideas on what to do when he's back tomorrow.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The CDN respects the caching headers that ftp.m.o sends, right ? If so, switching away from the CDN doesn't help us if Zeus is still caching for multiple days. I like Jake's suggestion of setting a shorter caching period for appropriate directories, although I'd advocate something like an hour instead of a day. And would be worth capturing that in puppet.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> The CDN respects the caching headers that ftp.m.o sends, right ? If so,
> switching away from the CDN doesn't help us if Zeus is still caching for
> multiple days. I like Jake's suggestion of setting a shorter caching period
> for appropriate directories, although I'd advocate something like an hour
> instead of a day. And would be worth capturing that in puppet.

This sounds great to me!

Comment 7

[Jake Maul [:jakem]]

Zeus and the CDN obey cache headers sent from the origin nodes. However, Zeus has a cap at 1800 seconds (600 if accessed over SSL). The CDNs will obey the headers outright.

Having said that, a cache header of less than a day *really* cuts into the hit rate on a CDN. If we stop using the CDN and force bouncer to serve these builds from the FTP cluster directly, then there's no point in messing with the TTLs at all... just leave them alone and let Zeus cache for up to 1800 seconds (30min).


In order to do this, I need a guaranteed way (using RewriteCond/RewriteRule, or similar) to define what constitutes a nightly or aurora product... not the dmo ?product= string, but the actual filename on FTP cluster, so I can force the CDN origin vhost(s) to 403 for that content. This needs to be specified in such a way that it will not require constant updating. How can we do that? We also need to include logic for nightly/aurora updates, if they're affected similarly (I don't know).

Also note that last I heard doing nightly "Beta" builds is also on the roadmap. When that happens, Beta will become non-CDN-eligible as well. I'm not the right person to say if this is an acceptable situation... someone will have to make that call.

Comment 8

[[e-mail] (:bhearsum)]

There's no full links I can give you that won't require updating for nightly and aurora. They will be at:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer{-stub,}.exe
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-18.0a1.en-US.win32.installer{-stub,}.exe

And the version numbers will change every 6 weeks. If you can match based on everything except the filename, that might work.

Prior to 18 making it to those channels, stub installers for them will be in https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/stub/. We can make sure to change filenames + update bouncer when the binaries change though, so I don't think that's a big deal.

Once 18 makes it to Beta the URLs will be like: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/18.0b1/win32/en-US/Firefox 18.0b1 {Stub ,}Installer.exe, and change roughly once a week. However, new files will always go in a new place, so we don't have to worry about caching.

And once it makes it to Release the URLs will be like https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/18.0/win32/en-US/Firefox 18.0 {Stub ,}Installer.exe and change every 6 weeks (plus whenever we have an out of band release). Again, we don't overwrite files here and thus don't need to worry about caching.

Long story short, if you can match on https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central and https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora that should be fine.

Comment 9

[Jason Smith [:jsmith]]

Here's an update on testing this - got new information that applies generally to what's served in ftp.mozilla.org/pub/mozilla.org/firefox/nightly:

If I try to install firefox with the stub installer for a nightly channel, I always get the latest bits for nightly. So If I install nightly with the 10/5 stub installer, I'll get today's build of 10/9, not the 10/5 build.

Is this a separate problem or the same root cause in this bug?

Comment 10

[[e-mail] (:bhearsum)]

Any stub instaler for Nightly should give you the latest nightly - this is the expected behaviour. This bug is about getting older nightlies from a newer stub installer, aiui.

Comment 12

[Jason Smith [:jsmith]]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Any stub instaler for Nightly should give you the latest nightly - this is
> the expected behaviour. This bug is about getting older nightlies from a
> newer stub installer, aiui.

Okay, makes sense. Just found that this also applies to Aurora builds as well as found out by the dup on this bug.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> There's no full links I can give you that won't require updating for nightly
> and aurora. They will be at:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> central/firefox-19.0a1.en-US.win32.installer{-stub,}.exe
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> aurora/firefox-18.0a1.en-US.win32.installer{-stub,}.exe
> 
> And the version numbers will change every 6 weeks. If you can match based on
> everything except the filename, that might work.

> Long story short, if you can match on
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> central and
> https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-
> aurora that should be fine.

Moving to Server Ops to see if we can do this.

Comment 14

[Peter Radcliffe [:pir]]

I'm going to (possibly rashly) assume that this isn't urgent enough to wake up webops folks who are all currently asleep. If that is not in fact the case let me know here or on irc (my nick is pir) and I'll find someone who can take a look at this (likely Jake).

Comment 15

[[e-mail] (:bhearsum)]

Totally not wake-up worthy. This bug is marked critical because it's a stub installer blocker.

Comment 16

[Peter Radcliffe [:pir]]

Cool. I'll make sure someone looks at it as soon as they're awake and around.

Comment 17

[Brandon Burton [:solarce]]

:jakem will need to implement this, webops is currently in SF and I'll make sure he sees this once we're all in the office this morning

Comment 18

[Bob Moss :bmoss]

Where are we at on this? It's needed for stub installer to work.

Comment 19

[Bob Moss :bmoss]

To be clear, we would ideally like to see this addressed by tomorrow Saturday 10/13. Thanks.

Comment 20

[Jake Maul [:jakem]]

I have committed a fix for this to puppet, and it should deploy within 30 minutes.

I have also issued a cache purge for this directory (firefox/nightly) to Edgecast and Akamai... however it's possible we'll want to do this again later, after the fix in puppet is deployed (cache purges take a while, so it's analogous to a race condition). But I have to leave for a plane, so I wanted to get something in before then. This is only necessary because as long as the CDN's have the content, Sentry will detect them as being valid sources for it, and send traffic there.

Comment 21

[[e-mail] (:bhearsum)]

Thanks Jake. What expiry time do you end up using?

Comment 22

[Jake Maul [:jakem]]

None. The CDN is no longer a valid target for these files. Requests should come directly to the FTP cluster, as they have in the past. Having them on the CDN in the first place was an oversight... an unintended consequence of starting to query bouncer for things it was previously never used for.

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Jake Maul [:jakem] from comment #22)
> None. The CDN is no longer a valid target for these files. Requests should
> come directly to the FTP cluster, as they have in the past. Having them on
> the CDN in the first place was an oversight... an unintended consequence of
> starting to query bouncer for things it was previously never used for.

Full installers for Nightly and Aurora *must* be served through Bouncer, as I understand it, because it needs a versionless link. The stub installer points at the following URLs to download the full installers on these channels:
http://download.mozilla.org/?product=firefox-nightly-latest&os=win&lang=${AB_CD}
http://download.mozilla.org/?product=firefox-aurora-latest&os=win&lang=${AB_CD}

Comment 24

[Brandon Burton [:solarce]]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> (In reply to Jake Maul [:jakem] from comment #22)
> > None. The CDN is no longer a valid target for these files. Requests should
> > come directly to the FTP cluster, as they have in the past. Having them on
> > the CDN in the first place was an oversight... an unintended consequence of
> > starting to query bouncer for things it was previously never used for.
> 
> Full installers for Nightly and Aurora *must* be served through Bouncer, as
> I understand it, because it needs a versionless link. The stub installer
> points at the following URLs to download the full installers on these
> channels:
> http://download.mozilla.org/?product=firefox-nightly-
> latest&os=win&lang=${AB_CD}

Just noting part of getting the above working was in bug 801017

> http://download.mozilla.org/?product=firefox-aurora-
> latest&os=win&lang=${AB_CD}

Comment 25

[Jake Maul [:jakem]]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> (In reply to Jake Maul [:jakem] from comment #22)
> > None. The CDN is no longer a valid target for these files. Requests should
> > come directly to the FTP cluster, as they have in the past. Having them on
> > the CDN in the first place was an oversight... an unintended consequence of
> > starting to query bouncer for things it was previously never used for.
> 
> Full installers for Nightly and Aurora *must* be served through Bouncer, as
> I understand it, because it needs a versionless link. The stub installer
> points at the following URLs to download the full installers on these
> channels:
> http://download.mozilla.org/?product=firefox-nightly-
> latest&os=win&lang=${AB_CD}
> http://download.mozilla.org/?product=firefox-aurora-
> latest&os=win&lang=${AB_CD}

We have a misunderstanding here... I see now that my word choice probably contributed. Sorry about that.

When I said "directly to the FTP cluster", what I meant was "the CDNs will not be a valid mirror for those products, and bouncer will direct the download to the FTP cluster... as the FTP cluster *is* a mirror, which will have the files."

These will be in bouncer, and they will work... they just won't come from a CDN mirror. Effectively it will be just like it always has - nightly/aurora served from FTP - but with the extra step of bouncer in between.


Note that I'm not sure if webdev is going to be using the actual "firefox-*-stub" products for serving the stub installer in time for this launch... they may link directly to it for now. But that's up to them, not me.

Comment 26

[[e-mail] (:bhearsum)]

(In reply to Jake Maul [:jakem] from comment #25)
> (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > (In reply to Jake Maul [:jakem] from comment #22)
> > > None. The CDN is no longer a valid target for these files. Requests should
> > > come directly to the FTP cluster, as they have in the past. Having them on
> > > the CDN in the first place was an oversight... an unintended consequence of
> > > starting to query bouncer for things it was previously never used for.
> > 
> > Full installers for Nightly and Aurora *must* be served through Bouncer, as
> > I understand it, because it needs a versionless link. The stub installer
> > points at the following URLs to download the full installers on these
> > channels:
> > http://download.mozilla.org/?product=firefox-nightly-
> > latest&os=win&lang=${AB_CD}
> > http://download.mozilla.org/?product=firefox-aurora-
> > latest&os=win&lang=${AB_CD}
> 
> We have a misunderstanding here... I see now that my word choice probably
> contributed. Sorry about that.
> 
> When I said "directly to the FTP cluster", what I meant was "the CDNs will
> not be a valid mirror for those products, and bouncer will direct the
> download to the FTP cluster... as the FTP cluster *is* a mirror, which will
> have the files."
> 
> These will be in bouncer, and they will work... they just won't come from a
> CDN mirror. Effectively it will be just like it always has - nightly/aurora
> served from FTP - but with the extra step of bouncer in between.
> 
> 
> Note that I'm not sure if webdev is going to be using the actual
> "firefox-*-stub" products for serving the stub installer in time for this
> launch... they may link directly to it for now. But that's up to them, not
> me.

OK, as long as these are being served directly from FTP only this should be fine.

Thanks!

Comment 27

[Jake Maul [:jakem]]

This was fixed last week.