1096367 - move aurora updates onto cdn temporarily

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

FTP is dying under load because of Aurora updates. We need to get them onto the CDN for now. Whatever we do in the short term will get reset when the next nightlies are built. We probably don't need this in the longer term. I suggest we do the quickest fix for now, and see how things look when tomorrow's nightlies reset things -- traffic should've calmed down by then.

Comment 1

[Rail Aliiev [:rail]]

Firefox-Developer-Edition-2014-11-10-00-40-02-complete bouncer product created

Comment 2

[[e-mail] (:bhearsum)]

I created a new release in Balrog called "Firefox-Aurora-CDN-Workaround-20141110". It's the same as the as the latest Aurora blob except that the en-US complete MARs point at bouncer links such as http://download.mozilla.org/?product=Firefox-Developer-Edition-2014-11-10-00-40-02-complete&os=linux&lang=en-US.

We've switched the aurora rules to point at this blob. Note that l10n is not included here. I suspect between switching en-US updates and en-US+l10n installers over the CDN that this will be enough. It will take longer to switch the l10n updates.

When the latest aurora nightlies fire, they will NOT update this blob, which means the new updates will not automatically publish. I suggest that we switch the aurora rule back to the "latest" blob tomorrow morning, after things have calmed down. This will have the implication of moving the updates back to FTP, so we don't want to do this if load is still high tomorrow.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Today we faced lots of broken Mozmill tests for the Firefox 34.0b8 candidate builds, because downloads are taking soo long from the FTP server. We have a transfer speed of about 11kb/s. So I'm happy of this even temporarily move for now to CDN. I wonder about upcoming updates for Aurora DevEdition, and how our user base has been increased. It means we could have the same load each and every day on our FTP servers. All this drastically kills our ability to test candidate builds in a timely manner. So a discussion about a permanent fix would be very appreciated! Thank's.

Comment 4

[Pete Moore [:pmoore][:pete]]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> I created a new release in Balrog called
> "Firefox-Aurora-CDN-Workaround-20141110". It's the same as the as the latest
> Aurora blob except that the en-US complete MARs point at bouncer links such
> as
> http://download.mozilla.org/?product=Firefox-Developer-Edition-2014-11-10-00-
> 40-02-complete&os=linux&lang=en-US.
> 
> We've switched the aurora rules to point at this blob. Note that l10n is not
> included here. I suspect between switching en-US updates and en-US+l10n
> installers over the CDN that this will be enough. It will take longer to
> switch the l10n updates.
> 
> When the latest aurora nightlies fire, they will NOT update this blob, which
> means the new updates will not automatically publish. I suggest that we
> switch the aurora rule back to the "latest" blob tomorrow morning, after
> things have calmed down. This will have the implication of moving the
> updates back to FTP, so we don't want to do this if load is still high
> tomorrow.

Based on this advice ("we don't want to do this if load is still high tomorrow"), and whimboo's observation that ftp load is still high (in comment 3), I will not switch the rule back to latest at this moment. Let's discuss in a few hours when the East Coast comes online, about how to proceed.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sorry for that wrong observation. The tests I was looking at were indeed from yesterday and not today. So all the beta candidate tests we are running now are working perfectly fine. No slow serving of builds from the FTP server today.

Comment 6

[Pete Moore [:pmoore][:pete]]

Please also note network is heavily burdened due to bug 1096337, which wouldn't help ftp traffic.

Comment 7

[[e-mail] (:bhearsum)]

Any slowness with Beta updates is completely unrelated to this - they're already on the CDN. If you want to discuss that further, let's take it elsewhere - this bug is pretty tightly scoped.

Comment 8

[[e-mail] (:bhearsum)]

I've moved updates back to FTP now that load has calmed down. I'll keep an eye on FTP load and move them to the CDN again if things slow down. I think we'll be fine now that we're 24 hours out from launch.