765441 - European mirrors don't have Firefox 13.0.1 windows files or updates

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The last successful sync by mozilla.openap.net was at Fri Jun 15 12:55:01 PDT 2012, and they're the primary distribution point to European mirrors. This is what's causing
 geodns1.vips.scl3.mozilla.com:releasesrsynclag is CRITICAL: Mirrors CRITICAL - mozilla.openap.net [EMEA primary] : 400 minutes
from nagios.

This blocks us turning down CDNetworks, which is currently handling 60% of all update traffic for FF 13.0.1.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

In terms of uptake, we have 
Product 	OS 	Available
Firefox-13.0.1	linux	134870
Firefox-13.0.1	linux64	134630
Firefox-13.0.1	osx	128050
Firefox-13.0.1	win	60365
Firefox-13.0.1-Complete	linux	114706
Firefox-13.0.1-Complete	linux64	109682
Firefox-13.0.1-Complete	osx	103264
Firefox-13.0.1-Complete	win	62521
Firefox-13.0.1-EUballot	win	60365
Firefox-13.0.1-Partial-13.0	linux	114706
Firefox-13.0.1-Partial-13.0	linux64	109682
Firefox-13.0.1-Partial-13.0	osx	103255
Firefox-13.0.1-Partial-13.0	win	62521

We'd like to turn the CDN off, but too much of the windows coverage is from it.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

mburns has looked at netstat on pv-mirror01 for connections from mozilla.openap.net (or close) and didn't see anything, so we can't kick a hung session to get it pulling the files. Eimann in #mirrors is the person we want to reach, but it's 5am in Europe so no response yet.

http://mozilla.openap.net/firefox/releases/13.0.1/update/win32/es-CL/ is as far as it got before the sync died - that dir has partial content, should be a complete mar & asc there too.

The best workaround I see is to modify releases-rsync.m.o (which seems to be in 3crowd) so that requests from Europe head to the N. American master mirror instead, which is anynymirror101.mirrors.tds.net. Not clear to mburns how to do that so dumitru paged.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The conditions for turning the CDN off, given that Europe is waking up and will be pulling traffic soon
* we have at least 50k weight from non-CDN mirrors, on all products and locations at 
   https://bounceradmin.mozilla.com/stats/uptake/?p=1772&p=1773&p=1771
  You can subtract the 35k weight of the CDN from those values
* uptake is increasing as Euro-mirrors sync from an up-to-date host (either openap fixed or tds)

The CDN is controlled at https://bounceradmin.mozilla.com/admin/mirror/mirror/368/. The normal state is to have it Active unchecked (we only use it for chemspill releases).

Remember geoip is off in bouncer right now, so we don't have to look at in-region uptake.

Comment 4

[Jake Maul [:jakem]]

Per comment 2, I have changed the 3crowd rules for releases-rsync.mozilla.org so that EMEA traffic is served from anynymirror101.mirrors.tds.net.

After about 35 minutes from that change, the weights look like this:

Product 	OS 	Available 	Total
Firefox-13.0.1 	                linux 	157008 	185061
Firefox-13.0.1 	                linux64	156850 	185061
Firefox-13.0.1 	                osx 	147272 	185061
Firefox-13.0.1 	                win 	83326 	185061
Firefox-13.0.1-Complete 	linux 	149303 	185061
Firefox-13.0.1-Complete 	linux64	133665 	185061
Firefox-13.0.1-Complete 	osx 	128656 	185061
Firefox-13.0.1-Complete 	win 	83820 	185061
Firefox-13.0.1-Partial-13.0 	linux 	149303 	185061
Firefox-13.0.1-Partial-13.0 	linux64	133665 	185061
Firefox-13.0.1-Partial-13.0 	osx 	128665 	185061
Firefox-13.0.1-Partial-13.0 	win 	83820 	185061

We are very close to being able to shut off the CDN.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We got over 100K uptake and decided to turn off the CDN (2219 Pacific). We're now at least 70K on all 13.0.1 products in bouncer.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Eimann reports that he's up to date, eg
 http://mozilla.openap.net/zz/marker.txt
so we can swap the 3crowd rules back now.

Comment 7

[Jake Maul [:jakem]]

(In reply to Nick Thomas [:nthomas] from comment #6)
> Eimann reports that he's up to date, eg
>  http://mozilla.openap.net/zz/marker.txt
> so we can swap the 3crowd rules back now.

Done.