457614 - Mirrors get added by sentry before all of the files are available because we check for a file that is early in the rsync

Status: RESOLVED FIXED

(Webtools :: Bouncer, defect)

Description

[Dave Miller [:justdave]]

We discovered during this last release that several mirrors were getting added by sentry before they completed rsyncing all of the files.  The problem seems to be that sentry is only checking for the en-US version and then assumes everything is there if that one is.  During this last release, several of the mirrors took quite a while to rsync all of the files in, and sentry apparently made a pass at them just after the en-US files landed, but before much else had.  Since the file transfers tend to happen in alphabetical order, and en-US is actually pretty near the beginning of the list of locales, you can probably see where this is going.

As a stopgap measure until bouncer/sentry are actually tracking the individual locales, we should probably check for the zh-TW files rather than en-US, since that would be the last locale to transfer.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

zh-TW sounds like a good idea, but I don't think we can do that based on how bouncer serves d/l requests:

  // LANGUAGE HACK
  if (!empty($_GET['lang'])) {
     $location['location_path'] = 
          str_replace('en-US',$_GET['lang'],$location['location_path']);
  }

Assuming this is the right place to be looking:
http://viewvc.svn.mozilla.org/vc/projects/bouncer/1.0/branches/production/php/index.php?revision=12259&view=markup

Perhaps it could do the replace on something more generic, eg either en-US or LOCALE ? Looks like str_replace supports passing an array there.

Comment 2

[Dave Miller [:justdave]]

We were having problems with this again this morning... I just locally hacked sentry to s/en-US/zh-TW/ on the URL before checking it.

Comment 4

[matthew zeier [:mrz]]

Sounds like we need a pseudo locale that exists for all products.  We keep hitting this for various products (Camino the other day, Thunderbird now).

Comment 5

[Dave Miller [:justdave]]

The hack version:
Committed revision 25871.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

IIRC, sentry is replacing :lang with zh-TW when polling for files. There is a locale after that (zu), but the time window for enabling the d/l and the last being available is only seconds. Moving to CDN has also made this much less of an issue (no long sync times).