333123 - Some FTP mirrors in releases.mozilla.org RRDNS not updating

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

User comments have come in about the FTP mirrors falling behind (by 1/2 days):
05:49 < Cameron> the news from the night is that some of the mirrors are VERY behind (like.. over a day... maybe 2 or something) I dunno what you want to do
                 about it, but yeah

I looked at this, and indeed some servers are not being updated with the new versions of extensions while some are.

Could someone look into this?

Comment 1

[Dave Miller [:justdave]]

Can you find me one?  Or a file that hasn't updated?  I just converted the test script we use on ftp.m.o to run on releases.m.o instead and it says the furthest one behind is 20 minutes.

Comment 2

[Michael Morgan [:morgamic]]

http://ftp.mozilla.org/pub/mozilla.org/extensions/wizz_rss_news_reader/

vs.

http://releases.mozilla.org/pub/mozilla.org/extensions/wizz_rss_news_reader/

The last two versions are not updated -- which makes it impossible to download the newest version.

Comment 3

[Dave Miller [:justdave]]

status update: most of the world rsyncs from TDS.  TDS has two boxes, and they are out-of-sync with each other.  I've paged the admin at TDS, he replied to my page saying he'd be on it in about 30 minutes.

Comment 4

[Dave Miller [:justdave]]

TDS mirror1 is out of disk space, TDS mirror2 is okay.  ftp-rsync and releases-rsync have been moved to point at mirror2.  Hopefully everyone should catch up in the next hour or so.

Comment 5

[Dave Miller [:justdave]]

Anyone having any problems related to this still?  

Comment 6

[Dave Miller [:justdave]]

mirror2.tds is fine now, but mirror1.tds is now having problems, mirror1 has been dropped from the rsync RRDNS.

Comment 7

[Cameron]

*** Bug 334420 has been marked as a duplicate of this bug. ***

Comment 8

[Rob Marshall [tH]]

*** Bug 334476 has been marked as a duplicate of this bug. ***

Comment 9

[Rob Marshall [tH]]

I think the DNS is the wrong way round. I'm seeing releases-rsync.mozilla.org pointing to 216.165.129.134 here, which is mirror1.mirrors.tds.net. Manually browsing mirror2 shows it's up to date.

Comment 10

[Christopher Finke]

I've noticed this problem with my extension (RSS Ticker).  It was updated to 1.4 last night (April 17), and 12 hours later, the "not a valid install package" errors are still coming (from releases.mozilla.org), while ftp.mozilla.org has the file.

Comment 11

[Dave Miller [:justdave]]

Based on comments here, I just swapped them back again, so mirror1 is live, and mirror2 is out for now.  I've pinged the TDS admin again.

Comment 12

[Rob Marshall [tH]]

It's looking a lot better now. Of 12 r.m.o mirrors I checked a few minutes ago, only 1 was behind, but I've just checked now and it's up-to-date.

Comment 13

[Dave Miller [:justdave]]

OK, this is resolved.  The latest issues were the OSL mirror running out of disk space.  We've changed the way we monitor the mirrors so we can tell if their last sync completed and not just if they started it, so we should be able to stay on top of pulling people out of the round robin when they have rsyncs that don't complete now.