595515 - Shutdown Squid Proxy after all minis have left Castro

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Axel Hecht [:Pike]]

Can't figure out the exact data here, but I don't see a bump in http://build.mozilla.org/builds/pending/, and the latest nightlies seem to be old. At least for mac de central.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/ looks pretty up to date to me. All but 3 locales on windows are dated 2010-09-11.

Comment 2

[Axel Hecht [:Pike]]

I'm not getting updates on https://aus2.mozilla.org/update/3/Firefox/4.0b6pre/20100910030657/Darwin_x86-gcc3-u-ppc-i386/de/nightly/Darwin%209.8.0/default/default/update.xml?force=1, though. Some locales seem to be fine, others are not.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I found the 2010-09-11 nightly build for de [1], and it has both the buildid and previous_buildid set to 20100910030657. It pulled down
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-4.0b6pre.en-US.mac.dmg
to create the repack, and got a 30525597 byte file. The only en-US nightly with that size in September is the one in 2010-09-10-03-mozilla-central/, which has the buildid that matches the one above. The de nightly started 50 minutes after the en-US one finished, so somehow we got a stale file from ftp.m.o.

Getting the wrong en-US leads to a bogus repack, and things like:
 partialMarFilename
    firefox-4.0b6pre.de.mac.partial.20100910030657-20100910030657.mar
 partialMarSize    7687
and not publishing the snippets properly (in particular the empty snippets that should have been created for 20100911HHMMSS).

Approximately 15 locales were affected in the same way, several of which were built on moz2-darwin9-slave29. That slave is in Castro and so it has the squid proxy between it and ftp.m.o. Looking into that.


[1] http://production-master03.build.mozilla.org:8010/builders/Firefox%20mozilla-central%20macosx%20l10n%20nightly/builds/3412

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: squid logs

These are munged logs with the timestamps converted to PDT.

Key:
* 10.250.48.206 is moz2-darwin9-slave29
* 10.250.48.211 is moz2-darwin9-slave34

Timeline:
* slave29 pulls the dmg at 3:17 on the 11th for a dep build, populating the cache
* slave29 pulls the dmg at 3:56 for another dep build, cache hit, which is fine
* en-US nightly for the 11th is uploaded at 4:49, rest of job finishes at 4:52
* slave34 starts l10n nightly and pulls the dmg at 4:52:12, gets a cache HIT but  should have been a miss
* lots more l10n nightly builds on slave29 and slave34 that get the wrong file
* a l10n dep build at 8:40 makes the cache realize it has a stale file (I think)

jabba, what's going on here with the cache expiry ? Could do with more log history too please.

Comment 5

[matthew zeier [:mrz]]

How much effort needs to be spent on this?  We'll be moving these hosts out of MV real soon.

Comment 6

[Justin Dow [:jabba]]

I'm declaring that squid is causing more problems than it solves. Originally it was set up due to our extremely limited bandwidth to Castro. Since then we have a lot more bandwidth to Castro. Aside from that, all minis are leaving the building, so this will no longer be required. We should just disable it after all the minis have left and in the future, look at alternatives if caching is needed again.

Comment 7

[Justin Dow [:jabba]]

Checklist for successful completion of this bug:
1) All minis out of Castro
2) All remaining Releng Machines need to not be using this server as their DNS server. This will likely require manual intervention as some might have it statically set.
3) Firewall configs updated to remove the WCCP protocol
4) Machine shut down and all nagios checks removed, removed from dns/dhcp/inventory.

Comment 8

[matthew zeier [:mrz]]

Closing.