703729 - Remove pub/mozilla.org/firefox/releases/8.0.1 on pv-mirror01 and pv-mirror02

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Several releases got deleted (FF 8.0.1) or added to the exclude for prereleases (FF 8.0b1-6 and some SeaMonkey and Thunderbird) in the last day or so. Until we get bug 700798 those removals happen very slowly (10 files and/or dirs every 10 mins).

Could you please manually delete the 8.0.1 directory from firefox/releases on pv-mirror01.m.o and pv-mirror02. We're respinning that release and I'd like to make sure we have these two machines, and the CDN which uses pv-mirror01, reset to a clean state.

Comment 1

[Dave Miller [:justdave]]

Done.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Do we have a contact at cdnetworks who could dump all of firefox/releases/8.0.1/ from their systems ? They seem to have a 7 day timeout, eg

http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/update/linux-i686/af/firefox-8.0.1.complete.mar
HTTP/1.1 200 OK
Date: Mon, 21 Nov 2011 10:49:52 GMT
Server: PWS/1.7.3.7
X-Px: ms h0-s1011.p10-sjc ( h0-s1023.p10-sjc), ht-d h0-s1023.p10-sjc.cdngp.net
ETag: "2be68501-f2bdb9-4b1c4e3d5f940"
Cache-Control: max-age=604800
Expires: Wed, 23 Nov 2011 07:30:20 GMT
Age: 443972
Accept-Ranges: bytes
Content-Length: 15908281
Content-Type: application/octet-stream
Last-Modified: Tue, 15 Nov 2011 12:11:41 GMT
Connection: keep-alive

We could work around by using 8.0.1-real but ewwww.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

FYI, there's no firm ship time yet but probably going to be Pacific late morning or so.

Comment 4

[[e-mail] (:bhearsum)]

Raising to critical because we could be asked to push really soon now.

Comment 5

[Dave Miller [:justdave]]

OK, we do have a cache purge utility in the control panel for this CDN.

http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/ ===> REQUEST ACCEPTED

Says it'll take up to 15 minutes to purge.

Comment 6

[Dave Miller [:justdave]]

Appears to have completed. :)

Comment 7

[Rail Aliiev [:rail]]

I just checked it:

# stage
cd /pub/mozilla.org/firefox/nightly/8.0.1-candidates/build2/
find linux-* mac/ win32* update/ -type f | grep -vE 'asc|checksums|xpi|zip|txt|log' | sort > /tmp/cdnetworks-list.log
(while read line; do
  url="http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/$line"
  echo "$url"
  curl -sI "$url"
  echo
done < "/tmp/cdnetworks-list.log") | tee /tmp/cdnetworks-check2.log

grep -c "HTTP/1.1 200" /tmp/cdnetworks-check2.log
0

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

After this we pushed build2 to the mirrors, and re-enabled the CDN. It started serving build1 files in some cases so we had to turn it off again. 

The precise timing of enabling the CDN in bouncer, the rsync to pv-mirror01, and the temporary unavailability of pv-mirror01 (got buried by rsync load) is unclear but can probably be reconstructed if necessary. surf:/tmp/cdnetworks-check-postpurg2.log has info on which files were old (look for Age > 400k).

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Followup to this spun out to bug 707560.