Closed Bug 703729 Opened 8 years ago Closed 8 years ago

Remove pub/mozilla.org/firefox/releases/8.0.1 on pv-mirror01 and pv-mirror02

Categories

(mozilla.org Graveyard :: Server Operations, task, critical)

x86
Linux
task
Not set
critical

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: nthomas, Assigned: justdave)

References

Details

Several releases got deleted (FF 8.0.1) or added to the exclude for prereleases (FF 8.0b1-6 and some SeaMonkey and Thunderbird) in the last day or so. Until we get bug 700798 those removals happen very slowly (10 files and/or dirs every 10 mins).

Could you please manually delete the 8.0.1 directory from firefox/releases on pv-mirror01.m.o and pv-mirror02. We're respinning that release and I'd like to make sure we have these two machines, and the CDN which uses pv-mirror01, reset to a clean state.
Blocks: 701537
Done.
Assignee: server-ops → justdave
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Do we have a contact at cdnetworks who could dump all of firefox/releases/8.0.1/ from their systems ? They seem to have a 7 day timeout, eg

http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/update/linux-i686/af/firefox-8.0.1.complete.mar
HTTP/1.1 200 OK
Date: Mon, 21 Nov 2011 10:49:52 GMT
Server: PWS/1.7.3.7
X-Px: ms h0-s1011.p10-sjc ( h0-s1023.p10-sjc), ht-d h0-s1023.p10-sjc.cdngp.net
ETag: "2be68501-f2bdb9-4b1c4e3d5f940"
Cache-Control: max-age=604800
Expires: Wed, 23 Nov 2011 07:30:20 GMT
Age: 443972
Accept-Ranges: bytes
Content-Length: 15908281
Content-Type: application/octet-stream
Last-Modified: Tue, 15 Nov 2011 12:11:41 GMT
Connection: keep-alive

We could work around by using 8.0.1-real but ewwww.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
FYI, there's no firm ship time yet but probably going to be Pacific late morning or so.
Raising to critical because we could be asked to push really soon now.
Severity: major → critical
OK, we do have a cache purge utility in the control panel for this CDN.

http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/ ===> REQUEST ACCEPTED

Says it'll take up to 15 minutes to purge.
Status: REOPENED → NEW
Appears to have completed. :)
Status: NEW → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
I just checked it:

# stage
cd /pub/mozilla.org/firefox/nightly/8.0.1-candidates/build2/
find linux-* mac/ win32* update/ -type f | grep -vE 'asc|checksums|xpi|zip|txt|log' | sort > /tmp/cdnetworks-list.log
(while read line; do
  url="http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/$line"
  echo "$url"
  curl -sI "$url"
  echo
done < "/tmp/cdnetworks-list.log") | tee /tmp/cdnetworks-check2.log

grep -c "HTTP/1.1 200" /tmp/cdnetworks-check2.log
0
Status: RESOLVED → VERIFIED
After this we pushed build2 to the mirrors, and re-enabled the CDN. It started serving build1 files in some cases so we had to turn it off again. 

The precise timing of enabling the CDN in bouncer, the rsync to pv-mirror01, and the temporary unavailability of pv-mirror01 (got buried by rsync load) is unclear but can probably be reconstructed if necessary. surf:/tmp/cdnetworks-check-postpurg2.log has info on which files were old (look for Age > 400k).
Followup to this spun out to bug 707560.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.