Closed Bug 1099048 Opened 8 years ago Closed 8 years ago

Purge CDNs of thunderbird/releases/34.0b1

Categories

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: massimo, Assigned: cliang)

Details

(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1866] )

+++ This bug was initially created as a clone of Bug #968387 +++

+++ This bug was initially created as a clone of Bug #966637 +++

+++ This bug was initially created as a clone of Bug #901734 +++

We had to rebuild Thunderbird 34.0b1 for a code change, after it became visible to the CDNs. Please purge the path thunderbird/releases/34.0b1 from all the CDNs.
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1866]
reducing to normal as I can't do anything but ping webops which I did.
Severity: critical → normal
Assignee: server-ops-webops → cliang
I've purged the load balancer caches that serve as the CDN origin.
I have issued purge requests at Akamai, Edgecast, and Highwinds.   These may take up to 30 minutes to complete.
I believe the CDN purges are complete.
Still a couple of issues - re-running. Last results:
Fri Nov 14 09:08:47 PST 2014:  ====================================
Fri Nov 14 09:08:47 PST 2014:  KEY STATS
Fri Nov 14 09:08:47 PST 2014:  ====================================
Fri Nov 14 09:08:47 PST 2014:  
Fri Nov 14 09:08:47 PST 2014:  Config files scanned:                       4
Fri Nov 14 09:08:47 PST 2014:  Update xml files downloaded and parsed:     15710
Fri Nov 14 09:08:47 PST 2014:  Unique mar urls found:                      440
Fri Nov 14 09:08:47 PST 2014:  Failures:                                   2
Fri Nov 14 09:08:47 PST 2014:  Parallel processes used (maximum limit):    48
Fri Nov 14 09:08:47 PST 2014:  Execution time:                             223 seconds
Fri Nov 14 09:08:47 PST 2014:  
program finished with exit code 1
elapsedTime=241.266583
Condition not improving - back up to 8 failures

Fri Nov 14 10:10:47 PST 2014:  ====================================
Fri Nov 14 10:10:47 PST 2014:  KEY STATS
Fri Nov 14 10:10:47 PST 2014:  ====================================
Fri Nov 14 10:10:47 PST 2014:  
Fri Nov 14 10:10:47 PST 2014:  Config files scanned:                       4
Fri Nov 14 10:10:47 PST 2014:  Update xml files downloaded and parsed:     15710
Fri Nov 14 10:10:47 PST 2014:  Unique mar urls found:                      440
Fri Nov 14 10:10:47 PST 2014:  Failures:                                   8
Fri Nov 14 10:10:47 PST 2014:  Parallel processes used (maximum limit):    48
Fri Nov 14 10:10:47 PST 2014:  Execution time:                             325 seconds
Fri Nov 14 10:10:47 PST 2014:  
program finished with exit code 1
elapsedTime=392.515391
I believe that this has been resolved.

1:01:45 PM hwine: cyliang: we're all good now - edgecast took a bit longer. I've did a script and will update diagnosis instructions on our end
1:02:29 PM cyliang: hwine: Interesting.  Edgecast had claimed that it had purged pretty quickly: I'm wondering if we're going to have to consider what it tells us in the webUI a lie.
1:03:50 PM hwine: cyliang: not 100% sure - I had some errors in my first script runs, so it might not have really been them
1:04:00 PM hwine: but - next time we'll know! :)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Unfortunately, we still have an issue with one provider - hwcdn.net is still showing old content:

http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Tue, 11 Nov 2014 00:00:54 GMT
< Content-Length: 16759952
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161

(output from http://hg.mozilla.org/build/braindump/file/e9dd382b4fed/releases-related/check_cdn )
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
So, Akami appears to have internal consistency issues. 2 runs below done seconds apart - first "succeeds", second "fails":

$ ./check_cdn 
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
0 [Hal@Hals-MacBook-Air releases-related]
$ ./check_cdn 
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Tue, 11 Nov 2014 00:00:54 GMT
< Content-Length: 16759952
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
0 [Hal@Hals-MacBook-Air releases-related]
5:58:02 PM cyliang: hwine: So, I don't know that it will be any more or less consistent after I get the ACK back from Akamai that they think the purge is done.
5:58:13 PM hwine: cyliang: so, I'm not sure this is worth chasing at this point on a Friday ...
5:58:18 PM cyliang: hwine: is this enough of an issue that I need to escalate this with them?
5:58:36 PM hwine: ... it turns out the tbird team wasn't ready to release today (we didn't know that earlier)
5:58:43 PM cyliang: Ah.
5:59:16 PM hwine: Certainly not worth keeping you from a well deserved lemon drop!
5:59:29 PM cyliang: Is it okay to loop back to this on Monday and see if we're still getting inconsistent results?
6:00:00 PM hwine: wfm
FWIW, there's a known issue with Akamai and HEAD requests, see bug 966637 comment #29.
FWIW, the check has been reporting consistently for me this morning (Nov. 17th).

$ ./cdn_check.sh
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
2:38:18 PM cyliang: hwine: Circling back around with you to see if it is safe to close bug 1099048
2:39:10 PM hwine: yes - and thank's very much!
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.