1099048 - Purge CDNs of thunderbird/releases/34.0b1

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Massimo Gervasini [:massimo]]

+++ This bug was initially created as a clone of Bug #968387 +++

+++ This bug was initially created as a clone of Bug #966637 +++

+++ This bug was initially created as a clone of Bug #901734 +++

We had to rebuild Thunderbird 34.0b1 for a code change, after it became visible to the CDNs. Please purge the path thunderbird/releases/34.0b1 from all the CDNs.

Comment 1

[Ludovic Hirlimann [:Usul]]

reducing to normal as I can't do anything but ping webops which I did.

Comment 2

[C. Liang [:cyliang]]

I've purged the load balancer caches that serve as the CDN origin.
I have issued purge requests at Akamai, Edgecast, and Highwinds.   These may take up to 30 minutes to complete.

Comment 3

[C. Liang [:cyliang]]

I believe the CDN purges are complete.

Comment 4

[Hal Wine [:hwine] (use NI)]

Still a couple of issues - re-running. Last results:
Fri Nov 14 09:08:47 PST 2014:  ====================================
Fri Nov 14 09:08:47 PST 2014:  KEY STATS
Fri Nov 14 09:08:47 PST 2014:  ====================================
Fri Nov 14 09:08:47 PST 2014:  
Fri Nov 14 09:08:47 PST 2014:  Config files scanned:                       4
Fri Nov 14 09:08:47 PST 2014:  Update xml files downloaded and parsed:     15710
Fri Nov 14 09:08:47 PST 2014:  Unique mar urls found:                      440
Fri Nov 14 09:08:47 PST 2014:  Failures:                                   2
Fri Nov 14 09:08:47 PST 2014:  Parallel processes used (maximum limit):    48
Fri Nov 14 09:08:47 PST 2014:  Execution time:                             223 seconds
Fri Nov 14 09:08:47 PST 2014:  
program finished with exit code 1
elapsedTime=241.266583

Comment 5

[Hal Wine [:hwine] (use NI)]

Condition not improving - back up to 8 failures

Fri Nov 14 10:10:47 PST 2014:  ====================================
Fri Nov 14 10:10:47 PST 2014:  KEY STATS
Fri Nov 14 10:10:47 PST 2014:  ====================================
Fri Nov 14 10:10:47 PST 2014:  
Fri Nov 14 10:10:47 PST 2014:  Config files scanned:                       4
Fri Nov 14 10:10:47 PST 2014:  Update xml files downloaded and parsed:     15710
Fri Nov 14 10:10:47 PST 2014:  Unique mar urls found:                      440
Fri Nov 14 10:10:47 PST 2014:  Failures:                                   8
Fri Nov 14 10:10:47 PST 2014:  Parallel processes used (maximum limit):    48
Fri Nov 14 10:10:47 PST 2014:  Execution time:                             325 seconds
Fri Nov 14 10:10:47 PST 2014:  
program finished with exit code 1
elapsedTime=392.515391

Comment 6

[C. Liang [:cyliang]]

I believe that this has been resolved.

1:01:45 PM hwine: cyliang: we're all good now - edgecast took a bit longer. I've did a script and will update diagnosis instructions on our end
1:02:29 PM cyliang: hwine: Interesting.  Edgecast had claimed that it had purged pretty quickly: I'm wondering if we're going to have to consider what it tells us in the webUI a lie.
1:03:50 PM hwine: cyliang: not 100% sure - I had some errors in my first script runs, so it might not have really been them
1:04:00 PM hwine: but - next time we'll know! :)

Comment 7

[Hal Wine [:hwine] (use NI)]

Unfortunately, we still have an issue with one provider - hwcdn.net is still showing old content:

http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Tue, 11 Nov 2014 00:00:54 GMT
< Content-Length: 16759952
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161

(output from http://hg.mozilla.org/build/braindump/file/e9dd382b4fed/releases-related/check_cdn )

Comment 8

[Hal Wine [:hwine] (use NI)]

So, Akami appears to have internal consistency issues. 2 runs below done seconds apart - first "succeeds", second "fails":

$ ./check_cdn 
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
0 [Hal@Hals-MacBook-Air releases-related]
$ ./check_cdn 
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Tue, 11 Nov 2014 00:00:54 GMT
< Content-Length: 16759952
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
0 [Hal@Hals-MacBook-Air releases-related]

Comment 9

[C. Liang [:cyliang]]

5:58:02 PM cyliang: hwine: So, I don't know that it will be any more or less consistent after I get the ACK back from Akamai that they think the purge is done.
5:58:13 PM hwine: cyliang: so, I'm not sure this is worth chasing at this point on a Friday ...
5:58:18 PM cyliang: hwine: is this enough of an issue that I need to escalate this with them?
5:58:36 PM hwine: ... it turns out the tbird team wasn't ready to release today (we didn't know that earlier)
5:58:43 PM cyliang: Ah.
5:59:16 PM hwine: Certainly not worth keeping you from a well deserved lemon drop!
5:59:29 PM cyliang: Is it okay to loop back to this on Monday and see if we're still getting inconsistent results?
6:00:00 PM hwine: wfm

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, there's a known issue with Akamai and HEAD requests, see bug 966637 comment #29.

Comment 11

[C. Liang [:cyliang]]

FWIW, the check has been reporting consistently for me this morning (Nov. 17th).

$ ./cdn_check.sh
http://ftp.mozilla.org/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://cds.d6b5y3z2.hwcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161
http://wpc.1237.edgecastcdn.net/pub/thunderbird/releases/34.0b1/update/linux-x86_64/zh-TW/thunderbird-33.0b1-34.0b1.partial.mar
< Last-Modified: Thu, 13 Nov 2014 14:47:30 GMT
< Content-Length: 16759161

Comment 12

[C. Liang [:cyliang]]

2:38:18 PM cyliang: hwine: Circling back around with you to see if it is safe to close bug 1099048
2:39:10 PM hwine: yes - and thank's very much!