849233 - All the builds installed with the stub installer have the same Build ID

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Simona Badau, Desktop QA]

Mozilla/5.0 (Windows NT 6.0; rv:22.0) Gecko/20130305 Firefox/22.0

Steps to reproduce:
1. Download the latest stub installer from:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-22.0a1.en-US.win32.installer-stub.exe 
or 
http://nightly.mozilla.org/
2. Install it
3. Launch Firefox and check the about:support page.

Expected results:
The Build ID match the latest date (which in this case is March 7th).

Actual results:
The Build ID is 20130305030833 (March 5th) instead of March 7th.

Note:
This is irrelevant of the stub installer downloaded from ftp (March 2nd, February 13th, March 7th).
Reproducible also with the latest Aurora stub installer.

Comment 1

[Jake Maul [:jakem]]

This is quite strange due to the fix in bug 829207. In this bug, Expires headers for nightly/aurora files were changed to be "modification time + 23 hours". This means the product delivery CDNs should only cache these files for up to a little under 1 day... by the time the next build rolls around, it should be ready for a new file.

I just installed Nightly myself from the link in comment 0, and received 22.0a1 (2013-03-08)... which is as expected.


However, after much gnashing of curl, I was able to generate some surprising data. Here are the ETag, Expires, and Last-Modified header of 100 requests, one per second:

First, to Akamai's CDN:
for i in {1..100}; do curl -I -H 'Host: download.cdn.mozilla.net' 'http://download-akamai.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-22.0a1.en-US.win32.installer.exe'; sleep 1; done | grep -E '^(ETag|Last-Modified|Expires)' | sort | uniq -c

  30 ETag: "1b62ff3-1551d60-4d76ac466de9f"
  70 ETag: "f700d9-1534e30-4d72e621ae312"
   2 Expires: Sat, 09 Mar 2013 13:44:20 GMT
  28 Expires: Sat, 09 Mar 2013 14:32:20 GMT
   2 Expires: Sun, 10 Mar 2013 02:14:53 GMT
  68 Expires: Sun, 10 Mar 2013 02:15:20 GMT
  30 Last-Modified: Fri, 08 Mar 2013 14:37:48 GMT
  70 Last-Modified: Tue, 05 Mar 2013 14:35:21 GMT

There's an obvious pattern... it looks almost like some nodes have really old data. A purge may fix this forever. My guess is this is old data from before the fix in bug 928207. I did do a purge when that fix went in, but it's possibly we still had some old data in Zeus' cache, which some Akamai nodes may have pulled before it expired from there.

I'll do a new purge, which should correct this. I'll also verify that Akamai's config is correct, and that it's properly obeying Cache-Control and/or Expires headers.


And for Edgecast:
for i in {1..100}; do curl -s -I -H 'Host: download.cdn.mozilla.net' 'http://wpc.1237.edgecastcdn.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-22.0a1.en-US.win32.installer.exe'; sleep 1; done | grep -E '^(ETag|Last-Modified|Expires)' | sort | uniq -c
 100 ETag: "1b62ff3-1551d60-4d76ac466de9f"
  89 Expires: Sat, 09 Mar 2013 13:37:48 GMT
   1 Expires: Sat, 09 Mar 2013 13:54:34 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:02 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:05 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:20 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:23 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:30 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:33 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:35 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:45 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:47 GMT
   1 Expires: Sat, 09 Mar 2013 13:55:53 GMT
 100 Last-Modified: Fri, 08 Mar 2013 14:37:48 GMT
Curious that there are very slight fluctuations in the Expires header here. Not sure why that would be, though you can see from ETag and Last-Modified that it is all the same file.


And finally, direct to us:
for i in {1..100}; do curl -I -H 'Host: download.cdn.mozilla.net' 'http://download-origin.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-22.0a1.en-US.win32.installer.exe' 2>/dev/null; sleep 1; done | grep -E '^(ETag|Last-Modified|Expires)' | sort | uniq -c
 100 ETag: "1b62ff3-1551d60-4d76ac466de9f"
 100 Expires: Sat, 09 Mar 2013 13:37:48 GMT
 100 Last-Modified: Fri, 08 Mar 2013 14:37:48 GMT

No problems here, exactly as expected.

Comment 2

[Jake Maul [:jakem]]

I have verified that this Akamai property is indeed set up to honor both Cache-Control and Expires headers. I have contacted Akamai for assistance. I can do a cache purge and that should solve the problem at least temporarily, but I still want their eyes on it to verify that should be a permanent fix and we won't have this problem again later on.

Comment 3

[Jake Maul [:jakem]]

This should be fixed now. Discussed with them, appears to have been a timing issue between the config change and content purge.

I have also further reduced the expires headers, to reduce the likelihood of future instances of this problem.