745553 - Updates for some Nightly/ESR10 builds are missing

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Seen by our Mozmill CI tests lately. There are some platforms which do not offer updates to the latest version of localized builds. Clicking on the AUS link in the reports show an empty document. Here one example:

ja on Win Vista: http://mozmill-ci.blargon7.com/#/update/report/4a49d39bc99bcebf1f54d433671df007

All broken updates can be found in the overview:
http://mozmill-ci.blargon7.com/#/update/reports?branch=14.0&platform=All&from=2012-04-12&to=2012-04-15

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm, the same applies to the OS X 10.0esr10 nightly for en-US:
http://mozmill-ci.blargon7.com/#/update/report/4a49d39bc99bcebf1f54d433671c1cf9

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Speaking to comment #0:
* according to the overview, mozmill checked en-US, de, and ja; both locales weren't served an update
* in fact that's a problem for all localised builds for Windows on m-c (from inspection of the update snippets)
* it turns out that both of the properties previous_buildid and buildid have value 20120414030731, which means we didn't receive the en-US build from the 15th when downloading it from ftp.m.o
* we pulled down these files
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-14.0a1.en-US.win32.zip
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-14.0a1.en-US.win32.installer.exe

I bet this is more problems with caching. The filename doesn't change from day to day, and ftp.m.o caches files for one hour after a request. It doesn't check if the file has changed if it gets another request in that hour. So if anyone happens to download the en-US zip/exe in the hour before the new build finishes then ftp.m.o will serve the older build. This isn't windows specific, but is more likely to happen for windows since that's where the users are.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Oh, and we still uploadeded snippets to AUS but that intentionally requires that the buildID must increase for it to make an offer. When it's the same it just returns the empty <updates></updates>.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Hm, the same applies to the OS X 10.0esr10 nightly for en-US:
> http://mozmill-ci.blargon7.com/#/update/report/
> 4a49d39bc99bcebf1f54d433671c1cf9

This was some missing symlinks for Mac, and would have been broken since nightly updates on esr10 were setup up. Fixed.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Nick Thomas [:nthomas] from comment #4)
> This was some missing symlinks for Mac, and would have been broken since
> nightly updates on esr10 were setup up. Fixed.

Thanks. That part works now for ESR builds on OS X:
http://mozmill-ci.blargon7.com/#/update/report/4a49d39bc99bcebf1f54d433672d18e9

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Has something been fixed here for the Windows builds yesterday? I cannot see any failure for today's updates:

http://mozmill-ci.blargon7.com/#/update/reports?branch=14.0&platform=All&from=2012-04-14&to=2012-04-17

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

As I said in comment #2, given the caching on ftp.m.o we are the whim of whatever requests come in just prior to doing nightly repacks.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Use wget --no-cache

Axel, this will mean we always pull the very latest en-US files from a latest-<branch> directory on ftp.m.o, bypassing any caching going on there. I'm hoping this will be a temporary work around while we work on something better with IT, but no-one hold me to that.

Comment 9

[Axel Hecht [:Pike]]

Comment on attachment 627100 [details] [diff] [review]
Use wget --no-cache

Review of attachment 627100 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me, passing an additional request to Kyle.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 627100 [details] [diff] [review]
Use wget --no-cache

http://hg.mozilla.org/integration/mozilla-inbound/rev/eb6953f1b20b

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/eb6953f1b20b