1336732 - Aurora CDN deploys multiple build versions depending on DNS entry (same URL, different sha512 checksums!)

Status: RESOLVED FIXED

(Cloud Services :: Operations: Product Delivery, task)

Description

[denys]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170203084124

Steps to reproduce:

Depending on what CDN is used (which is selected via a load balancer / DNS I suppose), the results of `wget https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-aurora/firefox-53.0a2.en-US.linux-x86_64.tar.bz2 -O /tmp/ff && sha512sum /tmp/ff && rm /tmp/ff` differ.


Actual results:

When 52.222.242.7 was used yesterday, the file sha512 sum of firefox-53.0a2.en-US.linux-x86_64.tar.bz2 was `ce8e6da4d3e15e9c47c5020c653997516a05d9de44f072ce80571085ae78b6768ce75da1fc60a7d442f4574473619b9cbbac24eefeeaaaf39807bfc6deef91d6`, but when 54.192.53.86 was selected (again, I'm pretty sure it's a DNS thing) the file served was different: its hash in fact was `390427cf5c5794c576dfbaded28f74f19af55ed4b8e62a1b10d361ca6d30b5ed4adbfc96008a10f5dda5e4ab7a0c5ab3ed62b47ac06b25d374befef7e35f7419`

I'm not sure if this is the expected result, but I'm pretty sure that two builds should not be served under the same URL.
The problem came when I implemented my aur-firefox-dev-automaintainer (http://github.com/denysvitali/aur-firefox-dev-automaintainer) which basically checks if there is an update for firefox-dev (via https://aus5.mozilla.org/update/6/Firefox/%VERSION%/%BUILDID%/Linux_x86_64-gcc3/en-US/aurora/Linux/NA/default/default/update.xml?force=1) and if there is a new build it pushes some changes to this AUR: https://aur.archlinux.org/packages/firefox-dev/ .
I had to manually calculate the sha512sum for every file, since apparently the checksums for Linux aren't calculated on Mozilla's side (see https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-aurora/ , Linux build checksums are available only for 52.0a2 - but for 53.0a2 there isn't any) and noticed this strange behavior.

Is it expected? If yes, can we have a custom URL to download a specific build w/ the correct sha512 files checksum for Linux?

You can see my result here:
https://gist.github.com/denysvitali/f5db971fa06731752363b0ff319f6532
and here:
https://gist.github.com/denysvitali/8ff7e74e762fa8d5ace048c491118d10


Expected results:

Same checksum, for every time the file is downloaded from any CDN, plus (eventually) SHA512sums for the latest linux builds

Comment 1

[Loic]

Retriage to NetOps team if need be.

Comment 2

[denys]

We're still having some troubles with https://archive.mozilla.org/
This command gives different results / fails, depending on which server is chosed by the Load Balancer / DNS

```
wget https://archive.mozilla.org/pub/firefox/nightly/2017/02/2017-02-08-08-41-06-mozilla-aurora/firefox-53.0a2.en-US.linux-x86_64.tar.bz2 https://archive.mozilla.org/pub/firefox/nightly/2017/02/2017-02-08-08-41-06-mozilla-aurora/firefox-53.0a2.en-US.linux-x86_64.tar.bz2.asc && gpg --verify firefox-53.0a2.en-US.linux-x86_64.tar.bz2.asc firefox-53.0a2.en-US.linux-x86_64.tar.bz2 && sha512sum firefox-53.0a2.en-US.linux-x86_64.tar.bz2.asc firefox-53.0a2.en-US.linux-x86_64.tar.bz2
```

Please, follow the thread on https://aur.archlinux.org/packages/firefox-dev/#news for more info

Comment 3

[Bruno Pagani]

Also just for reference, I find that too and posted about it here: https://bugzilla.mozilla.org/show_bug.cgi?id=1305139#c63

Comment 4

[Bruno Pagani]

Was reported to me again for french nightly tarball y-day.

Comment 5

[Bruno Pagani]

They are still some reports here for instance: https://aur.archlinux.org/pkgbase/firefox-developer/?comments=all

Can someone look into this?

Comment 6

[Bruno Pagani]

I should add that {archive,ftp}.mozilla.org seems to be also handled by CDN now, so they’re not suitable workaround (excepted for the fact that this too might work when the other one does not and vice-versa, but they could as well all not work I guess).

Comment 7

[Chris AtLee [:catlee]]

I expect the underlying issue is that the CDNs are caching the contents for too long, especially for any of the 'latest' URLs, e.g. https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-aurora/firefox-53.0a2.en-US.linux-x86_64.tar.bz2

:oremj, what options do we have for having a shorter cache lifetime on these files?

Comment 8

[Jeremy Orem [:oremj]]

We can change the cache time per file or globally. If we want to change it for certain files, it will need to be done at object upload time.

Comment 9

[Bruno Pagani]

You want it to change per file I think, only for those “latest” URLs indeed. Everything else is static from what I see.

Comment 10

[Chris AtLee [:catlee]]

Yeah, the cache time for 'latest' files should be fairly short - like one hour.

Comment 11

[Jeremy Orem [:oremj]]

Everything matching ^pub/.*/nightly/latest.* has "Cache-Control: max-age=3600". Is anything else needed here?

Comment 12

[Chris AtLee [:catlee]]

That applies only to objects published via post_upload. Bug 1344572 tracks setting the Cache-Control headers for objects that are PUT directly into S3.

Comment 13

[Jeremy Orem [:oremj]]

Can I close this out?

Comment 14

[Chris AtLee [:catlee]]

Yes, thanks!