950119 - Aurora download links give a 404 (for some chunk of time on every release day, after "latest-mozilla-aurora" directory is updated w/ new builds)

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Frederik Braun [:freddy]]

The download link on http://www.mozilla.org/en-US/firefox/aurora/ points to 27 which returns a 404, dholbert argued this might be because we just branched to 28 but the builds aren't done yet?!

Comment 1

[Daniel Holbert [:dholbert]]

I think the builds are done; the link just hasn't been updated yet.

In particular, the broken link (on my Linux machine) points to:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-27.0a2.en-US.linux-i686.tar.bz2
which is 404.

If I strip off the filename, http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/ lists a bunch of version-28 files, all with datestamps from today.

Comment 2

[Daniel Holbert [:dholbert]]

(I'm aware that Aurora 28 builds may not be ready for release yet; that's fine.

The issue is that we need to have *some* functional download link, pointing to whichever Aurora release we currently want visitors to that site to receive. Right now, it's a busted link, and that's pretty bad.)

Comment 3

[Kohei Yoshino [:kohei]]

The links has been updated to 28.

Comment 4

[Frederik Braun [:freddy]]

I think this bug wasn't only about the broken link but about the race condition between file changes and uploads. I suppose this happened not only this week, did it?

Comment 5

[Kohei Yoshino [:kohei]]

The product versions on the Web site are manually updated by the Release Drivers team for now, so yes, this issue is sometimes happening as you said. You can find the history at:

http://viewvc.svn.mozilla.org/vc/libs/product-details/?date=month&view=query

Bug 937865 has resolved the same issue for Beta. I'll ask the team to deal with Aurora too.

Comment 6

[Mike Alexis [:malexis]]

Kohei, is the remaining issue being worked on in another bug? Can we resolve this?

Comment 7

[Kohei Yoshino [:kohei]]

There's nothing here we can do for now. Closing as the reported issue has been fixed.

Comment 8

[Daniel Holbert [:dholbert]]

By "reported issue", do you mean "the specific time that this happened in December"?

If so, I don't think this should really be considered fixed; see comment 4, when this was previously reopened & clarified. [Clarifying summary per comment 4, too.]

Comment 9

[Kohei Yoshino [:kohei]]

Oh, true, but this is not an issue on the mozilla.org site but on the mirror system. Moving the product/component...

Comment 10

[Aki Sasaki (not active)]

Possibly a dup of bug 580495.

Comment 11

[Kohei Yoshino [:kohei]]

Thanks, adding a dependency to track.

Comment 12

[Daniel Holbert [:dholbert]]

(In reply to Aki Sasaki [:aki] from comment #10)
> Possibly a dup of bug 580495.

I'm not sure, but it doesn't look like it to me. (though I may be misunderstanding) That bug seems to be about tweaking the way we create the "latest-*" directory (whether it's a symlink vs. an actual copy).  I don't think that'd help here.

The underlying problem here is that the download URL uses the non-version-specific "latest-mozilla-aurora" directory, combined with a filename that has the version number hardcoded into it, per comment 1. For example:
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-27.0a2.en-US.linux-i686.tar.bz2

One way we could fix this would be to create a dummy file (or redirect) in latest-mozilla-aurora/ (or somewhere similar) that doesn't have the version number hardcoded into it, and make sure that dummy file or redirect gets updated to point to the right tarball, even when the version number changes.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

I don't understand why this is an issue with Aurora. Typically we merge from central and have builds with the new version in latest-mozilla-aurora{,-l10n} within 12 hours, when the next nightly happens. I would be really surprised if we are updating the website before QA has signed off on the new version, which is about a week after the merge.

Comment 14

[Daniel Holbert [:dholbert]]

(In reply to Nick Thomas [:nthomas] from comment #13)
> I don't understand why this is an issue with Aurora. Typically we merge from
> central and have builds with the new version in
> latest-mozilla-aurora{,-l10n} within 12 hours

Yup, and that's presumably the step that (currently) triggers the 404. (because e.g. firefox-27.0a2.en-US.linux-i686.tar.bz2 disappears from the "latest-mozilla-aurora" folder)

> I would be really surprised if we are updating the website before
> QA has signed off on the new version, which is about a week after the merge.

I don't know if we do or not (and I suspect we don't).  The point is, we should still have *some* Aurora build available for download during that time, whether it's the old one or the new one.  Currently, we'll just have a 404.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Daniel Holbert [:dholbert] from comment #14)
> Yup, and that's presumably the step that (currently) triggers the 404.
> (because e.g. firefox-27.0a2.en-US.linux-i686.tar.bz2 disappears from the
> "latest-mozilla-aurora" folder)

Ah, I forgot about the cleanup of old builds. en-US is done manually, but l10n is a find -mtime +7, ie 8 days or older. Unless someone jumps the gun in bug 703559 cleanup should not be an issue.

Right now I don't get a 404 for these
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-28.0a2.en-US.mac.dmg
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/firefox-28.0a2.de.mac.dmg
but I can't convince mozilla.org to give me a localized page, so the second one is constructed from knowing where the files live.

Comment 16

[Daniel Holbert [:dholbert]]

Ah, you're right -- latest-mozilla-aurora currently has both v28 and v29 in it. Good; I was assuming it only had one or the other.

So then this just comes down to making sure we don't clean up the old version before the website is updated to point to the new version, yeah.

Comment 17

[Daniel Holbert [:dholbert]]

See also bug 968901, for a related issue with http://www.mozilla.org/en-US/firefox/aurora/all/ (with different download URLs, not pointing directly to ftp)

Comment 19

[Frederik Braun [:freddy]]

I have seen this bug and duplicates crawl up every six weeks.
I'm concerned the root cause is not being fixed or tracked.

Comment 20

[Daniel Holbert [:dholbert]]

freddyb: did you actually see / hear of this bug's sort of issues cropping up this release-week?

(Note that the duping of bug 962096 here was incorrect, per bug 962096 comment 5.)

(I just tried this bug's STR and got correct results (successful download. (getting Firefox 29, so it looks like we're not offering the new Aurora version yet))

Comment 21

[Frederik Braun [:freddy]]

I must admit that I didn't see it but just assumed so from the dupe mail I got from following this bug. I also didn't follow up on the dupe itself, thanks for highlighting this :)

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Fixed by a Release Engineering process change to not delete old versions in latest-mozilla-aurora{,-l10n} until we move on to the new version.