581265 - upload firefox nightlies to latest-BRANCH-PLATFORM directories

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Robert Sayre]

Looks like it has been busted for two days.

http://nightly.mozilla.org/

Comment 1

[Aki Sasaki (not active)]

Attachment: stopgap nightly.m.o patch

This patch builds a dictionary of builds, and only includes the first of each file it encounters.  Since the parsing url is sorted by date (descending), this only grabs the most recent build.  It also links to the url sorted by date, which ameliorates the multiple-file issue in the index.

This doesn't solve the root cause, however.
I think the most elegant solution there is to have a latest- directory per branch per platform (e.g. latest-mozilla-central-win32) which would allow us to remove the directory/link and recreate it per upload, which sidesteps any per-file logic/regexes.

More on that here: http://drkscrtlv.livejournal.com/320620.html
(--link-to-latest would be bug 580495; there is no bug open to upload to a platform-specific latest- dir for firefox builds atm.)

I'm willing to do that work if there's buy-in, but I won't be able to in the next few weeks.  In the meantime, this patch should keep nightly.m.o from experiencing these symptoms.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 459705 [details] [diff] [review]
stopgap nightly.m.o patch

http://hg.mozilla.org/webtools/nightly/rev/cbcac1390d56

How does this get pushed live?

Comment 3

[[e-mail] (:bhearsum)]

I removed the old builds, too.

Comment 4

[Aki Sasaki (not active)]

Ok, the directory is cleaned up + the new script is live.
Morphing the bug for the longer term fix.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> Ok, the directory is cleaned up + the new script is live.
> Morphing the bug for the longer term fix.

Grabbing, as I *think* this is addressed by the proposal http://oduinn.com/blog/2011/01/11/change-to-fennec-firefox-xulrunner-directories-on-ftp-m-o/

(cross-posted in dev.planning https://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/6a58551742033faa )


If I am missing something, and this proposal does *not* address the problem, please give details.

Comment 7

[Aki Sasaki (not active)]

This does not address the problem.  Comment 1 has details.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 9

[Chris Cooper [:coop] (he/him)]

Mass re-assign of post_upload.py bugs to Tomcat.

Comment 10

[[e-mail] (:bhearsum)]

We're moving to s3, this doesn't make sense anymore! We also have bouncer links that point at the latest version, eg: https://download.mozilla.org/?product=firefox-aurora-latest-ssl&os=win64&lang=en-US

Comment 11

[Aki Sasaki (not active)]

11:41 <Callek> wow we ``still`` have Firefox 57 in https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

This does make sense, until we get rid of latest-BRANCH dirs.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1457816 for automated cleanup of the latest directories. That'll always have some sort of buffer for closed trees and needing to serve a nightly so this bug might still be a good plan.

FWIW, we no longer have post_upload's --link-to-latest available. With some special set of AWS credentials we might build a list of keys in platform-specific latest, push in the new files, and delete any keys that should no longer be present. Bonus points for doing a cache invalidation using bug 1393990.

Comment 13

[Aki Sasaki (not active)]

My vote is for killing the latest- dirs in favor of indexes.

Comment 14

[Julien Cristau [:jcristau]]

I don't think this is relevant anymore.