upload firefox nightlies to latest-BRANCH-PLATFORM directories

REOPENED
Unassigned

Status

Release Engineering
General
P3
normal
REOPENED
8 years ago
3 months ago

People

(Reporter: Robert Sayre, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [cleanup][automation][postupload])

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
Looks like it has been busted for two days.

http://nightly.mozilla.org/

Comment 1

8 years ago
Created attachment 459705 [details] [diff] [review]
stopgap nightly.m.o patch

This patch builds a dictionary of builds, and only includes the first of each file it encounters.  Since the parsing url is sorted by date (descending), this only grabs the most recent build.  It also links to the url sorted by date, which ameliorates the multiple-file issue in the index.

This doesn't solve the root cause, however.
I think the most elegant solution there is to have a latest- directory per branch per platform (e.g. latest-mozilla-central-win32) which would allow us to remove the directory/link and recreate it per upload, which sidesteps any per-file logic/regexes.

More on that here: http://drkscrtlv.livejournal.com/320620.html
(--link-to-latest would be bug 580495; there is no bug open to upload to a platform-specific latest- dir for firefox builds atm.)

I'm willing to do that work if there's buy-in, but I won't be able to in the next few weeks.  In the meantime, this patch should keep nightly.m.o from experiencing these symptoms.
Attachment #459705 - Flags: review?

Updated

8 years ago
Attachment #459705 - Flags: review? → review?(sayrer)
(Reporter)

Updated

8 years ago
Attachment #459705 - Flags: review?(sayrer) → review+

Comment 2

8 years ago
Comment on attachment 459705 [details] [diff] [review]
stopgap nightly.m.o patch

http://hg.mozilla.org/webtools/nightly/rev/cbcac1390d56

How does this get pushed live?
Attachment #459705 - Flags: checked-in+
I removed the old builds, too.

Updated

8 years ago
Depends on: 581484

Comment 4

8 years ago
Ok, the directory is cleaned up + the new script is live.
Morphing the bug for the longer term fix.
Depends on: 580495
Priority: -- → P3
Summary: 2 copies of nightly builds since we changed the name → upload firefox nightlies to latest-BRANCH-PLATFORM directories
(In reply to comment #4)
> Ok, the directory is cleaned up + the new script is live.
> Morphing the bug for the longer term fix.

Grabbing, as I *think* this is addressed by the proposal http://oduinn.com/blog/2011/01/11/change-to-fennec-firefox-xulrunner-directories-on-ftp-m-o/

(cross-posted in dev.planning https://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/6a58551742033faa )


If I am missing something, and this proposal does *not* address the problem, please give details.
Assignee: nobody → joduinn
OS: Mac OS X → All

Updated

7 years ago
Duplicate of this bug: 672626

Comment 7

7 years ago
This does not address the problem.  Comment 1 has details.

Updated

7 years ago
Blocks: 679827
found in triage.
Assignee: joduinn → nobody
Assignee: nobody → coop
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: release → catlee
Whiteboard: [cleanup][automation]
Whiteboard: [cleanup][automation] → [cleanup][automation][postupload]
Mass re-assign of post_upload.py bugs to Tomcat.
Assignee: coop → cbook
Assignee: cbook → nobody
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
We're moving to s3, this doesn't make sense anymore! We also have bouncer links that point at the latest version, eg: https://download.mozilla.org/?product=firefox-aurora-latest-ssl&os=win64&lang=en-US
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX

Comment 11

4 months ago
11:41 <Callek> wow we ``still`` have Firefox 57 in https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

This does make sense, until we get rid of latest-BRANCH dirs.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Bug 1457816 for automated cleanup of the latest directories. That'll always have some sort of buffer for closed trees and needing to serve a nightly so this bug might still be a good plan.

FWIW, we no longer have post_upload's --link-to-latest available. With some special set of AWS credentials we might build a list of keys in platform-specific latest, push in the new files, and delete any keys that should no longer be present. Bonus points for doing a cache invalidation using bug 1393990.

Comment 13

3 months ago
My vote is for killing the latest- dirs in favor of indexes.
(Assignee)

Updated

3 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.