Allow uploading directories to stage for ReleaseToLatest scenario

RESOLVED DUPLICATE of bug 563382

Status

Release Engineering
General
P3
normal
RESOLVED DUPLICATE of bug 563382
8 years ago
5 years ago

People

(Reporter: armenzg, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [l10n][automation])

Attachments

(1 obsolete attachment)

(Reporter)

Description

8 years ago
Created attachment 411685 [details] [diff] [review]
enabled "preserve_dirs" in ReleaseToLatest to enable uploading dirs

Ben used the attached patch locally to allow uploading a complete directory.
If we look at the output below the --base-path equals to "../../dist/stage" and the file wanted to be uploaded is "../../dist/stage/somedir/firefox-3.7a1pre.en-US.l*". Therefore, "somedir/" is preserved.

For the mobile scenario we can make the sub-directory by using PKG_PATH:
 make package PKG_PATH=something
We will have to see if the .deb files take advantage of it or if we will have to move them around manually.

Example of subdir created:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/somedir/

Log of what bhearsum did:
[cltbld@moz2-linux-slave17 installer]$ export | grep UPLOAD
declare -x POST_UPLOAD_CMD="post_upload.py --tinderbox-builds-dir mozilla-central-linux -i 20091111000000 -p firefox --release-to-tinderbox-dated-builds -b mozilla-central --release-to-latest --release-to-dated"
declare -x UPLOAD_HOST="staging-stage.build.mozilla.org"
declare -x UPLOAD_SSH_KEY="/home/cltbld/.ssh/ffxbld_dsa"
declare -x UPLOAD_TO_TEMP="1"
declare -x UPLOAD_USER="ffxbld"
[cltbld@moz2-linux-slave17 installer]$ python ../../../build/upload.py --base-path=../../dist/stage ../../dist/stage/somedir/firefox-3.7a1pre.en-US.l*
Uploading /builds/slave/mozilla-central-linux/build/obj-firefox/dist/stage/somedir/firefox-3.7a1pre.en-US.langpack.xpi
firefox-3.7a1pre.en-US.langpack.xpi                                                                                                                                                   100%  114KB 114.2KB/s   00:00    
Uploading /builds/slave/mozilla-central-linux/build/obj-firefox/dist/stage/somedir/firefox-3.7a1pre.en-US.linux-i686.crashreporter-symbols.zip
firefox-3.7a1pre.en-US.linux-i686.crashreporter-symbols.zip                                                                                                                           100%  120MB  24.0MB/s   00:05    
Uploading /builds/slave/mozilla-central-linux/build/obj-firefox/dist/stage/somedir/firefox-3.7a1pre.en-US.linux-i686.tar.bz2
firefox-3.7a1pre.en-US.linux-i686.tar.bz2                                                                                                                                             100%   10MB   9.8MB/s   00:00    
Running post-upload command: post_upload.py --tinderbox-builds-dir mozilla-central-linux -i 20091111000000 -p firefox --release-to-tinderbox-dated-builds -b mozilla-central --release-to-latest --release-to-dated
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1257926400/firefox-3.7a1pre.en-US.langpack.xpi
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1257926400/firefox-3.7a1pre.en-US.linux-i686.crashreporter-symbols.zip
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1257926400/firefox-3.7a1pre.en-US.linux-i686.tar.bz2
Upload complete
Comment on attachment 411685 [details] [diff] [review]
enabled "preserve_dirs" in ReleaseToLatest to enable uploading dirs

I'm happy to r+ this one, but I want to see staging running with it for awhile, to make sure it doesn't affect any existing builds.
Attachment #411685 - Flags: review?(bhearsum)
(Reporter)

Comment 2

8 years ago
(In reply to comment #1)
> (From update of attachment 411685 [details] [diff] [review])
> I'm happy to r+ this one, but I want to see staging running with it for awhile,
> to make sure it doesn't affect any existing builds.
So you don't want to be turned on and off depending on an extra parameter?
I will leave the local change there and see how it goes.
Summary: Allow uploading to directories to stage → Allow uploading directories to stage for ReleaseToLatest scenario
(In reply to comment #2)
> (In reply to comment #1)
> > (From update of attachment 411685 [details] [diff] [review] [details])
> > I'm happy to r+ this one, but I want to see staging running with it for awhile,
> > to make sure it doesn't affect any existing builds.
> So you don't want to be turned on and off depending on an extra parameter?
> I will leave the local change there and see how it goes.

Right, I think it's better for ReleaseToLatest to be the same for everything.
Comment on attachment 411685 [details] [diff] [review]
enabled "preserve_dirs" in ReleaseToLatest to enable uploading dirs

Turns out that just doing this will affect other builds, so we'll have to find another way.
Attachment #411685 - Attachment is obsolete: true
Attachment #411685 - Flags: review?(bhearsum)
(Reporter)

Comment 5

8 years ago
I have removed the local change.
BTW the tools repo on staging-stage is pointing to  http://hg.mozilla.org/users/catlee_mozilla.com/tools.
Not sure if we want to put it back to the build repos.

Catlee?

Comment 6

8 years ago
Re-assigning to Armen, but Future-ing as well. He can re-prioritize when he gets back on Monday.
Assignee: nobody → armenzg
Component: Release Engineering → Release Engineering: Future
Priority: -- → P3
(Reporter)

Updated

8 years ago
Blocks: 524519
(Reporter)

Updated

8 years ago
No longer blocks: 524519
(Reporter)

Comment 7

8 years ago
Restating what is to be done in here:
* If we want to use post_upload.py we have to make it allow to upload directories for the Maemo single-locale case and the Maemo L10n builds
* To do this we have to use preserve_dirs as it gets used for releases (http://mxr.mozilla.org/build/source/tools/stage/post_upload.py#139)
* After we fix the code in post_upload.py is about replacing the uses of MozillaStage for "upload" targets in bug 524519 or another bug to keep track of it

(In reply to comment #4)
> (From update of attachment 411685 [details] [diff] [review])
> Turns out that just doing this will affect other builds, so we'll have to find
> another way.
>
Ben do you remember which other builds was this affecting? I think we talked about it on IRC but I can't remember. I can see fixing this by adding a --preserve-dirs option and then make preserve_dirs=True to CopyFilesToDir if it gets used. What do you think?
> (In reply to comment #4)
> > (From update of attachment 411685 [details] [diff] [review] [details])
> > Turns out that just doing this will affect other builds, so we'll have to find
> > another way.
> >
> Ben do you remember which other builds was this affecting? I think we talked
> about it on IRC but I can't remember. I can see fixing this by adding a
> --preserve-dirs option and then make preserve_dirs=True to CopyFilesToDir if it
> gets used. What do you think?

IIRC it affected windows nightly builds. That sounds like the plan we talked about.
(Reporter)

Updated

8 years ago
Blocks: 524519
(Reporter)

Comment 9

8 years ago
For now and until using "make upload" takes a higher priority in our list I will unassign this and leave it in the future queue.
Assignee: armenzg → nobody
Priority: P3 → --
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3

Updated

8 years ago
Whiteboard: [l10n][automation]
(Assignee)

Updated

8 years ago
Assignee: nobody → aki
(Assignee)

Comment 11

8 years ago
Is there anyone who wants this outside of the Maemo build?
If so, any objections to using link to latest?

(My current plan is to fix --link-to-latest, then close this as a dup or incomplete).
(Reporter)

Comment 12

8 years ago
I am fine with your plan.
(Assignee)

Comment 13

7 years ago
My solution here is two pronged:

1) use --link-to-latest for mobile, bug 580495 (not fixed, but will be at some point in the future)
2) allow for subdirectory uploads (patch in bug 563382, r+'ed, will land tomorrow, hopefully).

I believe the r+'ed patch in bug 563382 is satisfactory for this bug; it'll solve the multilocale make upload scenario, which is specifically why this bug was filed iirc.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 563382
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.