524519 - use post_upload.py instead of MozillaStage for Maemo builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

This will help us be more consistent with what the rest of the factories are using and gives us the benefit and flexibility of using post_upload.py

Comment 1

[Armen [:armenzg]]

We need to upload the single-locale build of Fennec so we can create a localized deb from it. We currently upload the fennec deb with the multi-locale's one.

Comment 2

[Armen [:armenzg]]

Desired upload directory for the single locale build makes sense to upload to:
- latest-mobile-<branch>/en-US
- latest-mobile-<branch>-l10n/en-US

Comment 3

[Armen [:armenzg]]

Attachment: [xulrunner] add upload and upload-deb targets

Comment 4

[Armen [:armenzg]]

Attachment: [mobile] add upload and upload-deb targets

Comment 5

[Armen [:armenzg]]

Both patches have worked but a warning is added because they shadow the target in http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#516

Comment 6

[Armen [:armenzg]]

We also need to mount the keys in sbox's .ssh directory.
We will have to do this through puppet.

mkdir /scratchbox/users/cltbld/home/cltbld/.ssh
mount --bind /home/cltbld/.ssh /scratchbox/users/cltbld/home/cltbld/.ssh

Comment 7

[Axel Hecht [:Pike]]

Comment on attachment 411204 [details] [diff] [review]
[xulrunner] add upload and upload-deb targets

Not sure why xulrunner would be locale-dependent.

I'd prefer to have the pieces going into the deb file name factored somewhere between changelog.in and Makefile.in, or at least a comment in both places.

Comment 8

[Axel Hecht [:Pike]]

Comment on attachment 411205 [details] [diff] [review]
[mobile] add upload and upload-deb targets

I don't like the shadowing of the packager.mk target in both patches. Not sure why we're having a upload-deb and a upload target.

Comment 9

[Armen [:armenzg]]

(In reply to comment #7)
> (From update of attachment 411204 [details] [diff] [review])
> Not sure why xulrunner would be locale-dependent.
> 
You are right it should not. Since we added multi-locale builds we have been using AB_CD='multi' to avoid overwriting. I can fix this.

> I'd prefer to have the pieces going into the deb file name factored somewhere
> between changelog.in and Makefile.in, or at least a comment in both places.
Would you mind expanding?

(In reply to comment #8)
> (From update of attachment 411205 [details] [diff] [review])
> I don't like the shadowing of the packager.mk target in both patches.
Me neither but I can't think of other way that I *like*
> Not sure why we're having a upload-deb and a upload target.
Dependent builds should not be uploading deb files that is why I will not call the upload-deb on buildbot. The other option is to have an UPLOAD_DEB environment variable as UPLOAD_TESTS.

Comment 10

[Armen [:armenzg]]

(In reply to comment #9)
> > Not sure why we're having a upload-deb and a upload target.
> Dependent builds should not be uploading deb files that is why I will not call
> the upload-deb on buildbot. The other option is to have an UPLOAD_DEB
> environment variable as UPLOAD_TESTS.
I remembered the reason behind this (since we do upload debs for dependent builds). We need to allow calling the upload-deb target separately from plain upload since we will want to upload to the chinook repositories at some point.

Comment 11

[Axel Hecht [:Pike]]

I guess that we still want to have upload put the built stuff up in a dir, and potentially do the debs into a place where the repo logic can pick them up?

Something for the mobiles to make a call on.

I'm still concerned about the overloaded target, slight refactorings make those go bad. I'd rather go for a upload-mobile target or something instead.

Comment 12

[Aki Sasaki (not active)]

Yes, please upload to ftp as normal, and the deb signing buildbot will pick those up and upload/sign the repository.

Comment 13

[Armen [:armenzg]]

Attachment: [mobile] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

Comment 14

[Armen [:armenzg]]

Attachment: [xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

Comment 15

[Armen [:armenzg]]

The work with regards where DEB_BUILD_ARCH should be set is being tracked in bug 527740.

Do we want "upload" to be declared in build.mk like in http://mxr.mozilla.org/mozilla1.9.2/source/xulrunner/build.mk#86?

Comment 16

[(not currently active) Ted Mielczarek]

You need that if you want "make upload" in the root of the objdir to work. Of course, for mobile's dual-sub-objdir setup, it's not clear that that's a big win.

Comment 17

[Armen [:armenzg]]

Attachment: [mobile] add the ability to call "make upload" from root of mobile's objdir

This will help us keep up with consistency with other projects.

Comment 18

[Axel Hecht [:Pike]]

... actually, how does including the deb package in the upload target impact platforms for which there is no deb, i.e., desktop win and mac?

Comment 19

[Armen [:armenzg]]

(In reply to comment #18)
> ... actually, how does including the deb package in the upload target impact
> platforms for which there is no deb, i.e., desktop win and mac?
>
Right now the only consumer will be the Maemo builds. One step at a time will get us to Rome.

Comment 20

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 411514 [details] [diff] [review]
[mobile] add the ability to call "make upload" from root of mobile's objdir

How much is this needed?

Comment 21

[Armen [:armenzg]]

(In reply to comment #20)
> (From update of attachment 411514 [details] [diff] [review])
> How much is this needed?
>
This one is not essential but it makes things consistent.

Comment 22

[Armen [:armenzg]]

We can land these on Monday when I am back if you prefer me to be around.
Otherwise I believe they are no risk to land (I did throw the xulrunner patch into try without affecting the build process of them).

Comment 23

[Chris Cooper [:coop] (he/him)]

(In reply to comment #22)
> We can land these on Monday when I am back if you prefer me to be around.
> Otherwise I believe they are no risk to land (I did throw the xulrunner patch
> into try without affecting the build process of them).

I'm planning on landing these tomorrow morning EST when it is quiet.

Comment 24

[Chris Cooper [:coop] (he/him)]

Comment on attachment 411483 [details] [diff] [review]
[mobile] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

http://hg.mozilla.org/mobile-browser/rev/4f1a9afbddc1

Comment 25

[Chris Cooper [:coop] (he/him)]

Comment on attachment 411514 [details] [diff] [review]
[mobile] add the ability to call "make upload" from root of mobile's objdir

http://hg.mozilla.org/mobile-browser/rev/4f1a9afbddc1

Comment 26

[Chris Cooper [:coop] (he/him)]

Comment on attachment 411484 [details] [diff] [review]
[xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

http://hg.mozilla.org/mozilla-central/rev/61937ebd2a96

Comment 27

[Armen [:armenzg]]

Comment on attachment 411484 [details] [diff] [review]
[xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

Pretty straight forward patch which does not modify any targets currently in use.

This patch will allow us to use the "make upload" target in the mobile scenario like in Firefox.

Comment 28

[Armen [:armenzg]]

Let me reword this bug's description now that things have changed a little.

* We currently upload, in the mobile scenarios, with MozillaStage and we want to move to post_upload.py instead. The make targets are ready (except for xulrunner in 1.9.2 - check comment 27)
* There are different scenarios that we want to start using the "make upload" (which call post_upload.py) targets:
** Maemo nightly and dep builds
** Maemo multi-locale build
** Desktop builds. I haven't looked into it but I bet it uses MozillaStage.
** WinMO builds. I haven't looked into it but I bet it uses MozillaStage.

For Maemo en-US single-locale we need bug 527928 to preserve the "en-US" directory which contains the binaries. This will be also needed for the "locale" sub-directories that we currently upload for their deb files.

Overall this bug helps us be in sync with Firefox's factories and will help us have to modify less code for Maemo's release factories (bug 525257).

Comment 29

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 411484 [details] [diff] [review]
[xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

a192=beltzner

Comment 30

[Armen [:armenzg]]

Comment on attachment 411484 [details] [diff] [review]
[xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

check-in needed for 1.9.2

Comment 31

[Reed Loden [:reed]]

Comment on attachment 411484 [details] [diff] [review]
[xulrunner] use UPLOAD_EXTRA_FILES to upload the deb files (no need for new update targets)

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/72daa308d3a6

Comment 32

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

With this now landed in 1.9.2, whats left to do here?

(Moving from Future, as Armen is actively working on this.)

Comment 33

[Aki Sasaki (not active)]

I imagine changing all of our mobile factories, including the release factories, to use make upload. + testing.

I'd argue that that is a sizeable chunk of work and we have a working solution, so I'd lower priority on this, but it's your call.

Comment 34

[Armen [:armenzg]]

For now until using "make upload" takes a higher priority in our list I will unassign this and leave it in future.

Comment 35

[Aki Sasaki (not active)]

Actually moving to future per comment 34.

Comment 36

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 37

[Aki Sasaki (not active)]

We should get this for free with bug 525327.

Comment 38

[Aki Sasaki (not active)]

Attachment: add deb_name.txt to UPLOAD_EXTRA_FILES

This bug is fixed by bug 525327, once this patch is applied.  (repacks and signing jobs use this to determine the deb name)

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 511669 [details] [diff] [review]
add deb_name.txt to UPLOAD_EXTRA_FILES

http://hg.mozilla.org/mobile-browser/rev/b94f83f03ff3

Comment 40

[Aki Sasaki (not active)]

Fixed in nightlies. Releases are tracked in bug 574764.