438436 - l10n - make files should reduce some steps to repackage a locale

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

Attachment: script that shows steps followed by the repackaging process in tinderbox

Steps to get a locale repackaged
1) make -f client.mk checkout
2) make -f client.mk configure
3) make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config && make -C $TOPSRCDIR/modules/libmar
4) wget -nv --output-document $ZIP_IN $LATEST_EN_US_BUILD 
5) mkdir -p $TOPSRCDIR/dist/install
6) make -C $TOPSRCDIR/browser/locales installers-$LOCALE ZIP_IN=$ZIP_IN

-----------------------
A problem that I run into in the buildbot configuration that I have is that I have to wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-*****3.1a1pre*****.en-US.mac.dmg but I do name as firefox-*****3.0pre*****.en-US.mac.dmg so "make installers-foo" could be called without specifying the $ZIP_IN value. "make installers-foo" looks under mozilla/dist

|dmg_name = "firefox-3.1a1pre.en-US.mac.dmg"
|#zipin_mac = "firefox.dmg"
|#QUESTION: Where is the line in the Makefile that decides that this is the |file needed?
|#Why didn't it take 3.1a1pre?? -->> More research
|zipin_mac = "firefox-3.0pre.en-US.mac.dmg"
|latest_mac_en_US ="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/" \
|                  + "latest-trunk/" + dmg_name
|
|command=["wget", "-nv", "--output-document", zipin_mac, latest_mac_en_US]

----------------------

What sounds ideal to me:

1) make -f client.mk checkout
2) make -f client.mk configure
3) make -C $TOPSRCDIR/browser/locales installers-$LOCALE

----------------------
Things that I would like to fix:
1) It should run "make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config && make -C $TOPSRCDIR/modules/libmar"
2) if I do a "make installers-foo", it should create the folder dist/install
3) If no ZIP_IN file is specified it should look for the .dmg file under dist (which currently does) and WGET the right file from ftp if it is not existent

Comment 1

[Armen [:armenzg]]

After rewriting a small script I have realized that dist/install is the first thing created:
|installers-%:
|	$(NSINSTALL) -D $(DIST)/install
|	@$(MAKE) langpack-$*

Some other discoveries:
* browser/locales/Makefile.in
#192 include $(topsrcdir)/toolkit/mozapps/installer/packager.mk
#274 repackage-zip-%: ZIP_IN=$(_ABS_DIST)/$(PACKAGE)
Trying to find where $(PACKAGE) was defined led me here:
* toolkit/mozapps/installer/packager.mk
#75 PACKAGE       = $(PKG_BASENAME)$(PKG_SUFFIX)
which lead me to:
* toolkit/mozapps/installer
#81 PKG_BASENAME = $(MOZ_PKG_APPNAME)-$(MOZ_PKG_VERSION).$(AB_CD).$(MOZ_PKG_PLATFORM)

NOTE: I will continue later

Comment 2

[Axel Hecht [:Pike]]

(In reply to comment #0)
> Created an attachment (id=324540) [details]
> script that shows steps followed by the repackaging process in tinderbox
> 
> Steps to get a locale repackaged
> 1) make -f client.mk checkout

You should try to get done with only l10n-checkout.

> 2) make -f client.mk configure

Use --disable-compile-environment to configure, too.

> 3) make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config && make -C
> $TOPSRCDIR/modules/libmar

Not exactly sure what those are needed for.

> 4) wget -nv --output-document $ZIP_IN $LATEST_EN_US_BUILD 
> 5) mkdir -p $TOPSRCDIR/dist/install
> 6) make -C $TOPSRCDIR/browser/locales installers-$LOCALE ZIP_IN=$ZIP_IN
> 
> -----------------------
> A problem that I run into in the buildbot configuration that I have is that I
> have to wget
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-*****3.1a1pre*****.en-US.mac.dmg
> but I do name as firefox-*****3.0pre*****.en-US.mac.dmg so "make
> installers-foo" could be called without specifying the $ZIP_IN value. "make
> installers-foo" looks under mozilla/dist
> 
> |dmg_name = "firefox-3.1a1pre.en-US.mac.dmg"
> |#zipin_mac = "firefox.dmg"
> |#QUESTION: Where is the line in the Makefile that decides that this is the
> |file needed?
> |#Why didn't it take 3.1a1pre?? -->> More research
> |zipin_mac = "firefox-3.0pre.en-US.mac.dmg"
> |latest_mac_en_US ="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/" \
> |                  + "latest-trunk/" + dmg_name

Please get the en-US builds from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/, that's where they need to be.

> |command=["wget", "-nv", "--output-document", zipin_mac, latest_mac_en_US]
> 
> ----------------------
> 
> What sounds ideal to me:
> 
> 1) make -f client.mk checkout
> 2) make -f client.mk configure
> 3) make -C $TOPSRCDIR/browser/locales installers-$LOCALE
> 
> ----------------------
> Things that I would like to fix:
> 1) It should run "make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config &&
> make -C $TOPSRCDIR/modules/libmar"
> 2) if I do a "make installers-foo", it should create the folder dist/install
> 3) If no ZIP_IN file is specified it should look for the .dmg file under dist
> (which currently does) and WGET the right file from ftp if it is not existent
> 

I'm not sure if I like the idea of coding up our ftp hosting setup in the regular makefiles, as annoying it is that you have to find out the right name and version and all for the installer files. We could probably add a hack rule to the makefile to output the full installer name without doing anything, or possibly re-use something that's elsewhere in the build ecosystem. Ben, do you have a step for that?

Comment 3

[Armen [:armenzg]]

* toolkit/mozapps/installer/package-name.mk
#43 MOZ_PKG_VERSION = $(MOZ_APP_VERSION)

* configure (I assume since multiple matching)
#13258 MOZ_APP_VERSION=$MOZILLA_VERSION
#5821  MOZILLA_VERSION=`$PERL $srcdir/config/milestone.pl -topsrcdir $srcdir`
Which throws 1.9pre, this means that my assumption was wrong

In fact it comes from: *browser/config/version.txt
and it might be through FIREFOX_VERSION instead of MOZILLA_VERSION but this doesn't matter to me anymore

I have found that I should have not been wgetting from latest-trunk but from latest-mozilla1.9.0

The string "latest-mozilla1.9.0" (where the latest en-US is placed) has to be defined somewhere else and therefore I do not know how the "make installers-foo" should know where to grab the latest en-US dmg file

------------
I have been trying to find out how after a build is completed, 1) where is the code that generates the package and 2) where is the code that uploads it to ftp.

Comment 4

[Armen [:armenzg]]

NOTE= The script added is by mimicking by following the flow of the tinderbox logs for l10n repackages. 

(In reply to comment #2)
> (In reply to comment #0)
> > Created an attachment (id=324540) [details] [details]
> > script that shows steps followed by the repackaging process in tinderbox
> > 
> > Steps to get a locale repackaged
> > 1) make -f client.mk checkout
> 
> You should try to get done with only l10n-checkout.
I am writing another much simpler script and I will test it with l10n-checkout

> > 3) make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config && make -C
> > $TOPSRCDIR/modules/libmar
> 
> Not exactly sure what those are needed for.
I believe that config is required to get nsinstall (This is used by "make installers-foo)
The other two, I will have to see what they are used for
> 
> > 4) wget -nv --output-document $ZIP_IN $LATEST_EN_US_BUILD 
> > 5) mkdir -p $TOPSRCDIR/dist/install
> > 6) make -C $TOPSRCDIR/browser/locales installers-$LOCALE ZIP_IN=$ZIP_IN
> > 
> > -----------------------
> > A problem that I run into in the buildbot configuration that I have is that I
> > have to wget
> > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-*****3.1a1pre*****.en-US.mac.dmg
> > but I do name as firefox-*****3.0pre*****.en-US.mac.dmg so "make
> > installers-foo" could be called without specifying the $ZIP_IN value. "make
> > installers-foo" looks under mozilla/dist
> > 
> > |dmg_name = "firefox-3.1a1pre.en-US.mac.dmg"
> > |#zipin_mac = "firefox.dmg"
> > |#QUESTION: Where is the line in the Makefile that decides that this is the
> > |file needed?
> > |#Why didn't it take 3.1a1pre?? -->> More research
> > |zipin_mac = "firefox-3.0pre.en-US.mac.dmg"
> > |latest_mac_en_US ="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/" \
> > |                  + "latest-trunk/" + dmg_name
> 
> Please get the en-US builds from
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/,
> that's where they need to be.
> 
> > |command=["wget", "-nv", "--output-document", zipin_mac, latest_mac_en_US]
> > 
> > ----------------------
> > 
> > What sounds ideal to me:
> > 
> > 1) make -f client.mk checkout
> > 2) make -f client.mk configure
> > 3) make -C $TOPSRCDIR/browser/locales installers-$LOCALE
> > 
> > ----------------------
> > Things that I would like to fix:
> > 1) It should run "make -C $TOPSRCDIR/nsprpub && make -C $TOPSRCDIR/config &&
> > make -C $TOPSRCDIR/modules/libmar"
> > 2) if I do a "make installers-foo", it should create the folder dist/install
> > 3) If no ZIP_IN file is specified it should look for the .dmg file under dist
> > (which currently does) and WGET the right file from ftp if it is not existent
> > 
> 
> I'm not sure if I like the idea of coding up our ftp hosting setup in the
> regular makefiles, as annoying it is that you have to find out the right name
> and version and all for the installer files. We could probably add a hack rule
> to the makefile to output the full installer name without doing anything, or
> possibly re-use something that's elsewhere in the build ecosystem. Ben, do you
> have a step for that?
> 
You are right in this, as I have realized from my previous comment. For now, a variable inside the master.cfg will do for me.

This information MIGHT be composed out of some information in bootstrap.cfg OR composed out of one of the tinder-config.pl files.

Comment 5

[Axel Hecht [:Pike]]

If you configure with --disable-compile-environment, configure will see that you don't want to compile and use the pyhthon nsinstall replacement in config/nsinstall.py. So no need to compile anything so far.

Comment 6

[Armen [:armenzg]]

Attachment: repackage-locale - it creates .xpi and .dmg of en-GB

1) I am not using --disable-compile-environment in this shot. Axel, which steps should change if that is added? (I will give myself a try)
2) l10n-checkout gets just good enough source code which is awesome
3) l10n-checkout does not checkout nsprpub, that is why I added it - nsprpub/configure is needed by the step make configure
4) A fresh run took 5m38secs - I am going to try it with "make checkout" instead of "make l10n-checkout" to see the improvement

Comment 7

[Armen [:armenzg]]

The process is reduced to:
1) cvs -d $CVS_ROOT co mozilla/client.mk mozilla/nsprpub
2) make -f mozilla/client.mk l10n-checkout
3) make -f mozilla/client.mk configure 
4) cvs -d $CVS_L10N_ROOT co l10n/en-GB
5) make -C mozilla/config
6) wget $LATEST_EN_US_BUILD
7) make -C mozilla/browser/locales installers-en-GB

The only improvements (if they are worth that much after all) that I see in the previous process are:
1) "installers-foo" would do step 5
2) l10n-checkout checks nsprpub 

Comment 8

[Armen [:armenzg]]

Using "make -f mozilla/client.mk checkout" instead of "make -f mozilla/client.mk checkout" gave me this time:
14m33.279s
instead of:
5m38secs

Is there any improvements that --disable-compile-environments would bring?

Comment 9

[Axel Hecht [:Pike]]

Now I remember, l10n is just broken. --disable-compile-environment not needing dependent configures was regressed, and checking out via the standard pserver is broken on windows. That cheers me up.

Armen, your comment 8 has two times for one command?

Comment 10

[Armen [:armenzg]]

(In reply to comment #9)
> Now I remember, l10n is just broken. --disable-compile-environment not needing
> dependent configures was regressed, and checking out via the standard pserver
> is broken on windows. That cheers me up.

I don't know what you mean but as long as you are cheered up will do :D

 
> Armen, your comment 8 has two times for one command?

My mistake:
"make -f mozilla/client.mk checkout" 14m33.279s
"make -f mozilla/client.mk l10n-checkout"  5m38secs

Comment 11

[Armen [:armenzg]]

(In reply to comment #7)
> 1) "installers-foo" would do step 5
This makes no sense, "make -C config" is part of the common code
> 2) l10n-checkout checks out nsprpub 
This is the only thing that makes sense

> 4) cvs -d $CVS_L10N_ROOT co l10n/en-GB
> 5) make -C mozilla/config
> 6) wget $LATEST_EN_US_BUILD
5 and 6 should happen before checking out an specific locale and after that consider the following as "locale-specific"

Common code steps:
 4) make -C mozilla/config
 5) wget $LATEST_EN_US_BUILD
"locale-specific" steps:
 6) cvs -d $CVS_L10N_ROOT co l10n/en-GB
 7) make -C mozilla/browser/locales installers-en-GB

Comment 12

[Axel Hecht [:Pike]]

l10n doesn't really need nspr, and IMHO, we should try to fix that again. Neither did we need to make config.

Comment 13

[Armen [:armenzg]]

(In reply to comment #12)
> l10n doesn't really need nspr, and IMHO, we should try to fix that again.
I do not know what nsprpub is for. I just know that I could now go further w/o checking it out
> Neither did we need to make config.
> 
I could not run make installers w/o doing make config first

Any pieces of code that you want to point at so I could fix these things?

For now I am continuing with the other two bugs and since this bug blocks bug 434878, I should not be able to go w/o fixing this one aswell

Comment 14

[Axel Hecht [:Pike]]

Filed bug 440417 on the current issues with client.mk's l10n-checkout.

That was simpler than I thought. If Ted gives me an r+, that is :-)

Comment 15

[Armen [:armenzg]]

Can I close this bug?

Comment 16

[Axel Hecht [:Pike]]

I'm fine with closing this, I'm filed bug 458014 with some refactorings in browser/locales/Makefile.in