252941 - Don't pull all locales

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Benjamin Smedberg]

We're gonna have many locales in the CVS tree under toolkit/locales and
browser/locales and at 750k+ per locale, that adds up quickly. Don't pull any
locales except en-US by default (controlled by a mozconfig option).

Comment 1

[Benjamin Smedberg]

Attachment: allow mk_add_options MOZ_CO_LOCALES="en-US sl-SI"

This adds mk_add_options MOZ_CO_LOCALES="en-US sl-SI" or MOZ_CO_LOCALES=all
(default is en-US).

Comment 2

[Ben Goodger (use ben at mozilla dot org for email)]

Comment on attachment 154237 [details] [diff] [review]
allow mk_add_options MOZ_CO_LOCALES="en-US sl-SI"

a=[e-mail]

Comment 3

[Benjamin Smedberg]

Attachment: Make it work with the new multiple CVSROOTs

Note that fast-update won't work, and may not ever work. I just updated the
variables for sanity's sake.

Comment 4

[Benjamin Smedberg]

shaver, you're welcome to review this also, since we discussed the effects of
"set -e" earlier.

Comment 5

[Mike Shaver (:shaver emeritus)]

Comment on attachment 157433 [details] [diff] [review]
Make it work with the new multiple CVSROOTs

Looks OK to me, but bryner will be taking the hits for any errors, so let's
hear from him as well.

We've seen this work on windows and mac as well, right?

Comment 6

[Benjamin Smedberg]

Tested on unix/mac. I don't have a win32 box to test from here, but I'm not that
worried about windows. I'm more worried about oddball *nixes with stupid
"/bin/sh", but I'm not doing anything here that isn't done in other parts of the
mozilla build, so I think I'm ok.

Also, discovered an interesting edge-case when testing mac: if the SSH
connection fails (for instance, no private key installed), "cvs" fails with $? =
0. This means that we can't really detect errors. But that's no different than
the current behavior, so I'm not going to worry about it.

Comment 7

[Ben Goodger (use ben at mozilla dot org for email)]

Comment on attachment 157433 [details] [diff] [review]
Make it work with the new multiple CVSROOTs

a=[e-mail]

Comment 8

[Benjamin Smedberg]

Fixed, branch and trunk

Comment 9

[amano]

Will the source tarballs fetaure all locales?

Comment 10

[Benjamin Smedberg]

OK, this doesn't quite cut it. It pulls correctly, but when you update it
deletes/repulls the locale files, which will cause bad server/network load.
Patch coming up.

Comment 11

[Benjamin Smedberg]

Attachment: Rename locale directories during pull

This is kinda hacky. I tried making the locale dirs readonly, but that ran into
platform issues and CVS complained. So I just rename them and then un-rename
them.

Comment 12

[Andrew Schultz]

Comment on attachment 157433 [details] [diff] [review]
Make it work with the new multiple CVSROOTs

>+	@set -e;
>+	fast_update() { set -e; config/cvsco-fast-update.pl $$@ 2>&1 | tee -a $(CVSCO_LOGFILE); }; \

there needs to be a "\" after "@set -e;" so that it doesn't print out all the
commands.

Comment 13

[Benjamin Smedberg]

> there needs to be a "\" after "@set -e;" so that it doesn't print out all the
> commands.

(and so that the set -e actually works correctly). I checked in this simple fix
on branch and trunk.

Comment 14

[Axel Hecht]

(In reply to comment #10)
> OK, this doesn't quite cut it. It pulls correctly, but when you update it
> deletes/repulls the locale files, which will cause bad server/network load.
> Patch coming up.

And of course will drop all your changes.

I was wondering whether removing the locales from /cvsroot would clean stuff up.
I don't think that we ever need to get those 1.5 days alive again, right?

Comment 15

[Axel Hecht]

Comment on attachment 157906 [details] [diff] [review]
Rename locale directories during pull

I applied the patch to my tree and end up with a 
Syntax error: ";" unexpected

No idea where, though.
cygwin env, GNU Make 3.80

Comment 16

[Ben Goodger (use ben at mozilla dot org for email)]

bsmedberg says this work does not actually block PR, just l10n

Comment 17

[Axel Hecht]

Comment on attachment 157906 [details] [diff] [review]
Rename locale directories during pull

I found the culprit, my bash on linux was a tad more verbose than the cygwin
one.

>Index: client.mk
>===================================================================
>RCS file: /cvsroot/mozilla/client.mk,v
>retrieving revision 1.205.2.2.6.5
>diff -u -4 -r1.205.2.2.6.5 client.mk
>--- client.mk	1 Sep 2004 20:36:12 -0000	1.205.2.2.6.5
>+++ client.mk	4 Sep 2004 20:38:21 -0000
>@@ -488,14 +488,23 @@
>   LOCALES_CO_FLAGS := $(LOCALES_CO_FLAGS) -r $(LOCALES_CO_TAG)
> endif
> 
> ifndef MOZ_CO_LOCALES
>+LOCK_LOCALES := true
> FASTUPDATE_LOCALES := true
> CHECKOUT_LOCALES := true
>+UNLOCK_LOCALES := true
> else
> ifeq (all,$(MOZ_CO_LOCALES))
> MOZCONFIG_MODULES += $(addsuffix /all-locales,$(LOCALE_DIRS))
> 
>+LOCK_LOCALES := \
>+  for dir in $(LOCALE_DIRS); do \
>+    for locale in `cat $$dir/all-locales`; do \
>+      mv $$dir/$$locale $$dir/$$locale-tmp || true; \
>+    done; \
>+  done
>+
> FASTUPDATE_LOCALES := \
>   for dir in $(LOCALE_DIRS); do \
>     for locale in `cat $$dir/all-locales`; do \
>       fast_update $(CVS) $(CVS_FLAGS) -d $(LOCALES_CVSROOT) co $$dir/$$locale; \
>@@ -508,15 +517,23 @@
>       cvs_co $(CVS) $(CVS_FLAGS) -d $(LOCALES_CVSROOT) co $(LOCALES_CO_FLAGS) $$dir/$$locale; \
>     done; \
>   done
> 
>+UNLOCK_LOCALES := \
>+  for dir in $(LOCALE_DIRS); do \
>+    for locale in `cat $$dir/all-locales`; do \
>+      mv $$dir/$$locale-tmp $$dir/$$locale || true; \
>+    done; \
>+  done
> else
> LOCALE_CO_DIRS = $(foreach locale,$(MOZ_CO_LOCALES),$(addsuffix /$(locale),$(LOCALE_DIRS)))
> 
> CVSCO_LOCALES := $(CVS) $(CVS_FLAGS) -d $(LOCALES_CVSROOT) co $(LOCALES_CO_FLAGS) $(LOCALE_CO_DIRS)
> 
>+LOCK_LOCALES := for dir in $(LOCALE_CO_DIRS); do mv $$dir $$dir-tmp || true; done
> FASTUPDATE_LOCALES := fast_update $(CVSCO_LOCALES)
> CHECKOUT_LOCALES := cvs_co $(CVSCO_LOCALES)
>+LOCK_LOCALES := for dir in $(LOCALE_CO_DIRS); do mv $$dir-tmp $$dir || true; done

This wants to be UNLOCK_LOCALES

> endif
> endif #MOZ_CO_LOCALES
> 
> #######################################################################
>@@ -574,8 +591,9 @@
> real_checkout:
> 	@set -e; \
> 	cvs_co() { set -e; echo "$$@" ; \
> 	  "$$@" 2>&1 | tee -a $(CVSCO_LOGFILE); }; \
>+	$(LOCK_LOCALES); \
> 	$(CHECKOUT_STANDALONE); \
> 	$(CHECKOUT_STANDALONE_NOSUBDIRS); \
> 	cvs_co $(CVSCO_NSPR); \
> 	cvs_co $(CVSCO_NSS); \
>@@ -590,8 +608,9 @@
> 	$(CHECKOUT_PHOENIX); \
> 	$(CHECKOUT_THUNDERBIRD); \
> 	$(CHECKOUT_STANDALONE_COMPOSER); \
> 	$(CHECKOUT_CODESIGHS); \
>+	$(UNLOCK_LOCALES); \
> 	$(CHECKOUT_LOCALES); \
> 	cvs_co $(CVSCO_SEAMONKEY);
> 	@echo "checkout finish: "`date` | tee -a $(CVSCO_LOGFILE)
> # update the NSS checkout timestamp
>@@ -641,8 +660,9 @@
> 	  "$$@" 2>&1 | tee -a $(CVSCO_LOGFILE); }; \
> 	fast_update $(CVSCO_NSPR); \
> 	cd $(ROOTDIR); \
> 	cvs_co $(CVSCO_NSS); \
>+	$(LOCK_LOCALES); \
> 	cd mozilla; \
> 	fast_update $(CVSCO_PSM); \
> 	fast_update $(CVSCO_LDAPCSDK); \
> 	fast_update $(CVSCO_ACCESSIBLE); \
>@@ -654,8 +674,11 @@
> 	$(FASTUPDATE_PHOENIX); \
> 	$(FASTUPDATE_THUNDERBIRD); \
> 	$(FASTUPDATE_STANDALONE_COMPOSER); \
> 	$(FASTUPDATE_CODESIGHS); \
>+	cd $(ROOTDIR); \
>+	$(UNLOCK_LOCALES); \
>+	cd mozilla; \
> 	$(FASTUPDATE_LOCALES); \
> 	fast_update $(CVSCO_SEAMONKEY);
> 	@echo "fast_update finish: "`date` | tee -a $(CVSCO_LOGFILE)
> # update the NSS checkout timestamp

Anyway, from looking at the cvsco log, which still has a
?mozilla/browser/locales/de-DE-tmp,
clining the repository might be just as good.

Comment 18

[Axel Hecht]

Attachment: define UNLOCK_LOCALES for single locales, add $(CVS_CO_DATE_FLAGS)

Comment 19

[Hiro]

In the case of Mac, it is not completely localized only with a language package.
see bug172444

Comment 20

[Ben Goodger (use ben at mozilla dot org for email)]

Not a "blocker"

Comment 21

[Asa Dotzler [:asa]]

This bug seems to have an aviary branch checkin associated with it. If this has
landed on the aviary branch (as much as it's going to for 1.0) can you please
add the "fixed-aviary1.0" keyword? Thanks.

Comment 22

[Axel Hecht]

Attachment: the final bit, co locales by date

Comment 23

[Axel Hecht]

Checked in the last part, co didn't happen by date yet.
Marking fixed, with a semi-doesn't-apply-to-the-trunk and a
semi-checked-in-on-the-trunk