451088 - master.cfg changes to run L10n "NIGHTLY" repackages in *parallel* in staging1.9 and in production1.9

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

Attachment: staging-1.9 configuration to run L10n repackages

This bug will contain the patches that change the master.cfg for staging1.9 and production1.9 to run L10n repackages in parallel

Comment 1

[Armen [:armenzg]]

Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages

>+L10nBuildFactory.addStep(ShellCommand(
>+    command=["cvs", "-q", "-z3", "-d", cvsStagingRoot, "co",
>+             "mozilla/client.mk",
>+             "mozilla/browser/locales",
>+             "mozilla/nsprpub",
>+             # We need this for windows
>+             "mozilla/other-licenses/7zstub/firefox"],
>+    descriptionDone="client.mk checked out",
>
This is not appropriate because it does more than checking out client.mk which is misleading

>+L10nBuildFactory.addStep(ShellCommand,
>+    command=["make", "-f", "client.mk", "l10n-checkout"],
>
I should write a make target that will just check out the code needed to repackage or get the missing that I added in the previous comment (mozilla/nsprpub, mozilla/other-licenses)
Axel, is the l10n-checkout target the target to checkout the necessary code to run a compare-locale step?

>+#Since we are wgetting from ftp.m.o but using code from staging
>+L10nBuildFactory.addStep(ShellCommand(
>+    command=["cvs", "-q", "-z3", "-d", cvsroot, "co",
>+             "mozilla/browser/config/version.txt"],
>+    descriptionDone="version.txt",
>
This step will go away for the production configuration

>+L10nBuildFactory.addStep(ShellCommand,
>+    command=["make", "-f", "client.mk", "configure"],
>
I would really love to have another configure file to run that would do the minimum verification for just l10n-repackages. Just for the sake of improving the performance even it is few seconds

You can look at a log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1219095408.1219095453.17772.gz&fulltext=1
and see that you have a lot of messages like: "can't read blah blah"

>+L10nBuildFactory.addStep(ShellCommand,
>+    name = "upload locale",
>+# This will upload everything on dist/upload, I assume that the first 
>+# step run in this build is "rm -rf mozilla/dist/upload"
>+    command=['sh','-c','scp -r * '+ftpserver+":"+uploadPath],
>+    description="uploading packages", 
>+    descriptionDone="uploaded packages",
>+    workdir="mozilla/dist/upload"
>+)
>
maybe another make target for this?

Comment 2

[Armen [:armenzg]]

Attachment: master.cfg for production 1.9 as the previous one

This configurations has an extra element which is an extra TinderboxMailNotifier for the Mozilla-l10n page

This patch also adds as the previous one, it adds the "mozconfig-l10n" file

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 334375 [details] [diff] [review]
master.cfg for production 1.9 as the previous one

let's wait until staging configs are checked in before tackling this

Comment 4

[Axel Hecht [:Pike]]

Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages

r-.

There are different reasons for this, first and foremost, this bug should depend on a bug to actually implement buildbotcustom.l10n.

I don't think we need the nsprpub checkout. Other extra parts should probably go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout.

The locks are wrong.

This bug should depend on the make targets, too.

The blocking bugs are likely somewhere in the dependency tree, but we should have the dependencies between the bugs set up right in order to attach this beast.

Comment 5

[Benjamin Smedberg]

FWIW, I think this is a bad idea in general, as I've noted on IRC a couple times: doing l10n repacks in parallel means you have to replicate work across a lot of builds. It looks like, from http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current repacks take about 30 minutes for all locales. Why are we trying to improve on this number, when it will significantly slow down the rest of the system?

Comment 6

[Armen [:armenzg]]

(In reply to comment #5)
> FWIW, I think this is a bad idea in general, as I've noted on IRC a couple
> times: doing l10n repacks in parallel means you have to replicate work across a
> lot of builds. It looks like, from
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current
> repacks take about 30 minutes for all locales. Why are we trying to improve on
> this number, when it will significantly slow down the rest of the system?
One main reason:
1) Windows repackages take too long - 4hours and 20 minutes -> WINNT 5.2 l10n-win32-tbox Depend Fx-Moz1.9.0-l10n- Started 06:02, finished 10:22

Comment 7

[Armen [:armenzg]]

This time will increase in the long run since we keep on adding more and more locales to our system. Being scalable will help us.
It sucks to have logic being repeated, but it allow us to throw more slaves in if we want to improve the performance

In staging, I have been running this code and with 3 windows slaves the execution time was less than an hour and a half when the work being done by a single slave was more than 3 hours and a half

Comment 8

[Benjamin Smedberg]

Do we know why? I have found that in my windows slave VM, a repack without running compare-locales takes 6 minutes per locale. Almost all of this time (60%) is spent in the upload step. Does this match up with your timings?

Comment 9

[Armen [:armenzg]]

(In reply to comment #8)
> Do we know why? I have found that in my windows slave VM, a repack without
> running compare-locales takes 6 minutes per locale. Almost all of this time
> (60%) is spent in the upload step. Does this match up with your timings?
I am not sure why it is taking that long but my timings where under 3 minutes on windows and around 30 secs on the xserve but I can't remember right now.
BTW, I used "make -f client.mk l10n-checkout" to checkout less code
This is an old blog post that I did in some early staging runs:
http://armenzg.blogspot.com/2008/08/l10n-parallelization-results.html

There are things that could be improved at a later point like checking out less code and have a shorter configure step

Comment 10

[Armen [:armenzg]]

(In reply to comment #4)
> There are different reasons for this, first and foremost, this bug should
> depend on a bug to actually implement buildbotcustom.l10n.
> 
I have been fixing the list of dependent bugs
bug 451056 for the buildbotcsutom.l10n implementation

> I don't think we need the nsprpub checkout. Other extra parts should probably
> go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout.
> 
I will play with this again
I will file a bug about nsprpub when I find why was the reason that I was checking out and find a way of not having to check it out

> The locks are wrong.
True
> 
> This bug should depend on the make targets, too.
> 
I have added the dependence to the three make targets

> The blocking bugs are likely somewhere in the dependency tree, but we should
> have the dependencies between the bugs set up right in order to attach this
> beast.
:( I wish I had a better leash for this creature
Added dependency as mentioned before

Comment 11

[Axel Hecht [:Pike]]

So here's my take on the timings: We currently have a smaller list of languages on central, i.e. 46 instead of 63. Then you're not running compare-locales right now.

I assume your machine is beafier, too, but when we have round about 2 minutes per locale, 60 locales, we get 2 hours, and growing.

We're currently facing a few issues that make the parallization cost not per slave, but per build, but those can be fixed. Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around. I wonder if we can make the wget target only fetch the file if it's newer, too. The l10n checkout is not that bad, as soon as you limit the check-outs to the locale you're building, that's scaling pretty well.

Comment 12

[Armen [:armenzg]]

(In reply to comment #11)
> Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around.
What do you mean?

> I wonder if we can make the wget target only fetch the file if it's newer, too.
I have thought of this and I think it can be done

Comment 13

[Axel Hecht [:Pike]]

(In reply to comment #12)
> (In reply to comment #11)
> > Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around.
> What do you mean?

Sry, that was mercurial specific. I think the corresponding ticket on bb is http://buildbot.net/trac/ticket/277.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages

I really don't have time to review this stuff right now - sorry. Assuming this patch is still valid, I'm passing to coop.

Comment 15

[Chris Cooper [:coop] (he/him)]

Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages

I think this review is probably out-of-date by now. Most of the dependency issues that Axel brought up seem to have been filed at least, if not yet fixed.

As Axel noted, the locks are wrong. Looks like you've cut-n-pasted the linux_lock for all platforms.

Comment 16

[Armen [:armenzg]]

As coop mentions this master.cfg has not been updated and most of the issues that Axel raised are being landed by coop

This master.cfg has been in use during the first 2 weeks of August with correct results but a cleaner master.cfg patch will be posted as soon as I have the system running again in staging-1.9

Comment 17

[Armen [:armenzg]]

Attachment: [checked in] staging-1.9 configuration to run L10n repackages

This configuration file is working right now in staging-1.9 correctly
This is ready to be reviewed and commited when review process goes trough

It should be reporting to MozillaTest and MozillaTest-%locale% as the emails pasted at the bottom show but for an unknown reason is not showing in http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

While rerunning all of this, we should be filing as Axel mentions a bug for "mozilla/other-licenses/7zstub/firefox" to be included in l10n-checkout which should be an easy fix and another to find out why do I need to check out "nsprpub" to make-installers (if I remove the line it won't generate the installers)

-----------------------------------------------------
tinderbox: tree: MozillaTest
tinderbox: builddate: 1222047002
tinderbox: status: busted
tinderbox: build: win32_l10n_nightly
tinderbox: errorparser: unix
tinderbox: binaryurl: 
tinderbox: logcompression: bzip2
tinderbox: logencoding: base64
tinderbox: END

QlpoOTFBWSZTWepo/UkAAGNfgAAQQIGAEAwAQAB/L98wIACJCSo9EAepoGgARUTIE9TCaGjNGK9r
Cgy0YN0x3GgKpY6whBYBMHcCYbhmmDeOQzJJV4f7Bs8u7nslToMem449N0woRgFFyeQhqkGVPgaX
hnN2ItiUAkzeQwcWSimZDcYXr74uexPJUh+LuSKcKEh1NH6kgA==
-----------------------------------------------------
tinderbox: tree: MozillaTest-ar
tinderbox: builddate: 1222047002
tinderbox: status: busted
tinderbox: build: win32_l10n_nightly
tinderbox: errorparser: unix
tinderbox: binaryurl: http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/
tinderbox: logcompression: bzip2
tinderbox: logencoding: base64
tinderbox: END

QlpoOTFBWSZTWaqUo5sAAKLfgAAQQIP8MAQBQAD/59/wMADmEJRTSeKeQIaMaTajTQ0GMmmQMmhk
GRpgRgkkmEmmjQZGmmI0HqaZb4Kq2iKY0AyZqUmzhzBgak35zhfqY74cSNkYnuzo26VY5se22ZzM
5VSpFLqktQ68KEgFfpBe7H0BAMfp9DHkQhsStqHM8TTTFTs0pmDlHeey7knjbGpXTmNk88jXA41D
UibbDeHzAnaELBEuwWo6L8rXPL5BnQDnm2C63L8V3hSMEChEA14pTBqyMqUOWS3EEXUczORGRLRl
FuHdp5bkaSOINei14QUqW/JCTw/i7kinChIVUpRzYA==

Comment 18

[Armen [:armenzg]]

You can now see the results of the l10n repackages in parallel on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

Comment 19

[Armen [:armenzg]]

Attachment: production-1.9 configuration to run L10n repackages

This patch as not been tried in production-1.9 but it is basically the same as what is being run in staging-1.9

Comment 20

[Chris Cooper [:coop] (he/him)]

Comment on attachment 339721 [details] [diff] [review]
[checked in] staging-1.9 configuration to run L10n repackages   


> ####### BUILDERS
> 
> cvsroot = ":ext:[e-mail]:/cvsroot"
>+cvsStagingRoot = 'staging-1.9-master:/builds/cvsmirror/cvsroot'
>+cvsl10nroot = 'staging-1.9-master:/builds/cvsmirror/l10n'
> cvsmodule = "mozilla/tools/release"
> ftpserver = "fx-linux-1.9-slave1"
> automation_tag = "HEAD"
>+uploadPath = "/home/ftp/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/"
>+enUS_binaryURL = 'http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0'

Shouldn't this be pointing at firefox/experimental/nightly/ for staging builds?

Comment 21

[Chris Cooper [:coop] (he/him)]

Comment on attachment 339802 [details] [diff] [review]
production-1.9 configuration to run L10n repackages   

Every third l10n build or so on the MozillaTest tree seems to be red right now. Is that expected?

Comment 22

[Armen [:armenzg]]

(In reply to comment #21)
> (From update of attachment 339802 [details] [diff] [review])
> Every third l10n build or so on the MozillaTest tree seems to be red right now.
> Is that expected?
>
Yes, it is expected. Not all l10n repos are in the staging cvs mirror and not every locale succeeds since they might have missing entities or files

Comment 23

[Armen [:armenzg]]

Attachment: [checked in] production-1.9 configuration that will report to MozillaStaging and push to experimental

Hey coop, I didn't know what was the best way to write this patch

Once we decide to do the full switch I will write a patch to don't point to experimental and to MozillaStaging

I left a keyword #CHANGE to know exactly which lines to modify next time; is that fine?

Comment 24

[Chris Cooper [:coop] (he/him)]

Comment on attachment 339855 [details] [diff] [review]
[checked in] production-1.9 configuration that will report to MozillaStaging and push to experimental

(In reply to comment #23)
> I left a keyword #CHANGE to know exactly which lines to modify next time; is
> that fine?

If you and bhearsum have already had the conversation about where these builds should be directed to, that's fine by me.

Comment 25

[Chris Cooper [:coop] (he/him)]

Armen: I've checked in the production master.cfg patch. Are we really still waiting for a review on the staging patch?

Comment 26

[Armen [:armenzg]]

(In reply to comment #25)
> Armen: I've checked in the production master.cfg patch. Are we really still
> waiting for a review on the staging patch?
>
No, go ahead whenever you can

Comment 27

[Armen [:armenzg]]

Attachment: [checked in] fixing some typos and correct paths

This patch fixes some missed typos, correct the experimental ftp path, use correct make target and report to MozillaTest instead of Mozilla-l10n