Closed
Bug 451088
Opened 16 years ago
Closed 16 years ago
master.cfg changes to run L10n "NIGHTLY" repackages in *parallel* in staging1.9 and in production1.9
Categories
(Release Engineering :: General, defect, P2)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: armenzg, Assigned: armenzg)
References
Details
Attachments
(3 files, 3 obsolete files)
10.63 KB,
patch
|
coop
:
review+
|
Details | Diff | Splinter Review |
10.23 KB,
patch
|
coop
:
review+
|
Details | Diff | Splinter Review |
3.55 KB,
patch
|
coop
:
review+
|
Details | Diff | Splinter Review |
This bug will contain the patches that change the master.cfg for staging1.9 and production1.9 to run L10n repackages in parallel
Attachment #334327 -
Flags: review?(bhearsum)
Assignee | ||
Updated•16 years ago
|
Summary: code to run L10n repackages in staging1.9 and in production1.9 → master.cfg changes to run L10n repackages in staging1.9 and in production1.9
Assignee | ||
Updated•16 years ago
|
Attachment #334327 -
Flags: review?(l10n)
Assignee | ||
Comment 1•16 years ago
|
||
Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages
>+L10nBuildFactory.addStep(ShellCommand(
>+ command=["cvs", "-q", "-z3", "-d", cvsStagingRoot, "co",
>+ "mozilla/client.mk",
>+ "mozilla/browser/locales",
>+ "mozilla/nsprpub",
>+ # We need this for windows
>+ "mozilla/other-licenses/7zstub/firefox"],
>+ descriptionDone="client.mk checked out",
>
This is not appropriate because it does more than checking out client.mk which is misleading
>+L10nBuildFactory.addStep(ShellCommand,
>+ command=["make", "-f", "client.mk", "l10n-checkout"],
>
I should write a make target that will just check out the code needed to repackage or get the missing that I added in the previous comment (mozilla/nsprpub, mozilla/other-licenses)
Axel, is the l10n-checkout target the target to checkout the necessary code to run a compare-locale step?
>+#Since we are wgetting from ftp.m.o but using code from staging
>+L10nBuildFactory.addStep(ShellCommand(
>+ command=["cvs", "-q", "-z3", "-d", cvsroot, "co",
>+ "mozilla/browser/config/version.txt"],
>+ descriptionDone="version.txt",
>
This step will go away for the production configuration
>+L10nBuildFactory.addStep(ShellCommand,
>+ command=["make", "-f", "client.mk", "configure"],
>
I would really love to have another configure file to run that would do the minimum verification for just l10n-repackages. Just for the sake of improving the performance even it is few seconds
You can look at a log:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1219095408.1219095453.17772.gz&fulltext=1
and see that you have a lot of messages like: "can't read blah blah"
>+L10nBuildFactory.addStep(ShellCommand,
>+ name = "upload locale",
>+# This will upload everything on dist/upload, I assume that the first
>+# step run in this build is "rm -rf mozilla/dist/upload"
>+ command=['sh','-c','scp -r * '+ftpserver+":"+uploadPath],
>+ description="uploading packages",
>+ descriptionDone="uploaded packages",
>+ workdir="mozilla/dist/upload"
>+)
>
maybe another make target for this?
Assignee | ||
Comment 2•16 years ago
|
||
This configurations has an extra element which is an extra TinderboxMailNotifier for the Mozilla-l10n page
This patch also adds as the previous one, it adds the "mozconfig-l10n" file
Attachment #334375 -
Flags: review?(bhearsum)
Comment 3•16 years ago
|
||
Comment on attachment 334375 [details] [diff] [review]
master.cfg for production 1.9 as the previous one
let's wait until staging configs are checked in before tackling this
Attachment #334375 -
Flags: review?(bhearsum)
Assignee | ||
Updated•16 years ago
|
Summary: master.cfg changes to run L10n repackages in staging1.9 and in production1.9 → master.cfg changes to run L10n "NIGHTLY" repackages in staging1.9 and in production1.9
Assignee | ||
Updated•16 years ago
|
Summary: master.cfg changes to run L10n "NIGHTLY" repackages in staging1.9 and in production1.9 → master.cfg changes to run L10n "NIGHTLY" repackages in *parallel* in staging1.9 and in production1.9
Assignee | ||
Updated•16 years ago
|
Assignee: nobody → armenzg
Priority: -- → P2
Updated•16 years ago
|
Attachment #334327 -
Flags: review?(l10n) → review-
Comment 4•16 years ago
|
||
Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages
r-.
There are different reasons for this, first and foremost, this bug should depend on a bug to actually implement buildbotcustom.l10n.
I don't think we need the nsprpub checkout. Other extra parts should probably go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout.
The locks are wrong.
This bug should depend on the make targets, too.
The blocking bugs are likely somewhere in the dependency tree, but we should have the dependencies between the bugs set up right in order to attach this beast.
Assignee | ||
Updated•16 years ago
|
Comment 5•16 years ago
|
||
FWIW, I think this is a bad idea in general, as I've noted on IRC a couple times: doing l10n repacks in parallel means you have to replicate work across a lot of builds. It looks like, from http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current repacks take about 30 minutes for all locales. Why are we trying to improve on this number, when it will significantly slow down the rest of the system?
Assignee | ||
Comment 6•16 years ago
|
||
(In reply to comment #5)
> FWIW, I think this is a bad idea in general, as I've noted on IRC a couple
> times: doing l10n repacks in parallel means you have to replicate work across a
> lot of builds. It looks like, from
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla-l10n that the current
> repacks take about 30 minutes for all locales. Why are we trying to improve on
> this number, when it will significantly slow down the rest of the system?
One main reason:
1) Windows repackages take too long - 4hours and 20 minutes -> WINNT 5.2 l10n-win32-tbox Depend Fx-Moz1.9.0-l10n- Started 06:02, finished 10:22
Assignee | ||
Comment 7•16 years ago
|
||
This time will increase in the long run since we keep on adding more and more locales to our system. Being scalable will help us.
It sucks to have logic being repeated, but it allow us to throw more slaves in if we want to improve the performance
In staging, I have been running this code and with 3 windows slaves the execution time was less than an hour and a half when the work being done by a single slave was more than 3 hours and a half
Comment 8•16 years ago
|
||
Do we know why? I have found that in my windows slave VM, a repack without running compare-locales takes 6 minutes per locale. Almost all of this time (60%) is spent in the upload step. Does this match up with your timings?
Assignee | ||
Comment 9•16 years ago
|
||
(In reply to comment #8)
> Do we know why? I have found that in my windows slave VM, a repack without
> running compare-locales takes 6 minutes per locale. Almost all of this time
> (60%) is spent in the upload step. Does this match up with your timings?
I am not sure why it is taking that long but my timings where under 3 minutes on windows and around 30 secs on the xserve but I can't remember right now.
BTW, I used "make -f client.mk l10n-checkout" to checkout less code
This is an old blog post that I did in some early staging runs:
http://armenzg.blogspot.com/2008/08/l10n-parallelization-results.html
There are things that could be improved at a later point like checking out less code and have a shorter configure step
Assignee | ||
Comment 10•16 years ago
|
||
(In reply to comment #4)
> There are different reasons for this, first and foremost, this bug should
> depend on a bug to actually implement buildbotcustom.l10n.
>
I have been fixing the list of dependent bugs
bug 451056 for the buildbotcsutom.l10n implementation
> I don't think we need the nsprpub checkout. Other extra parts should probably
> go into mozconfig, for MOZ_CO_MODULES, or be a bug on l10n-checkout.
>
I will play with this again
I will file a bug about nsprpub when I find why was the reason that I was checking out and find a way of not having to check it out
> The locks are wrong.
True
>
> This bug should depend on the make targets, too.
>
I have added the dependence to the three make targets
> The blocking bugs are likely somewhere in the dependency tree, but we should
> have the dependencies between the bugs set up right in order to attach this
> beast.
:( I wish I had a better leash for this creature
Added dependency as mentioned before
Comment 11•16 years ago
|
||
So here's my take on the timings: We currently have a smaller list of languages on central, i.e. 46 instead of 63. Then you're not running compare-locales right now.
I assume your machine is beafier, too, but when we have round about 2 minutes per locale, 60 locales, we get 2 hours, and growing.
We're currently facing a few issues that make the parallization cost not per slave, but per build, but those can be fixed. Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around. I wonder if we can make the wget target only fetch the file if it's newer, too. The l10n checkout is not that bad, as soon as you limit the check-outs to the locale you're building, that's scaling pretty well.
Assignee | ||
Comment 12•16 years ago
|
||
(In reply to comment #11)
> Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around.
What do you mean?
> I wonder if we can make the wget target only fetch the file if it's newer, too.
I have thought of this and I think it can be done
Comment 13•16 years ago
|
||
(In reply to comment #12)
> (In reply to comment #11)
> > Clobbering en-US on checkout was one, which hopefully should be fixed in bb 0.7.9 and can be worked around.
> What do you mean?
Sry, that was mercurial specific. I think the corresponding ticket on bb is http://buildbot.net/trac/ticket/277.
Comment 14•16 years ago
|
||
Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages
I really don't have time to review this stuff right now - sorry. Assuming this patch is still valid, I'm passing to coop.
Attachment #334327 -
Flags: review?(bhearsum) → review?(ccooper)
Comment 15•16 years ago
|
||
Comment on attachment 334327 [details] [diff] [review]
staging-1.9 configuration to run L10n repackages
I think this review is probably out-of-date by now. Most of the dependency issues that Axel brought up seem to have been filed at least, if not yet fixed.
As Axel noted, the locks are wrong. Looks like you've cut-n-pasted the linux_lock for all platforms.
Attachment #334327 -
Flags: review?(ccooper) → review-
Assignee | ||
Comment 16•16 years ago
|
||
As coop mentions this master.cfg has not been updated and most of the issues that Axel raised are being landed by coop
This master.cfg has been in use during the first 2 weeks of August with correct results but a cleaner master.cfg patch will be posted as soon as I have the system running again in staging-1.9
Assignee | ||
Comment 17•16 years ago
|
||
This configuration file is working right now in staging-1.9 correctly
This is ready to be reviewed and commited when review process goes trough
It should be reporting to MozillaTest and MozillaTest-%locale% as the emails pasted at the bottom show but for an unknown reason is not showing in http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest
While rerunning all of this, we should be filing as Axel mentions a bug for "mozilla/other-licenses/7zstub/firefox" to be included in l10n-checkout which should be an easy fix and another to find out why do I need to check out "nsprpub" to make-installers (if I remove the line it won't generate the installers)
-----------------------------------------------------
tinderbox: tree: MozillaTest
tinderbox: builddate: 1222047002
tinderbox: status: busted
tinderbox: build: win32_l10n_nightly
tinderbox: errorparser: unix
tinderbox: binaryurl:
tinderbox: logcompression: bzip2
tinderbox: logencoding: base64
tinderbox: END
QlpoOTFBWSZTWepo/UkAAGNfgAAQQIGAEAwAQAB/L98wIACJCSo9EAepoGgARUTIE9TCaGjNGK9r
Cgy0YN0x3GgKpY6whBYBMHcCYbhmmDeOQzJJV4f7Bs8u7nslToMem449N0woRgFFyeQhqkGVPgaX
hnN2ItiUAkzeQwcWSimZDcYXr74uexPJUh+LuSKcKEh1NH6kgA==
-----------------------------------------------------
tinderbox: tree: MozillaTest-ar
tinderbox: builddate: 1222047002
tinderbox: status: busted
tinderbox: build: win32_l10n_nightly
tinderbox: errorparser: unix
tinderbox: binaryurl: http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/
tinderbox: logcompression: bzip2
tinderbox: logencoding: base64
tinderbox: END
QlpoOTFBWSZTWaqUo5sAAKLfgAAQQIP8MAQBQAD/59/wMADmEJRTSeKeQIaMaTajTQ0GMmmQMmhk
GRpgRgkkmEmmjQZGmmI0HqaZb4Kq2iKY0AyZqUmzhzBgak35zhfqY74cSNkYnuzo26VY5se22ZzM
5VSpFLqktQ68KEgFfpBe7H0BAMfp9DHkQhsStqHM8TTTFTs0pmDlHeey7knjbGpXTmNk88jXA41D
UibbDeHzAnaELBEuwWo6L8rXPL5BnQDnm2C63L8V3hSMEChEA14pTBqyMqUOWS3EEXUczORGRLRl
FuHdp5bkaSOINei14QUqW/JCTw/i7kinChIVUpRzYA==
Attachment #334327 -
Attachment is obsolete: true
Attachment #339721 -
Flags: review?(l10n)
Attachment #339721 -
Flags: review?(ccooper)
Assignee | ||
Comment 18•16 years ago
|
||
You can now see the results of the l10n repackages in parallel on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest
Assignee | ||
Comment 19•16 years ago
|
||
This patch as not been tried in production-1.9 but it is basically the same as what is being run in staging-1.9
Attachment #334375 -
Attachment is obsolete: true
Attachment #339802 -
Flags: review?(ccooper)
Comment 20•16 years ago
|
||
Comment on attachment 339721 [details] [diff] [review]
[checked in] staging-1.9 configuration to run L10n repackages
> ####### BUILDERS
>
> cvsroot = ":ext:stgbld@cvs.mozilla.org:/cvsroot"
>+cvsStagingRoot = 'staging-1.9-master:/builds/cvsmirror/cvsroot'
>+cvsl10nroot = 'staging-1.9-master:/builds/cvsmirror/l10n'
> cvsmodule = "mozilla/tools/release"
> ftpserver = "fx-linux-1.9-slave1"
> automation_tag = "HEAD"
>+uploadPath = "/home/ftp/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/"
>+enUS_binaryURL = 'http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0'
Shouldn't this be pointing at firefox/experimental/nightly/ for staging builds?
Comment 21•16 years ago
|
||
Comment on attachment 339802 [details] [diff] [review]
production-1.9 configuration to run L10n repackages
Every third l10n build or so on the MozillaTest tree seems to be red right now. Is that expected?
Assignee | ||
Comment 22•16 years ago
|
||
(In reply to comment #21)
> (From update of attachment 339802 [details] [diff] [review])
> Every third l10n build or so on the MozillaTest tree seems to be red right now.
> Is that expected?
>
Yes, it is expected. Not all l10n repos are in the staging cvs mirror and not every locale succeeds since they might have missing entities or files
Assignee | ||
Comment 23•16 years ago
|
||
Hey coop, I didn't know what was the best way to write this patch
Once we decide to do the full switch I will write a patch to don't point to experimental and to MozillaStaging
I left a keyword #CHANGE to know exactly which lines to modify next time; is that fine?
Attachment #339855 -
Flags: review?(ccooper)
Assignee | ||
Updated•16 years ago
|
Attachment #339802 -
Flags: review?(ccooper)
Comment 24•16 years ago
|
||
Comment on attachment 339855 [details] [diff] [review]
[checked in] production-1.9 configuration that will report to MozillaStaging and push to experimental
(In reply to comment #23)
> I left a keyword #CHANGE to know exactly which lines to modify next time; is
> that fine?
If you and bhearsum have already had the conversation about where these builds should be directed to, that's fine by me.
Attachment #339855 -
Flags: review?(ccooper) → review+
Updated•16 years ago
|
Attachment #339802 -
Attachment is obsolete: true
Updated•16 years ago
|
Attachment #339721 -
Flags: review?(ccooper) → review+
Assignee | ||
Updated•16 years ago
|
Keywords: checkin-needed
Updated•16 years ago
|
Attachment #339855 -
Attachment description: production-1.9 configuration that will report to MozillaStaging and push to experimental → [checked in] production-1.9 configuration that will report to MozillaStaging and push to experimental
Comment 25•16 years ago
|
||
Armen: I've checked in the production master.cfg patch. Are we really still waiting for a review on the staging patch?
Assignee | ||
Updated•16 years ago
|
Attachment #339721 -
Flags: review?(l10n)
Assignee | ||
Comment 26•16 years ago
|
||
(In reply to comment #25)
> Armen: I've checked in the production master.cfg patch. Are we really still
> waiting for a review on the staging patch?
>
No, go ahead whenever you can
Updated•16 years ago
|
Attachment #339721 -
Attachment description: staging-1.9 configuration to run L10n repackages → [checked in] staging-1.9 configuration to run L10n repackages
Updated•16 years ago
|
Assignee | ||
Comment 27•16 years ago
|
||
This patch fixes some missed typos, correct the experimental ftp path, use correct make target and report to MozillaTest instead of Mozilla-l10n
Attachment #340193 -
Flags: review?(ccooper)
Assignee | ||
Updated•16 years ago
|
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated•16 years ago
|
Attachment #340193 -
Attachment description: fixing some typos and correct paths → [checked in] fixing some typos and correct paths
Attachment #340193 -
Flags: review?(ccooper) → review+
Updated•16 years ago
|
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
You need to log in
before you can comment on or make changes to this bug.
Description
•