445254 - migrate firefox 1.9 L10n nightlies to release automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

l10n-linux-tbox, bm-xserve12 and l10n-win32-tbox generate L10n nightly repackages sometime around 9 am PDT every day and tinderbox gets called with --interval 1800

We would like to this work under generic build slaves machines in production-1.9-master.

This requires initially to be able to do this in staging-1.9-master by updating tinder-config.pl in the "l10n" branch of it.

NOTE: These machines also do Tb-Trunk-L10n, therefore, these machines won't be able to be shut down even if we move firefox l10n repackages to buildbot

Comment 1

[Armen [:armenzg]]

Attachment: linux/tinder-config.pl ("l10n" branch)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 329587 [details] [diff] [review]
linux/tinder-config.pl ("l10n" branch)

>Index: mozconfig
>===================================================================
>RCS file: /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/mozconfig,v
>retrieving revision 1.2.6.8
...
> mk_add_options MOZ_CO_LOCALES=all
>-mk_add_options LOCALES_CVSROOT=:ext:[e-mail]:/l10n
>+mk_add_options LOCALES_CVSROOT=[e-mail]:/l10n

You should use a CONFIG here too, i.e
# CONFIG: mk_add_options LOCALES_CVSROOT=%l10nCvsroot%

>Index: tinder-config.pl
>===================================================================
>RCS file: /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v
>retrieving revision 1.8.2.16
...
>-$moz_cvsroot   = ":ext:ffxbld\@cvs.mozilla.org:/cvsroot";
>+$moz_cvsroot   = '[e-mail]:/cvsroot'; 

Why not use:
# CONFIG: $moz_cvsroot   = '%mozillaCvsroot%';
 
> %WGetFiles = (
> 	      "http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/firefox-%version%.en-US.mac.dmg" =>

Did you decide to continue to d/l the production en-US builds ? I think we should have a chat to sync me 

>+# CONFIG: $ssh_server      = "%sshUser%";

s/server/user/

> $ssh_user      = "ffxbld";
>+# CONFIG: $ssh_key         = "'$ENV{HOME}/.ssh/%sshUser%_dsa'";
> $ssh_key       = "'$ENV{HOME}/.ssh/ffxbld_dsa'";

There's no cltbld_dsa file on the staging machines so this will fail when setting sshUser = cltbld for a staging config.

* Why only mac config files here ? 
* Which bootstrap.cfg will be used with this ?

We should probably have a chat to get me up to speed with your progress over the last two weeks.

Comment 3

[Armen [:armenzg]]

Attachment: tinder-config.pl for all 3 platforms in the "l10n" branch

It fixes all that you mention in the previous patch

It has all 3 platforms and the bootstrap.cfg file is this:
http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-nightly-staging-bootstrap.cfg

Once it works there, the next one would be:
http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz19-nightly-bootstrap.cfg

I would really love to have that meeting

Comment 4

[Armen [:armenzg]]

Attachment: add linux and windows slaves to staging and production 1.9

Comment 5

[Armen [:armenzg]]

Attachment: tinder-config.pl for all 3 platforms in the "l10n" branch

Nick, I have fixed the %version% substitution problem by not touching it and using $BaseUrlPath to determine the appropriate ftpServer for staging purposes

I also wget to /build/tinderbox/Fx-Trunk-l10n (which is %l10n_buildDir%) instead of %l10n_buildDir%/%l10n_plafotm% because of the kernel version problem

BTW, the kernel version problem does not affect normal builds, just the wget feature in l10n

This patch helps us do repackages for nightlies under our release automation system - the same way we do l10n repackages for releases

The log of it http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Staging/1216315503.1216318159.7817.gz&fulltext=1

I will be now be working on running it in another way that will do repackage-push-announce per locale instead of repackaging all locales, pushing all of them and then announcing them all

I will also make sure that windows and mac will be fine with the patch I gave you

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 329720 [details] [diff] [review]
add linux and windows slaves to staging and production 1.9

>Index: automation/production-1.9/master.cfg
>===================================================================
>RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v
>retrieving revision 1.31
>diff -u -8 -p -r1.31 master.cfg
>--- automation/production-1.9/master.cfg	2 Jul 2008 12:01:35 -0000	1.31
>+++ automation/production-1.9/master.cfg	15 Jul 2008 20:35:09 -0000
>@@ -12,17 +12,19 @@ c = BuildmasterConfig = {}
> # tuple of bot-name and bot-password. These correspond to values given to the
> # buildslave's mktap invocation.
> 
> from buildbot.buildslave import BuildSlave
> 
> c['slaves'] = [
>  BuildSlave("production-1.9-master",""),
>  BuildSlave("fx-linux-1.9-slave2",""),
>+ BuildSlave("fx-linux-1.9-slave4",""),
>  BuildSlave("fx-win32-1.9-slave2", ""),
>+ BuildSlave("fx-win32-1.9-slave4",""),
>  BuildSlave("fx-linux64-1.9-slave1", ""),
>  BuildSlave("fx-mac-1.9-slave2", ""),
> ]
> 
> # 'slavePortnum' defines the TCP port to listen on. This must match the value
> # configured into the buildslaves (with their --master option)
> c['slavePortnum'] = 9989
> 
>Index: automation/staging-1.9/master.cfg
>===================================================================
>RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v
>retrieving revision 1.40
>diff -u -8 -p -r1.40 master.cfg
>--- automation/staging-1.9/master.cfg	3 Jun 2008 11:49:07 -0000	1.40
>+++ automation/staging-1.9/master.cfg	15 Jul 2008 20:35:09 -0000
>@@ -12,17 +12,19 @@ c = BuildmasterConfig = {}
> # tuple of bot-name and bot-password. These correspond to values given to the
> # buildslave's mktap invocation.
> 
> from buildbot.buildslave import BuildSlave
> 
> c['slaves'] = [
>  BuildSlave("staging-1.9-master",""),
>  BuildSlave("fx-linux-1.9-slave1",""),
>+ BuildSlave("fx-linux-1.9-slave3",""),
>  BuildSlave("fx-win32-1.9-slave1", ""),
>+ BuildSlave("fx-win32-1.9-slave3",""),
>  BuildSlave("mini-test", ""),
>  BuildSlave("fx-mac-1.9-slave1", ""),
> ]
> 
> # 'slavePortnum' defines the TCP port to listen on. This must match the value
> # configured into the buildslaves (with their --master option)
> c['slavePortnum'] = 9989
> 



Are you going to do anything with them...? Let's add them to the list when they're attached to a builder.

Comment 7

[Armen [:armenzg]]

Comment on attachment 330072 [details] [diff] [review]
tinder-config.pl for all 3 platforms in the "l10n" branch

As per talk in IRC, I will be pulling from the real ftp instead of the staging ftp. I will modify the patch accordingly

Comment 8

[Armen [:armenzg]]

Attachment: staging1.9/master.cfg - It shows what it looks like repackages without tinderbox client or bootstrap

Comment 9

[Armen [:armenzg]]

Attachment: tinder-config.pl for all 3 platforms in the "l10n" branch

They will WGet from the official ftp server instead of adding the extra code to allow grabbing it from the staging ftp server

I would like this patch in even if we move away from Repack.pm and we do repackages according to attachment 330636 [details] [diff] [review] to have the configuration the same way we have it in the "release_l10n" branch

Comment 10

[Armen [:armenzg]]

Attachment: staging1.9/master.cfg - It shows what it looks like repackages without tinderbox client or bootstrap

Comment 11

[Armen [:armenzg]]

Comment on attachment 330666 [details] [diff] [review]
staging1.9/master.cfg - It shows what it looks like repackages without tinderbox client or bootstrap

>+import subprocess
>+
>+def getLocales():
>+    """
>+    It returns a list with all locales from all-locales in the repository
>+    """
>+    from twisted.python import log
>+    tuple =  subprocess.Popen(
>+        ['cvs', '-q', '-d', ':ext:[e-mail]:/cvsroot',
>+         'co', '-p', 'mozilla/browser/locales/all-locales'],
>+        stdout=subprocess.PIPE).communicate()
>+    #communicate() returns a tuple
>+    #the output of cvs has a \n at the end
>+    locales = tuple[0].split('\n')
>+    if locales[-1]=='':#I don't think that I need this 2 lines
>+        locales = locales[0:-1] #get rid of the last one
>+    return locales
>+ 

nthomas, bhearshum - where do you think I should put this functionality?

>+zipin = 'firefox-3.0.2pre.en-US.linux-i686.tar.bz2'
>+latest_en_US = 'http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/firefox-3.0.2pre.en-US.linux-i686.tar.bz2'
>+
>+
>+L10nBuildFactory.addStep(ShellCommand(
>+    command=["wget", "-nv", "--output-document", 
>+             zipin, latest_en_US],
>+    workdir="mozilla/dist/"
>+    ))
>+

The only thing that we need in here is to know which *version* (3.0.2pre - http://mxr.mozilla.org/mozilla/source/browser/config/version.txt)
I could do something like what I do with getLocales()

I think in moz2 we have if conditions to determine the platform we are using and I could have a look to learn how to determine which file type I should be wgetting

It would be good if there was a make target something like: "make wget-%locale" and by giving en-US, it would download the appropriate binary. This way we wouldn't have to figure it out in the chosen automation tool, in this case, buildbot

Comment 12

[Axel Hecht [:Pike]]

The current version in RCS (CVS or hg) is not necessarily the version of the latest nightly, our l10n trees burn reliably on each version bump for not finding the latest build of a release that wasn't built yet.

This is a chicken and egg problem, you need the source to get the version, you need the version to get the binary, you need the binary to get the source stamp.

IMHO, this should be solved in the automation.

Comment 13

[Armen [:armenzg]]

I want to bring to the attention one thing

The configuration that I attached it iterates in the list of locales and adds a couple of steps per locale there is. the problem with this is that you can't add anything like haltOnFailure since it would stop the whole process.

Why do I bring this to our attention? From what I learned in Axel's work that a build request is created per locale. Why would this be important? Because we could stop the creation of a locale if it doesn't pass the compare-locales test.

Axel does it makes sense?

Comment 14

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in (HG) - It allows to wget the right en-US from path given

You can use it like this:
export FTP_PATH=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0
make wget-en-US

and it will be downloaded to $(abs_dist)
and there is where make installers-FOO looks for

Comment 15

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in - It allows to wget the right en-US from path given

The same as before but for CVS instead of HG

Comment 16

[Axel Hecht [:Pike]]

I'd r- attachment 330832 [details] [diff] [review] and attachment 330833 [details] [diff] [review]. 

make echo-variable-PACKAGE

does most of this right away.

And the patches don't cover that windows needs both the PACKAGE and installer.

Regarding your question in comment 13, I'll whack up a quick blog post with my thoughts.

Comment 17

[Armen [:armenzg]]

(In reply to comment #16)
> I'd r- attachment 330832 [details] [diff] [review] and attachment 330833 [details] [diff] [review]. 
> 
> make echo-variable-PACKAGE
> 
> does most of this right away.
> 
I tried it and it works if all I want is to echo the filename, but I want the build system to know how to download the binaries for me without having to call tinderbox or a custom class of buildbot to be able to do a wget

Comment 18

[Axel Hecht [:Pike]]

Either way you shouldn't replicate the file name logic but just use $(PACKAGE) instead.

I'm not sure if wget is actually part of our build requirements, btw. And I don't know if a wget is really all that interesting if you need to pass in the right ftp path after all. And I don't see great value in coding up our ftp infrastructure in the build system.

Btw, 0.7.9 has a SetProperty step, with which you could just get the output of make echo-variable-PACKAGE and feed it to a build property, to be used in a buildbot download step.

Comment 19

[Armen [:armenzg]]

(In reply to comment #18)
> Either way you shouldn't replicate the file name logic but just use $(PACKAGE)
> instead.
> 
Right - I will modify to reuse it

> I'm not sure if wget is actually part of our build requirements, btw. And I
> don't know if a wget is really all that interesting if you need to pass in the
> right ftp path after all. And I don't see great value in coding up our ftp
> infrastructure in the build system.
> 
The ftp info cannot be extracted as you mention from our build system and it would not make sense to store it in it

I think that being able to do something like:
command=["make","wget-en-US"]
and env={FTP_PATH:"http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0"}
platform independent is IMHO an advantage and we don't have to rely on buildbot

> Btw, 0.7.9 has a SetProperty step, with which you could just get the output of
> make echo-variable-PACKAGE and feed it to a build property, to be used in a
> buildbot download step.
> 
That is a great step and would be good to have it now (aside of this specific problem) but we currently don't

Comment 20

[Armen [:armenzg]]

Axel, what is the echo variable that gives us the installer for windows?
I have been looking for a while

Comment 21

[Axel Hecht [:Pike]]

toolkit/mozapps/installer/windows/nsis/makensis.mk hardcodes that to $(DIST)/install/sea/$(PKG_BASENAME).installer.exe

I guess you could make it a variable there, and use that from browser/locales/Makefile.in. You should test that thoroughly as we're using PACKAGE twice, once with AB_CD = $(MOZ_UI_LOCALE) and once with AB_CD = $*. Probably should be good enough to make sure it's a foo=bar var instead of foo:=bar, but you should test that.

Comment 22

[Armen [:armenzg]]

(In reply to comment #21)
> toolkit/mozapps/installer/windows/nsis/makensis.mk hardcodes that to
> $(DIST)/install/sea/$(PKG_BASENAME).installer.exe
> 
> I guess you could make it a variable there, and use that from
> browser/locales/Makefile.in. 
>
What do you think of calling it like PACKAGE_INSTALLER or WIN32_INSTALLER/

> You should test that thoroughly as we're using
> PACKAGE twice, once with AB_CD = $(MOZ_UI_LOCALE) and once with AB_CD = $*.
> Probably should be good enough to make sure it's a foo=bar var instead of
> foo:=bar, but you should test that.
> 
I am not sure what you mean but I will make sure it works in all 3 platforms. My make gnu skills are limited :) 

Comment 23

[Axel Hecht [:Pike]]

You should check in with Ted, Benjamin or Rob Strong on the variable name. And if it should include install/sea/ or not. I'd be in favor of including it.

Comment 24

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in - It allows to wget the right en-US from path given

Axel, I have tried to set PACKAGE_INSTALLER in the file you mentioned but it did not capture it
I am going to use the .installer.exe way

Ted, I will add the hg patch afterwards

Comment 25

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in - It downloads the installer for windows in dist/install/sea

Comment 26

[Armen [:armenzg]]

Attachment: how to do nightly l10n repackages under buildbot

I just wanted you to have a look to know where I am

This works so far but there are some things missing
- Upload the windows installer
- Upload the xpi's with the right name, instead of firefox-3.0.2pre.af.xpi (I can't recall the name) have af.xpi (which is what I have seen in ftp)
- Enable update for locales - ** I am going to need as much help as possible - I know nothing **
- getLocales() - I bet it won't detect new locales added unless the configuration is read (buildbot restart) - am I right? - What is the behaviour if I use "for locale in getLocales()", is it the same? * More testing *
- getLocales() should accept different cvsroot (staging and production wise)
- I would like to try having a DependentL10n/SchedulerL10n scheduler with the subclassed Build class that worked for me. This allow us to see how the parallelization will work but with just one slave

Once we have the nightly L10n repackages having the right configuration under buildbot. 
 1) What do we do with *dependent L10n repackages*?
Shall we do dependent repackages every 2 hours and push them to tinderbox-builds? Could we stop calling it "dependent" builds and remove objdir and source but push it to tinderbox-builds?
Shall we use a poller and just repackage those locales that have changed?

 2) Whatever I make work for nightly builds should work for releases
What else is done after a repackage to be called a release?
Updating? Branding as well. Pulling a TAG instead of locale's tip

Comment 27

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in - It downloads en-US packages and the installer for windows in dist/install/sea - comments added

Comment 28

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in (HG) - It downloads en-US packages and the installer for windows in dist/install/sea - comments added

This is the HG patch

Comment 29

[Armen [:armenzg]]

Attachment: buildbotcustom/l10n - Contains NightlyL10n, PeriodicL10n, Build and L10nHelper classes

This would be the classes that will be used to make paralell builds. This is not final since I have to fix a bug regarding multiple builders

Comment 30

[Armen [:armenzg]]

Attachment: staging-1.9/master.cfg - It generates a build object per locale

Nick, how does it look? Do you know what am I doing when we put it in connection to the previous patch?

As I mentioned before I have to fix the multiple builders scenario

This patch is obviously not final but it looks a lot on how it should be
 * remove the remove step
 * remove the step to apply my patch

MENTAL NOTE: Add mozconfig-l10n to the patch

Comment 31

[(not currently active) Ted Mielczarek]

Comment on attachment 331330 [details] [diff] [review]
browser/locales/Makefile.in - It downloads en-US packages and the installer for windows in dist/install/sea - comments added

You don't need to submit separate patches for hg and cvs if they contain the same code.

Comment 32

[(not currently active) Ted Mielczarek]

Comment on attachment 331331 [details] [diff] [review]
browser/locales/Makefile.in (HG) - It downloads en-US packages and the installer for windows in dist/install/sea - comments added

+ifndef FTP_PATH

I think you should make this variable more descriptive, but I can't seem to come up with a better name for it.

+wget-en-US:+       @echo "Downloading $(FTP_PATH)/$(PACKAGE) to $(_ABS_DIST)/$(PACKAGE)"

That echo should be on the next line.

+       @wget -nv --output-document $(_ABS_DIST)/$(PACKAGE) $(FTP_PATH)/$(PACKAGE)

I don't think you should use wget here directly. You should add a check for it in configure, and then error out in this makefile target if wget is not installed.

It's pretty simple, just something like this:
http://mxr.mozilla.org/mozilla-central/source/configure.in#5785

You'll just need an AC_CHECK_PROGS and AC_SUBST, and then you'll need to add the variable to config/autoconf.mk.in.

r- for that, but fix that and you should get r+.

Comment 33

[Armen [:armenzg]]

Attachment: buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes

We have a queue per builder and now the queues are contained in the L10nHelper
In staging I have 3 builders, one of them with 3 slaves and it seems to work so far

Comment 34

[Armen [:armenzg]]

BTW, the patch does not add __init__.py to the folder l10n as I wanted - I used cvsdo add as with the other files

Comment 35

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in & configure.in - It checks that wget is available and helps us determine which en-US installers to install

I have changed the URL to something more meaningful since what we want is the URL prefix to the latest en-US binaries

The configure part seems pretty straight forward - is it right?

Comment 36

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in & configure.in - It checks that wget is available and helps us determine which en-US installers to install

I attached the old patch

Comment 37

[(not currently active) Ted Mielczarek]

Comment on attachment 333045 [details] [diff] [review]
browser/locales/Makefile.in & configure.in - It checks that wget is available and helps us determine which en-US installers to install

+if test -z "$WGET"; then
+    AC_MSG_ERROR([no wget found in \$PATH])
+fi

Remove this part from configure, we shouldn't make it a configure error to not have wget. You should use an ifdef in the wget-en-US rule in the makefile, something like:

wget-en-US:
ifdef WGET
... (all the stuff you have currently)
else
  $(error Wget not installed)
endif

You'll also need to modify config/autoconf.mk.in, and add a WGET = @WGET@ line for this to work properly.

Comment 38

[Armen [:armenzg]]

Attachment: browser/locales/Makefile.in, configure.in & config/autoconf.mk.in - It checks that wget is available and helps us determine which en-US installers to install

Comment 39

[Armen [:armenzg]]

Ted,
do you know why do I get this message?
"make: WGET@: Command not found"
I added the patch to the staging repo and I can pass through

make wget-en-US
 in dir /Users/cltbld/trunk-automation/macosx_l10n_nightly/mozilla/browser/locales (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', 'wget-en-US']
 environment:
  CVS_RSH=ssh
  EN_US_BINARY_URL=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages/:/tools/zope-interface/lib/python2.5/site-packages/
  SECURITYSESSIONID=485d10
  SHELL=/bin/bash
  SHLVL=1
  TBOX_CLIENT_CVS_DIR=/builds/tinderbox/mozilla/tools
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=133
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F6:0:0
 closing stdin
 using PTY: False
make: WGET@: Command not found
make: *** [wget-en-US] Error 127
program finished with exit code 2

Comment 40

[Armen [:armenzg]]

Attachment: wget the en-US binary and prepare L10n repackages to be uploaded

I have added a new target that organizes the binaries and xpi files into the same structure as what we offer in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/

Comment 41

[[e-mail] (:bhearsum)]

Comment on attachment 332979 [details] [diff] [review]
buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes

Can you post the master.cfg you're using? Not for review, just so I can see how this stuff fits together.

Comment 42

[(not currently active) Ted Mielczarek]

Comment on attachment 333664 [details] [diff] [review]
wget the en-US binary and prepare L10n repackages to be uploaded

+ifeq (Linux, $(OS_ARCH))
+XPI_DESTINATION = linux-xpi
+endif
+ifeq (Darwin, $(OS_ARCH))


You can structure this like:

ifeq (Linux, $(OS_ARCH))
XPI_DESTINATION = linux-xpi
else ifeq (Darwin, $(OS_ARCH))
...
endif

You should also put in a final:
else
XPI_DESTINATION = $(error Unsupported platform)
endif

just in case someone tries to run this on some other platform.

+# Move the windows' installer

nit: get rid of the apostrophe there.

The Makefile bits look fine to me, but I don't really know anything about the l10n repackaging. Do you think Axel would like to take a look at this?

r=me with those changes

Comment 43

[Armen [:armenzg]]

Attachment: staging-1.9/master.cfg - It repackages locales as individual build objects and pushes them to the staging ftp

This is the current master.cfg
I am working on reporting to Tinderbox properly as discussed in the email

Comment 44

[Armen [:armenzg]]

Attachment: It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly

Fixed what ted mentioned and added "win32." before the word installer.exe

Axel this patch is working so far for me, you can see the calls to this targets in the last master.cfg that I uploaded

Comment 45

[Armen [:armenzg]]

Attachment: tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree"

This allow us to report to locale specific tinderbox pages which are generated like this:
- MyTboxPage-locale
therefore:
tree = WithProperties("MyTboxPage-%(locale)s")

Comment 46

[Axel Hecht [:Pike]]

Comment on attachment 333999 [details] [diff] [review]
tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree"

>+        assert tree is type(tree) is str \
>+            or isinstance(tree, WithProperties), \
>+            "tree must be a string or a WithProperties instance"

tree is type(tree) is str?

>-        text += "%s tree: %s\n" % (t, self.tree)
>+        if type(self.tree) is str:
>+            # use the exact string given
>+            text += "%s tree: %s\n" % (t, self.tree)
>+        elif isinstance(self.tree, WithProperties):
>+            # interpolate the WithProperties instance, use that
>+            text += "%s tree: %s\n" % (t, self.tree.render(build))
>+        else:
>+            raise Exception("tree is an unhandled value")    
>+

I'd prefer isinstance(self.tree, basestring) in general.

Comment 47

[Armen [:armenzg]]

Attachment: tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree"

I have added Axel's suggestion and fixed a problem in the assert line

I have applied this patch only to staging-1.9-master:
[root@staging-trunk-automation status]# pwd
/tools/buildbot/lib/python2.5/site-packages/buildbot/status
[root@staging-trunk-automation status]# patch -p4 < /home/cltbld/tboxnotifier.445454.diff

Comment 48

[[e-mail] (:bhearsum)]

Comment on attachment 334010 [details] [diff] [review]
tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree"

Looks fine to me...

Comment 49

[Axel Hecht [:Pike]]

Comment on attachment 333972 [details] [diff] [review]
It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly

This bug is getting really hard to read, you should fork some items out to independent bugs.

r=me with one change, 

wget-en-US:
ifndef WGET
	$(error Wget not installed)
endif

at the start and then no else clause, that works just as fine, and removes the nested ifs which are hard to read.

Comment 50

[[e-mail] (:bhearsum)]

(In reply to comment #49)
> (From update of attachment 333972 [details] [diff] [review])
> This bug is getting really hard to read, you should fork some items out to
> independent bugs.
> 

seconded.

Comment 51

[Armen [:armenzg]]

Comment on attachment 332979 [details] [diff] [review]
buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes

Nick is going to be busy to review this

Comment 52

[Armen [:armenzg]]

(In reply to comment #50)
> (In reply to comment #49)
> > (From update of attachment 333972 [details] [diff] [review] [details])
> > This bug is getting really hard to read, you should fork some items out to
> > independent bugs.
> > 
> 
> seconded.
> 
Will do

Comment 53

[Armen [:armenzg]]

Comment on attachment 332979 [details] [diff] [review]
buildbotcustom/l10n - Contains NightlyL10n Build and L10nHelper classes

I have moved this patch to bug 451056

Comment 54

[Armen [:armenzg]]

There are now 3 not obsolete patches in this bug.
* The master.cfg file (which I will move to another bug when it is completely done)
* The wget patch which I will fix the comments of Axel
* The tinderbox.py modification which can land whenever someone can do that

From now on, I will keep this bug for the comments of wget and move the few upcoming master configurations to another bug

Comment 55

[Armen [:armenzg]]

Comment on attachment 333963 [details] [diff] [review]
staging-1.9/master.cfg - It repackages locales as individual build objects and pushes them to the staging ftp

A patch has been added to bug 451088

Comment 56

[Armen [:armenzg]]

Attachment: It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly

Fixed as per Axel's comments

Can please someone check this in?

Comment 57

[Armen [:armenzg]]

Attachment: It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly

I removed one line more than needed - It is ready now

Comment 58

[Armen [:armenzg]]

Comment on attachment 334010 [details] [diff] [review]
tools/buildbot/status/tinderbox.py - It enables WithProperties for the parameter "tree"

checkin-needed for this one

Comment 59

[Armen [:armenzg]]

Comment on attachment 334340 [details] [diff] [review]
It wgets the latest en-US and it also allows us to move installers and xpi files to dist/upload in the structure we use in ftp/nightly

this patch has been moved to bug 451466 where I ask for checkin-needed

NOTE: From now on as I have been told I will file smaller bugs. I am sorry this bug has turned into this monster

Comment 60

[Armen [:armenzg]]

(In reply to comment #58)
> (From update of attachment 334010 [details] [diff] [review])
> checkin-needed for this one
Can someone please check this patch in to CVS HEAD?

Comment 61

[Axel Hecht [:Pike]]

I suggest opening a new bug on the tinderbox changes, with proper deps. Is there an upstream bug for this?

Comment 62

[Armen [:armenzg]]

(In reply to comment #61)
> I suggest opening a new bug on the tinderbox changes, with proper deps. Is
> there an upstream bug for this?
Patch has been moved to Bug 453229

No more patches in this bug and I will attach no more to this one