511967 - Create complete and partial patches on build slaves instead of on prometheus-vm

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris Cooper [:coop] (he/him)]

Nightly updates for all our products/branches are currently generated by a single VM, prometheus-vm. The potential improvements in bug 511901 notwithstanding, this is a bottleneck that limits the number of updates we can generate in a given amount of time.

We already generate the complete mars on the build slaves. I don't think it would be too hard to generate the partial patches on the slaves as well. I *think* we already have enough information about the previous build simply based on the contents of latest-*/ on stage.mozilla.org. Maybe there are subtleties to the update graph that I'm not appreciating, but I think for any single build it's pretty straightforward. Please educate me if I'm wrong.

One ancillary benefit of adding update generation to the regular nightly build process on the slaves is that logs would be available to everyone. Partial patch generation is currently opaque to people watching the process from the outside.

Comment 1

[u88484]

Any update on this progress?  I know it is not a huge issue but merely a nuisance for nightly users. I understand the priority may be not that great but how about kicking the window builds off a few hours earlier?  Or is that a bad idea because the nightlies won't contain roughly the same patches?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to comment #1)
> Any update on this progress?  I know it is not a huge issue but merely a
> nuisance for nightly users.

I sympathize. Armen's work gets win32 updates for en-US out before noon PDT now, but that's still about 4-5 hours slower than before.

This patch is coming along. The current long pole is making sure that the mar tools work (and in a predictable way) on Windows, since historically they've only been used on linux (and occasionally on Mac).

> how about kicking the window builds off a few hours earlier?  Or is that a bad
> idea because the nightlies won't contain roughly the same patches?

Nightlies should ideally be built from the same changeset, so no, time-shifting them isn't really an option.

Comment 3

[Armen [:armenzg]]

I am working on a patch that the script will process a set number of updates rather than all pending ones. This will make the script refresh with the new pending updates increasing the chances of processing en-US earlier.

Comment 4

[Chris Cooper [:coop] (he/him)]

I have partial updates being generated on my local mac build slave now. Next I'm looking at the changes required for uploading those partials to the right places. There's also some cross-platform work still to do, as mentioned previously in comment #2.

Comment 6

[Chris Cooper [:coop] (he/him)]

I have slaves happily generating updates in staging now. Now I'm at the point where I need to extend the 'make upload' target to allow for uploading partial mars.

I've looked at how the COMPLETE_MAR is set and used:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk#108
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#526

I would like to do something similar for PARTIAL_MAR but there are 2 wrinkles:

1) the filename for the partial MAR contains the build IDs for the current and previous build (e.g. firefox-3.6b4pre.en-US.mac.partial.20091124033613-20091120151758.mar), and hence cannot be created by simple concatenation like the complete MAR; and,

2) not all builds will be generating partial mars, and I don't want to break those uploads.

Ted: what's the best way to do this?

I already have the partial MAR filename as a property in buildbot, so I can add it to the env. I've naively tried doing something like this:

http://pastebin.mozilla.org/686348

This works when the env var PARTIAL_MAR_FILENAME is set, i.e. the partial MAR is uploaded. PARTIAL_MAR ends up as a quoted dir entry ("../../dist/") when PARTIAL_MAR_FILENAME is not set, and the upload process fails. 

If I don't call QUOTED_WILDCARD on "$(DIST)/$(PARTIAL_MAR)", the upload also succeeds, but I don't know how kosher that is. :/

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> 1) the filename for the partial MAR contains the build IDs for the current and
> previous build (e.g.
> firefox-3.6b4pre.en-US.mac.partial.20091124033613-20091120151758.mar), and
> hence cannot be created by simple concatenation like the complete MAR; and,
>

You could probably work around this by doing
PARTIAL_MAR=$(PKG_UPDATE_PATH)$(PKG_UPDATE_BASENAME).partial.*.mar

IIRC the * gets expanded when it gets run through the QUOTED_WILDCARD macro.

> 2) not all builds will be generating partial mars, and I don't want to break
> those uploads.

Strange that just doing the same as the complete in packager.mk doesn't work. I don't see how those two cases are different =\.

Comment 8

[(not currently active) Ted Mielczarek]

For 1), you'll need to pass the old build ID somewhere that the build system can see it. You could pass it in the environment or on the make commandline.
For 2), as long as you use QUOTED_WILDCARD, you'll be fine. That uses the $(wildcard) function internally, which expands to an empty string when the file doesn't exist, so the filename gets dropped from the commandline.

Comment 9

[(not currently active) Ted Mielczarek]

Oh, or do what bhearsum said, that will work, since we are passing it through $(wildcard). Duh.

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to comment #9)
> Oh, or do what bhearsum said, that will work, since we are passing it through
> $(wildcard). Duh.

This seemed like the simplest approach, so I tried this first, setting PARTIAL_MAR just like COMPLETE_MAR but with a wildcard placeholder for the build IDs. 

http://pastebin.mozilla.org/686528

It *almost* works in that if the partial mar is present, PARTIAL_MAR gets added to the filelist for upload.py. Unfortunately, the wildcard doesn't get expanded, i.e. PARTIAL_MAR is passed as "../../dist/update/firefox-3.6b4pre.en-US.mac.partial.*.mar" to upload.py which causes it to fail.

Output from above, partial mar present: http://pastebin.mozilla.org/686529

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to comment #10)
> to the filelist for upload.py. Unfortunately, the wildcard doesn't get
> expanded

Must be something wrong with QUOTED_WILDCARD because if I do this:

$(call QUOTED_WILDCARD,$(wildcard $(DIST)/$(PARTIAL_MAR)))

...I get the expansion I need.

Ted: is that acceptable, or does QUOTED_WILDCARD need fixing?

Comment 12

[(not currently active) Ted Mielczarek]

Oh, duh, if you look at the definition of QUOTED_WILDCARD, it only uses the wildcard bit for the if clause, it then returns "$(1)", which is just whatever you pass in, but in quotes. I think the whole point of that is that $(wildcard) doesn't gracefully handle spaces in filenames. If your mar filenames aren't ever going to have spaces, then sticking the extra wildcard in there is probably fine.

Comment 13

[Chris Cooper [:coop] (he/him)]

Attachment: Upload partial mars when they exist [Checkin: Comment 20]

Adding the extra $(wildcard) does what I need: expands properly to include the partial mar when it is present, and disappears when it is not. Tested in staging this week.

This will need to land eon all branches where we do updates once it proves itself on m-c.

Comment 14

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves

Nick: given the size of this patch, it would be a good idea IMO to start with a cursory review to ensure I'm not completely off my rocker before delving more deeply. There are a couple of areas that I'm not terribly proud of:

* setting all the various dir variables (e.g fullSrcDir, mozillaSrcDir, etc) to allow sharing the code between en-US nightlies and l10n nightlies. If you can think of a more elegant way to accomplish this, I'm all ears.

* the way I make sure mar and mbsdiff are built and are in the right place. Any suggestions to clean this up are welcome.

There are various other gotchas I experienced in staging that are represented in this patch:

* need to specify and ausSshKey because snippets are uploaded by prometheus-vm under /opt/aus/build/0 as ffxbld, but under /opt/aus2/incoming/2 as ffxbld. I added the key elsewhere (e.g. release factories) as a precaution.

* need to run any command that needs shell expansion through 'bash -c' on windows, otherwise things like '~' and '..' aren't interpreted properly

Over the past week, I've generated partial patches on all 3 platforms for both en-US and l10n nightlies using this patch, but it has required a lot of handholding. This was especially true for the l10n builds where we often did not have previous mars present (I ended up syncing mars over from production). 

Due to the cycle time on Windows (and consequent demand by others for Windows slaves), I don't think I've seen all the various Windows builds complete successfully in staging while running this patch. All the Mac and Linux builders have gone green however. 

I'll continue to let the patch run on s-m2 to hit the other Windows builders, if you want to poke around at logs.

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: Required config changes to allow building partial mars on build slaves.

My goal was to allow us to turn partial generation on one branch at a time by toggling the create_partial var. That way we can stick with the existing centralized system for key branches while trying this code out on a single branch. prometheus-vm will happily continue creating partial patches for any snippets it sees in the regular spot (including those for Thunderbird), its load will just be lightened by however many branches we're able to divert.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 415887 [details] [diff] [review]
Upload partial mars when they exist
[Checkin: Comment 20]

>diff --git a/toolkit/mozapps/installer/package-name.mk b/toolkit/mozapps/installer/package-name.mk
>--- a/toolkit/mozapps/installer/package-name.mk
>+++ b/toolkit/mozapps/installer/package-name.mk
>+PARTIAL_MAR = $(PKG_UPDATE_PATH)$(PKG_UPDATE_BASENAME).partial.*.mar

Can you add a comment above this variable definition (and its twin below) noting that it needs to be run through $(wildcard) before use?

r=me with that addition.

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 416011 [details] [diff] [review]
buildbotcustom changes to create partials on slaves

Canceling reviews for now while I try to get the mar/bsdiff tools built by default with --enable-update-packaging.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I was guessing something like http://pastebin.mozilla.org/691087 but haven't tested that.

Comment 19

[Chris Cooper [:coop] (he/him)]

(In reply to comment #18)
> I was guessing something like http://pastebin.mozilla.org/691087 but haven't
> tested that.

Yep, I ran that very patch through the try-server last week. It works on our core platforms, but it currently fails for WinCE and WinMo. There is activity on both fronts to make it work.

Comment 20

[Chris Cooper [:coop] (he/him)]

Comment on attachment 415887 [details] [diff] [review]
Upload partial mars when they exist
[Checkin: Comment 20]

http://hg.mozilla.org/mozilla-central/rev/3481134b6bb4

Comment 21

[Chris Cooper [:coop] (he/him)]

Comment on attachment 415887 [details] [diff] [review]
Upload partial mars when they exist
[Checkin: Comment 20]

Ted: this patch will need to land on 1.9.2 and 1.9.1 as well. Should I seek approval for that, or would it be considered NPOTB?

Comment 22

[(not currently active) Ted Mielczarek]

I'm not sure if it's really NPOTB or not, even if it is essentially non-functional without the other patches here.

Comment 23

[Chris Cooper [:coop] (he/him)]

Attachment: [mozilla-1.9.2] Upload partial mars when they exist [Checkin: Comment 26]

Carrying forward review from Ted (we've had plenty of bake time on trunk), but requesting approval on 1.9.2 branch.

Comment 24

[Chris Cooper [:coop] (he/him)]

Attachment: [mozilla-1.9.1] Upload partial mars when they exist [Checkin: Comment 27]

Carrying forward review from Ted (we've had plenty of bake time on trunk), but requesting approval on 1.9.1 branch.

Comment 25

[Daniel Veditz [:dveditz]]

Comment on attachment 422410 [details] [diff] [review]
[mozilla-1.9.1] Upload partial mars when they exist
[Checkin: Comment 27]

Approved for 1.9.1.8, a=dveditz for release-drivers

Comment 26

[Chris Cooper [:coop] (he/him)]

Comment on attachment 422407 [details] [diff] [review]
[mozilla-1.9.2] Upload partial mars when they exist
[Checkin: Comment 26]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/e8f5e9c35f9a

Comment 27

[Chris Cooper [:coop] (he/him)]

Comment on attachment 422410 [details] [diff] [review]
[mozilla-1.9.1] Upload partial mars when they exist
[Checkin: Comment 27]

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/c616b502b9a5

Comment 28

[Daniel Veditz [:dveditz]]

Assuming "checked-in+" means fixed on that branch?

Comment 29

[Chris Cooper [:coop] (he/him)]

(In reply to comment #28)
> Assuming "checked-in+" means fixed on that branch?

Well, the part of this bug that's going to involve the main source repos is fixed. Still stuff to do on the build repo side.

Comment 31

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v2

Nick: ready for a proper review now. This patch has worked in staging for Linux/Mac/Win32 + l10n. 

The notes/caveats from comment #14 still apply, with the following addenda:

* bsdiff is created automatically now when --enable-update-packaging is specified, so we only need to build it for the l10n case. Luckily we already configure with the necessary option, so it's just a matter of make-ing the extra dirs.

* I moved around some of the SetProperty steps in MozillaBuildFFactory so they'd be done first and grouped together logically

* moved baseWorkDir up into MozillaBuildFactory so it could be shared

* some naming and indentation fixed for clarity

* removal of duplicated SetProperty steps for toolsdir from UnittestBuildFactories

Updated config patch coming up.

Comment 32

[Chris Cooper [:coop] (he/him)]

Attachment: [staging] Required config changes to allow building partial mars on build slaves, v2

Staging only, and only enabled for m-c.

Comment 34

[Chris Cooper [:coop] (he/him)]

Nick: I'm also curious about your thoughts on bug 392726.

My patches as posted just maintain the status quo, i.e. upload both complete and partial snippets at the end of the process assuming we don't error out earlier. IMO, I think we *do* want this all-or-nothing approach because it will help us catch errors more quickly. A complete absence of updates is more likely to elicit a bug report than a mix of working complete/partial updates.

I will note that we're already more likely to catch missing updates when we're generating updates on slaves because errors will end up directly in buildbot/TBPL output. prometheus-vm is an invisible source of failure for people outside the releng team right now.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 429051 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v2

Here's a first pass at this. Got part way through and need to wrap my head around the sub-classing and where everything is living.

>diff --git a/misc.py b/misc.py
>                 enable_ccache=pf.get('enable_ccache', False),
>+                hashType=config['hash_type'],

Why not add this next to the other aus/update params ? Perhaps there's a style rule I'm not aware of.

>                         clobberURL=config['base_clobber_url'],
>                         clobberTime=clobberTime,
>+                        hashType=config['hash_type'],
>+                        createPartial=config['create_partial'],

Same comment as above.

>diff --git a/process/factory.py b/process/factory.py
>-         command=['bash', '-c',
>-          'if [ ! -d tools ]; then hg clone %s tools;fi' % self.buildToolsRepo],
>+         command=['hg', 'clone', self.buildToolsRepo, 'tools'],

yay!

@@ -451,6 +471,15 @@ class MercurialBuildFactory(MozillaBuild
...
>+        # These following variables are useful for sharing build steps with
>+        # classes that don't use object dirs, e.g. update generation
>+        self.mozillaSrcDir = '.'
>+        self.fullSrcDir = self.baseWorkDir
>+        self.fullObjdir = '%s/%s' % (self.baseWorkDir, self.mozillaObjdir)

Could you add some documentation for these properties ? When you haven't looked at this in some time it's not clear if they're related to what SeaMonkey/Thunderbird need for their two-repo builds. 

>@@ -1149,16 +1200,261 @@ class NightlyBuildFactory(MercurialBuild
...
>+        # Download the previous nightly complete mar.
>+        # The mar file name will be the same from one day to the next,
>+        # so we can re-use the filename from the current mar provided
>+        # we rename it on download
>+        previousMarURL = WithProperties('http://%s' % self.stageServer + \
>+                          '%s' % self.latestDir + \
>+                          '/%(completeMarFilename)s')

Unfortunately this will not be true when the application version is bumped. No great solutions spring to mind, but maybe glob on ftp://ftp.m.o ? Or query AUS with a really old buildID and scrape mar location ?

>+    def addCreateSnippetsSteps(self):
>+        self.addStep(CreateCompleteUpdateSnippet(
...
>+        self.addStep(ShellCommand(

This looks like the same as on MercurialBuildFactory, except for using self.fullObjdir. Can call out to the parent factory I think, then 
>+        if self.createPartial:

>+    def addUploadSnippetsSteps(self):
>+        if self.createPartial:
>+            ausPreviousBuildUploadDir = "%s/%%(previous_buildid)s/en-US" % \
>+                                         self.ausFullUploadDir
>+        else:
>+            ausPreviousBuildUploadDir = self.ausFullUploadDir

The naming threw me a bit here, since the existing case it's not really the previous build. Could you just append to ausFullUploadDir when createPartial == True ? Guess it depends on this ...

>+        # We only need to worry about empty snippets (and partials obviously)
>+        # if we are creating partial patches on the slaves.

I think these empty snippets are an artifact of patch-packager and not required by AUS. A quick check on a local AUS instance confirms that you still get the latest build if they're removed. Do you know any more about that ?

>+            self.addStep(ShellCommand(
>+                name='create_empty_complete_snippet',
>+                command=['bash', '-c',
>+                         WithProperties('ssh -l %s ' %  self.ausUser +
>+                                        '-i ~/.ssh/%s %s ' % (self.ausSshKey,self.ausHost) +
>+                                        'touch %s/complete.txt' % ausCurrentBuildUploadDir)],
>+                description=['create', 'empty', 'complete', 'snippet'],
>+                haltOnFailure=True,
>+            ))

If this is required then we can do partial and complete in one step.

>@@ -1196,24 +1492,28 @@ class NightlyBuildFactory(MercurialBuild
>+            self.addStep(SetProperty(
>+                name='make_upload',
>+                command=uploadCmd,
>+                env=uploadEnv,
>+                workdir='build/%s' % self.objdir,

self.fullObjDir ?

>@@ -1419,23 +1718,24 @@ class BaseRepackFactory(MozillaBuildFact

Got to here.

Comment 36

[Robert Kaiser]

(In reply to comment #35)
> @@ -451,6 +471,15 @@ class MercurialBuildFactory(MozillaBuild
> ...
> >+        # These following variables are useful for sharing build steps with
> >+        # classes that don't use object dirs, e.g. update generation
> >+        self.mozillaSrcDir = '.'
> >+        self.fullSrcDir = self.baseWorkDir
> >+        self.fullObjdir = '%s/%s' % (self.baseWorkDir, self.mozillaObjdir)
> 
> Could you add some documentation for these properties ? When you haven't looked
> at this in some time it's not clear if they're related to what
> SeaMonkey/Thunderbird need for their two-repo builds. 

Right, "fullSrcDir" and "fullObjdir" are strange names for that. What does "full" mean there? Is that the directory that contains the application source or the one that contains the Mozilla platform source (note that this is the same directory for Firefox, but not for SeaMonkey or Thunderbird)?

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 429051 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v2

>@@ -1419,23 +1718,24 @@ class BaseRepackFactory(MozillaBuildFact

Onward we go ...
 
>@@ -1453,32 +1753,37 @@ class BaseRepackFactory(MozillaBuildFact
>+        # These following variables are useful for sharing build steps with
>+        # classes that use object dirs, e.g. update generation
>+        self.fullSrcDir = "%s/%s" % (self.baseWorkDir, self.mozillaSrcDir)

Definitely need some documentation if these differ from the en-US build.

>@@ -1958,16 +2294,32 @@ class NightlyRepackFactory(BaseRepackFac
>+    def makePartialTools(self):
>+        # Build the tools we need for update-packaging, specifically bsdiff.
>+        # Configure can take a while.
>+        self.addStep(ShellCommand(
>+            name='make_bsdiff',
>+            command=['sh', '-c',
>+                     'if [ ! -e %s/dist/host/bin/mbsdiff ]; then ' % self.mozillaObjdir +
>+                     'make -C nsprpub; make -C config;' +
>+                     'make -C modules/libmar; make -C modules/libbz2;' +
>+                     'make -C other-licenses/bsdiff;'
>+                     'fi'],
>+            description=['make', 'bsdiff'],

How much overhead does this add ? Existing l10n nightlies have already built in nsprpub, config, libmar IIRC, and we're up to 11 minutes per locale on windows.

>@@ -3219,38 +3564,42 @@ class ReleaseUpdatesFactory(ReleaseFacto
>-            self.addStep(ShellCommand(
>-             name='upload_snipets',
>+            self.addStep(ShellCommand,
>+             name='upload_snippets',
>              command=['rsync', '-av', localDir + '/',
>                       '%s@%s:%s' % (self.ausUser, self.ausHost, snippetDir)],

For completeness we should add 
  '-e', 'ssh -i ~/.ssh/%s' % self.ausSshKey
after the '-av'.

>diff --git a/steps/updates.py b/steps/updates.py
>-from buildbot.steps.shell import WithProperties
>+from buildbot.steps.shell import ShellCommand, WithProperties, SetProperty

What are these extra functions used for ?

>-        snippet += "%s\n" % self.getProperty('buildid') # double check case
>+        snippet += "build=%s\n" % self.getProperty('buildid') # double check case

Case is fine! Great to be switching to the annotated format.

To summarize, looking good but a few issues to address still, the major ones being
* downloading the previous mar when the version has been bumped. We could possibly create some sort of feed out of the build database to provide this information
* documentation for the various srcdir/objdir variables (with apologies for hitting you with that here)
* confirming empty snippets are unnecessary

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #34)
> Nick: I'm also curious about your thoughts on bug 392726.

I don't think that bug is more than an enhancement, tbh. Maintaining the status quo is fine, especially since we don't want end up in a situation where we give people a complete while we're generating the partial.

Comment 39

[Chris Cooper [:coop] (he/him)]

(In reply to comment #35) 
> Why not add this next to the other aus/update params ? Perhaps there's a style
> rule I'm not aware of.

I'll fix it up.

> @@ -451,6 +471,15 @@ class MercurialBuildFactory(MozillaBuild
> ...
> >+        # These following variables are useful for sharing build steps with
> >+        # classes that don't use object dirs, e.g. update generation
> >+        self.mozillaSrcDir = '.'
> >+        self.fullSrcDir = self.baseWorkDir
> >+        self.fullObjdir = '%s/%s' % (self.baseWorkDir, self.mozillaObjdir)
> 
> Could you add some documentation for these properties ? When you haven't looked
> at this in some time it's not clear if they're related to what
> SeaMonkey/Thunderbird need for their two-repo builds. 

Although I originally determined I'd needed some vars like this when I started this work back in the fall, mozillaSrcDir and mozillaObjDir are actually properties added by Armen as part of bug 531275. All I'm doing here is making sure that:
a) they exist for the default Firefox case, because they already exist for l10n, SeaMonkey, etc; and,
b) appending the baseWorkDir so I don't have to do that everywhere. Perhaps the var names aren't ideal, but I'm open to suggestions.
 
> Unfortunately this will not be true when the application version is bumped. No
> great solutions spring to mind, but maybe glob on ftp://ftp.m.o ? Or query AUS
> with a really old buildID and scrape mar location ?

Yeah, not sure what to do in this case. Maybe check the version that's in latest-*? Will think about it. :(

> This looks like the same as on MercurialBuildFactory, except for using
> self.fullObjdir. Can call out to the parent factory I think, then 

I'll fix this up.

> >+        if self.createPartial:
> 
> >+    def addUploadSnippetsSteps(self):
> >+        if self.createPartial:
> >+            ausPreviousBuildUploadDir = "%s/%%(previous_buildid)s/en-US" % \
> >+                                         self.ausFullUploadDir
> >+        else:
> >+            ausPreviousBuildUploadDir = self.ausFullUploadDir
> 
> The naming threw me a bit here, since the existing case it's not really the
> previous build. Could you just append to ausFullUploadDir when createPartial ==
> True ? Guess it depends on this ...
> I think these empty snippets are an artifact of patch-packager and not required
> by AUS. A quick check on a local AUS instance confirms that you still get the
> latest build if they're removed. Do you know any more about that ?

Again, just replicating the existing system without necessarily looking for improvements. I'm all for removing the empty snippet generation steps if they are redundant.
 
> >@@ -1196,24 +1492,28 @@ class NightlyBuildFactory(MercurialBuild
> >+            self.addStep(SetProperty(
> >+                name='make_upload',
> >+                command=uploadCmd,
> >+                env=uploadEnv,
> >+                workdir='build/%s' % self.objdir,
> 
> self.fullObjDir ?

I'll fix this up.

(In reply to comment #37)
> >@@ -1453,32 +1753,37 @@ class BaseRepackFactory(MozillaBuildFact
> >+        # These following variables are useful for sharing build steps with
> >+        # classes that use object dirs, e.g. update generation
> >+        self.fullSrcDir = "%s/%s" % (self.baseWorkDir, self.mozillaSrcDir)
> 
> Definitely need some documentation if these differ from the en-US build.

See comment above.
 
> >+                     'if [ ! -e %s/dist/host/bin/mbsdiff ]; then ' % self.mozillaObjdir +
> >+                     'make -C nsprpub; make -C config;' +
> >+                     'make -C modules/libmar; make -C modules/libbz2;' +
> >+                     'make -C other-licenses/bsdiff;'
> >+                     'fi'],
> >+            description=['make', 'bsdiff'],
> 
> How much overhead does this add ? Existing l10n nightlies have already built in
> nsprpub, config, libmar IIRC, and we're up to 11 minutes per locale on windows.

Not much. Yes, everything up to limber *should* already be build, and libbz2 and bsdiff are quick (about 20s and 10s respectively in casual, worst-case testing (Windows). I do check whether mbsdiff is already built, so using a smaller pool of slaves for l10n may buy us a little win here.
 
> >@@ -3219,38 +3564,42 @@ class ReleaseUpdatesFactory(ReleaseFacto
> >-            self.addStep(ShellCommand(
> >-             name='upload_snipets',
> >+            self.addStep(ShellCommand,
> >+             name='upload_snippets',
> >              command=['rsync', '-av', localDir + '/',
> >                       '%s@%s:%s' % (self.ausUser, self.ausHost, snippetDir)],
> 
> For completeness we should add 
>   '-e', 'ssh -i ~/.ssh/%s' % self.ausSshKey
> after the '-av'.

I'll fix that up.
 
> >diff --git a/steps/updates.py b/steps/updates.py
> >-from buildbot.steps.shell import WithProperties
> >+from buildbot.steps.shell import ShellCommand, WithProperties, SetProperty
> 
> What are these extra functions used for ?

Left over from an earlier version of this patch. I'll remove them. Good catch.

Comment 40

[Robert Kaiser]

(In reply to comment #39)
> > @@ -451,6 +471,15 @@ class MercurialBuildFactory(MozillaBuild
> > ...
> > >+        # These following variables are useful for sharing build steps with
> > >+        # classes that don't use object dirs, e.g. update generation
> > >+        self.mozillaSrcDir = '.'
> > >+        self.fullSrcDir = self.baseWorkDir
> > >+        self.fullObjdir = '%s/%s' % (self.baseWorkDir, self.mozillaObjdir)
> > 
> > Could you add some documentation for these properties ? When you haven't looked
> > at this in some time it's not clear if they're related to what
> > SeaMonkey/Thunderbird need for their two-repo builds. 
> 
> Although I originally determined I'd needed some vars like this when I started
> this work back in the fall, mozillaSrcDir and mozillaObjDir are actually
> properties added by Armen as part of bug 531275.

In this place, yes, but they have been along much longer in other places - their purpose is to enable Thunderbird/SeaMonkey to point to *different* base directories for application and platform files, while Firefox has the same one.

> All I'm doing here is making
> sure that:
> a) they exist for the default Firefox case, because they already exist for
> l10n, SeaMonkey, etc; and,
> b) appending the baseWorkDir so I don't have to do that everywhere. Perhaps the
> var names aren't ideal, but I'm open to suggestions.

It's fine (though even more complicating) to have absolute paths around as well, but please always make sure that "mozilla" is part of the name when it points to the *platform* directory and not when it's pointing to the *application* directory.

Maybe the name would better be absSrcDir/absObjdir and possibly abdMozillaSrcDir/absMozillaObjdir if neeed.

In any case, I'll have to try this patch on the SeaMonkey master once it becomes mostly final to make sure the paths work fine in the "comm-central" case.

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #37)
> * confirming empty snippets are unnecessary

Turns out AUS _is_ expecting them to be present. The getLatestComplete function
 http://mxr.mozilla.org/webtools/source/aus/xml/inc/patch.class.php#426
attempts to determine the current complete mar, by taking the second most recent YYYYMMDDHHMM directory (line 455). FWIW I'm planning to do away with this in bug 517947 but don't know when that will be ready.

Comment 42

[Chris Cooper [:coop] (he/him)]

(In reply to comment #40)
> It's fine (though even more complicating) to have absolute paths around as
> well, but please always make sure that "mozilla" is part of the name when it
> points to the *platform* directory and not when it's pointing to the
> *application* directory.
> 
> Maybe the name would better be absSrcDir/absObjdir and possibly
> abdMozillaSrcDir/absMozillaObjdir if neeed.

I hesitated to use "absolute" to define these paths because they are still relative to the buildbot slave working dir. There are other places where we need to have the true absolute path (via our pwd SetProperty hack).

Again though, this was merely a convenience for me to not have to concatenate multiple vars everywhere. If others think that the "abs" prefix is more appropriate, I'll go with it.

Comment 43

[Chris Cooper [:coop] (he/him)]

(In reply to comment #41)
> (In reply to comment #37)
> > * confirming empty snippets are unnecessary
> 
> Turns out AUS _is_ expecting them to be present. The getLatestComplete function
>  http://mxr.mozilla.org/webtools/source/aus/xml/inc/patch.class.php#426
> attempts to determine the current complete mar, by taking the second most
> recent YYYYMMDDHHMM directory (line 455). FWIW I'm planning to do away with
> this in bug 517947 but don't know when that will be ready.

OK then, I'll leave the steps in but combine the remote empty snippet creation ones as suggested in your previous comment.

Comment 44

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v3

Addresses review comments from nthomas and KaiRo, specifically:
* moves update related vars together
* switch to using absMozillaSrcDir and absMozillaObjDir vars (vs full* vars before)
* preforms a grep in the latest-%branch% dir on stage to determine what the previous complete mar is rather than assuming the filename is the same as the current one. This should keep us working across version bumps.
* condensed empty snippet creation into a single step

This patch has tested fine in staging for linux/mac/win32. I don't think I've broken l10n with this new patch, but I have a series of nightlies kicking off on sm02 in a few minutes to verify that.

Comment 45

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v4

Sorry for the churn, but I discovered pretty early on in my l10n testing that I hadn't subclassed getCompleteMarPatternMatch() to append the locale to the pattern match. Fixed now.

Comment 46

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 431761 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v3

Ok, here are a few comments anyway.

>diff --git a/process/factory.py b/process/factory.py
>@@ -532,16 +563,21 @@ class MercurialBuildFactory(MozillaBuild
>              command=['hg', 'up', '-C', '-r', self.buildRevision],
>              haltOnFailure=True
>             )
>             self.addStep(SetProperty,
>              name='set_got_revision',
>              command=['hg', 'identify', '-i'],
>              property='got_revision'
>             )
>+        self.addStep(SetProperty,
>+            name='set_got_revision',
>+            command=['hg', 'identify', '-i'],
>+            property='got_revision'
>+        )

What is this for ? Leakage from another bug ?

>         if self.productName == 'xulrunner':
>             self.addStep(SetProperty(
>-                command=['python', 'build%s/config/printconfigsetting.py' % self.mozillaDir,
>-                         'build/%s/dist/bin/platform.ini' % self.mozillaObjdir,
>+                command=['python', '%s/config/printconfigsetting.py' % self.absMozillaSrcDir,
>+                         '%s/dist/bin/platform.ini' % self.absMozillaObjDir,

Swapping build%(mozillaDir)s for %(absMozillaSrcDir)s is going to break Thunderbird/SeaMonkey isn't it ? While this example is in xulrunner, there are other cases where this substitution is made too. My intention wasn't to push for reconciling the variables we have for comm-central and l10n support, just documenting them.

Comment 47

[Chris Cooper [:coop] (he/him)]

Comment on attachment 431787 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v4

(In reply to comment #46)
> Swapping build%(mozillaDir)s for %(absMozillaSrcDir)s is going to break
> Thunderbird/SeaMonkey isn't it ? While this example is in xulrunner, there are
> other cases where this substitution is made too. My intention wasn't to push
> for reconciling the variables we have for comm-central and l10n support, just
> documenting them.

Sigh, yes, you're right. If I'm serious about using baseWorkDir everywhere I should be filing a separate bug and not trying to bolt it in as part of this patch. Or at least I should if I want a timely review here.

I'll post an updated patch shortly.

Comment 48

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v5

Same as v4, modulo:
* debugging stanza removed
* reigned in variable replacement for absMozilla{Src|Obj}Dir
* improved comments

Comment 49

[Robert Kaiser]

Comment on attachment 431905 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v5

>diff --git a/misc.py b/misc.py
>--- a/misc.py
>+++ b/misc.py
>@@ -707,21 +707,24 @@ def generateBranchObjects(config, name):

Please do the same changes to generateCCBranchObjects while you're touching this file.
This is right now only used by SeaMonkey, for which I'll care about the config changes once I'm ready to update the master with it.

>+        # These following variables are useful for sharing build steps (e.g.
>+        # update generation) with subclasses that don't use object dirs (e.g.
>+        # l10n repacks).
>+        # 
>+        # We also concatenate the baseWorkDir at the outset to avoid having to
>+        # do that everywhere.
>+        self.mozillaSrcDir = '.'
>+        self.absMozillaSrcDir = self.baseWorkDir

Sorry, but those definitions are wrong. Both are pointing to application directories, not platform/mozilla directories.

This would be correct:
        self.srcDir = '.'
        self.absSrcDir = self.baseWorkDir
        self.mozillaSrcDir = '.%s' % self.mozillaDir
        self.absMozillaSrcDir = '%s%s' % (self.baseWorkDir, self.mozillaDir)

Whatever you actually need of those is what you should use (note that self.mozillaDir is '' for Firefox but '/mozilla' for SeaMonkey and Thunderbird, and that dist/ is always in the *mozilla* directories).

>+        self.addStep(ShellCommand(
>+            name='cat_complete_snippet',
>+            command=['cat', 'complete.update.snippet'],
>+            workdir='%s/%s/dist/update' % (self.baseWorkDir,
>+                                           self.mozillaObjdir),
>+        ))

Nit: I think that workdir can use self.absMozillaObjdir, right?

>+        self.addStep(ShellCommand(
>+            name='upload_complete_snippet',
>+            command=['scp', '-o', 'User=%s' % self.ausUser,
>+                     '-o', 'IdentityFile=~/.ssh/%s' % self.ausSshKey,
>+                     'dist/update/complete.update.snippet',
>+                     WithProperties('%s:%s/complete.txt' % (self.ausHost,
>+                                                            self.ausFullUploadDir))],
>+             workdir='build/%s' % self.mozillaObjdir,
>+             description=['upload', 'complete', 'snippet'],
>+             haltOnFailure=True,
>+        ))

Same here?

>+            self.addStep(SetProperty(
>+                name='make_upload',
>+                command=uploadCmd,
>+                env=uploadEnv,
>+                workdir=self.absMozillaObjDir,
>+                extract_fn = get_url,
>+                haltOnFailure=True,
>+                description=['make', 'upload'],
>+            ))

May work, but here it should better be a self.absObjdir that is built with self.objdir instead of self.mozillaObjdir, as right now we're also calling this on the application level and not the Mozilla level (probably makes no difference, though, but if we switch this, we need to switch it everywhere we use "make upload").

>+        # We also concatenate the baseWorkDir at the outset to avoid having to
>+        # do that everywhere.
>+        self.absMozillaSrcDir = "%s/%s" % (self.baseWorkDir, self.mozillaSrcDir)
>+        self.absMozillaObjDir = '%s/%s' % (self.baseWorkDir, self.mozillaObjdir)

Ah, yes, those should work fine!

>+            # We need the appVersion to create snippets.
>+            self.addStep(FindFile(
>+                name='find_inipath',
>+                filename='application.ini',
>+                directory='dist/l10n-stage',
>+                filetype='file',
>+                max_depth=5,
>+                property_name='inipath',
>+                workdir=self.absMozillaSrcDir,
>+                haltOnFailure=True,
>+            ))

application.ini is in the *source* tree, not the *object* tree? I thought it was below absMozillaObjdir rather than absMozillaSrcDir here (note that this may not make a difference right now, but Armen plans to (correctly) make us use objdirs for repacks everywhere in the future).

(oh, and the printconfigsetting.py call below is correct, that file is in the source.)

>+        self.addStep(ShellCommand(
>+            name='make_bsdiff',
>+            command=['sh', '-c',
>+                     'if [ ! -e %s/dist/host/bin/mbsdiff ]; then ' % self.mozillaObjdir +
>+                     'make -C nsprpub; make -C config;' +
>+                     'make -C modules/libmar; make -C modules/libbz2;' +
>+                     'make -C other-licenses/bsdiff;'
>+                     'fi'],
>+            description=['make', 'bsdiff'],
>+            workdir=self.absMozillaSrcDir,
>+            haltOnFailure=True,
>+        ))

Any make call should be made from the absMozillaObjdir side, not directly in the source tree (if objdir is set, else the vars result to be the same anyhow).

>+        self.addStep(ShellCommand(name='rm_en-US_build',
>+                                  command=['bash', '-c', 'rm -rvf *.en-US.*'],
>+                                  description=['remove','en-US','build'],
>+                                  env=self.env,
>+                                  workdir='%s/%s/dist' % (self.baseWorkDir, self.origSrcDir),
>+                                  haltOnFailure=True)
>+         )

Don't we save the en-US builds to |'%/dist' % self.absMozillaObjdir| actually?


Still, this looks like a ton of really good work, I'm looking forward to trying and hopefully using it!

Comment 50

[Chris Cooper [:coop] (he/him)]

The code as-is doesn't work for branches on which we're turning on l10n updates for the first time (e.g. 1.9.1). I'm working on a fix in staging right now.

Comment 51

[Chris Cooper [:coop] (he/him)]

Attachment: Upload l10n partial mars when they exist [Checkin: Comment 61 & 74]

This is the same patch we took for the en-US 'make upload' case, but applied to l10n. It degrades gracefully when no partial mar is present.

Comment 52

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 431905 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v5

coop says he has another patch on the way, which addresses KaiRo's comments and some l10n issues. The changes here compared to v2 look ok from a quick glance.

>diff --git a/process/factory.py b/process/factory.py
>-        self.addStep(SetProperty,
>-         command=['bash', '-c', 'pwd'],
>-         property='toolsdir',
>-         workdir='tools'
>-        )

There's a merge to be done for the toolsdir changes following bug 549894.

Comment 53

[Nick Thomas [:nthomas] (UTC+12)]

Oh, and we talked about conditionally generating the partial mar and snippet if a previous complete mar could be found. Buildbot 0.7.11 supports a doStepIf parameter on a step, which is a boolean or function returning one. We have 0.7.10p1 right now so that'd have to be a future enhancement. In the meantime, for new branches we can enable createSnippet, and then createPartial a day later.

Comment 54

[Chris Cooper [:coop] (he/him)]

(In reply to comment #52)
> There's a merge to be done for the toolsdir changes following bug 549894.

I have an updated version of my patch that addresses the toolsdir changes running in staging overnight. 

(In reply to comment #53)
> Oh, and we talked about conditionally generating the partial mar and snippet if
> a previous complete mar could be found. Buildbot 0.7.11 supports a doStepIf
> parameter on a step, which is a boolean or function returning one. We have
> 0.7.10p1 right now so that'd have to be a future enhancement. In the meantime,
> for new branches we can enable createSnippet, and then createPartial a day
> later.

Yep. The new patch has added l10n-specific config vars for create_partial and aus_base_upload_dir to allow us to create l10n complete mars without necessarily also trying to generate partials.

One-off tests of linux en-US and locales have succeeded thus far, modulo the l10n partial mar upload for which we still need attachment 432984 [details] [diff] [review]. If all goes well on the other platform nightlies, expect a review request tomorrow.

Comment 55

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v6

Here are the changes from the previous patch version (v5):

* added config vars create_partial_l10n and aus_base_upload_dir_l10n so that l10n partials can be toggled off independently from en-US partials
* addressed KaiRo's comments re: absMozilla* var naming
* fixed up toolsdir SetProperty commands that have been added since v5
* added marFilenameToProperty function
* allow addCreatePartialUpdateSteps to take an optional extraArgs. This is needed for the l10n case to work properly (need to pass AB_CD=xx-XX)
* fix make_upload commands. We don't need to set PARTIAL_MAR_FILENAME, the make command figures out whether a partial mar is present on its own.
* reorganize find_inipath and associated get_build_id steps for l10n. We need the buildid property set for uploads to work properly.

This patch has worked for me in staging, generating complete and partial mars for en-US and l10n on all three platforms.

Comment 56

[Chris Cooper [:coop] (he/him)]

Attachment: Required config changes to allow building partial mars on build slaves, v3 [Checkin: Comment 62]

A small delta from the previous config patch, adding the new create_partial_l10n and aus2_base_upload_dir_l10n vars.

I've also turned on updates-on-slaves for m-c in production with this patch too.

Comment 57

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 433426 [details] [diff] [review]
Required config changes to allow building partial mars on build slaves, v3
[Checkin: Comment 62]

This r+ is dependent on l10n pushing to the right place ....

Comment 58

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 433422 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v6

Closer and closer, a few nits and one issue.

>diff --git a/process/factory.py b/process/factory.py
> class MercurialBuildFactory(MozillaBuildFactory):
>     def __init__(self, env, objdir, platform, configRepoPath, configSubDir,
>+            if self.createPartial:
>+                # e.g.:
>+                # /opt/aus2/incoming/2/Firefox/mozilla2/WINNT_x86-msvc

s/mozilla2/mozilla-central/ 4 times to correct an earlier error.

>+        self.srcDir = '.'
>+        self.absSrcDir = self.baseWorkDir

These two don't seem to end up used anywhere now.

>+        # Need to override toolsdir as set by MozillaBuildFactory because
>+        # we need Windows-style paths.
>+        if self.platform == 'win32':
>+            self.addStep(SetProperty,
>+                command=['bash', '-c', 'pwd -W'],
>+                property='toolsdir',
>+                workdir='tools'

I suspect that this pwd -W can be done once in MozillaBuildFactory but that's for another day.

>+    def addUploadSnippetsSteps(self):
>+        self.addStep(ShellCommand(
>+            name='upload_complete_snippet',
...
>+             workdir='build/%s' % self.mozillaObjdir,

This can be absMozillaObjDir.

>+def marFilenameToProperty(prop_name=None):
>+    '''Parse a file listing and return the first mar filename found as
>+       a named property.
>+    '''

A function to return a function eh ? Do you think you'll need this for more than previousMarFilename in future ? I find the get_url style easier to read, but it's not a game breaker.

> class NightlyBuildFactory(MercurialBuildFactory):
>+    def addUploadSnippetsSteps(self):
>+        if self.createPartial:
>+            ausCurrentBuildUploadDir = "%s/%%(buildid)s/en-US" % \
>+                                         self.ausFullUploadDir
>+            self.addStep(ShellCommand(
>+                name='create_aus_current_updir',
...
>+            self.addStep(ShellCommand(
>+                name='upload_partial_snippet',
...
>+            # Create remote empty complete/partial snippets for current build.
>+            self.addStep(ShellCommand(
>+                name='create_empty_snippets',

Please move the create_aus_current_dir step below upload_partial_snippet, so that it's next to create_empty_snippets.

>+class NightlyRepackFactory(BaseRepackFactory, NightlyBuildFactory):
...
>+            if self.createPartial:
>+                # e.g.:
>+                # /opt/aus2/incoming/2/Firefox/mozilla2/WINNT_x86-msvc
>+                self.ausFullUploadDir = '%s/%s' % (self.ausBaseUploadDir,
>+                                                   self.updatePlatform)

I can't find where this is appended in l10n land to add the correct locale, and ...

>+            NightlyBuildFactory.addUploadSnippetsSteps(self)

... this guy hard codes en-US. If I've found the right testing run at 
  http://staging-master.build.mozilla.org:8011/builders/Firefox%20mozilla-1.9.1%20win32%20l10n%20nightly/builds/1
(the last build on the master as of now) then it's all the AUS dir creation and snippet uploads is en-US, but that's a fr build.

Comment 59

[Axel Hecht [:Pike]]

Comment on attachment 432984 [details] [diff] [review]
Upload l10n partial mars when they exist
[Checkin: Comment 61 & 74]

I assume that Nick's comment on "upload l10n to the right place" doesn't involve this patch, so r=me.

Comment 60

[Chris Cooper [:coop] (he/him)]

Attachment: buildbotcustom changes to create partials on slaves, v7 [Checkin: Comment 63]

Fixed all nits.

(In reply to comment #58)
> >+def marFilenameToProperty(prop_name=None):
> >+    '''Parse a file listing and return the first mar filename found as
> >+       a named property.
> >+    '''
> 
> A function to return a function eh ? Do you think you'll need this for more
> than previousMarFilename in future ? I find the get_url style easier to read,
> but it's not a game breaker.

I was trying not to limit it to partial mars only and followed Axel's identToProperties example to do so. There's nowhere else we would use the function currently though.

> I can't find where this is appended in l10n land to add the correct locale, 

Fixed by adding a getPreviousBuildUploadDir() function that is overridden in the l10n case.

Here are the relevant staging run steps that show this:

en-US:
http://staging-master.build.mozilla.org:8011/builders/Linux%20mozilla-1.9.1%20nightly/builds/70/steps/upload_complete_snippet/logs/stdio

l10n (fr):
http://staging-master.build.mozilla.org:8011/builders/Firefox%20mozilla-1.9.1%20linux%20l10n%20nightly/builds/1513/steps/upload_complete_snippet/logs/stdio

Comment 61

[Chris Cooper [:coop] (he/him)]

Comment on attachment 432984 [details] [diff] [review]
Upload l10n partial mars when they exist
[Checkin: Comment 61 & 74]

http://hg.mozilla.org/mozilla-central/rev/56bb7f290c36

Comment 62

[Chris Cooper [:coop] (he/him)]

Comment on attachment 433426 [details] [diff] [review]
Required config changes to allow building partial mars on build slaves, v3
[Checkin: Comment 62]

http://hg.mozilla.org/build/buildbot-configs/rev/3cb1a8198039

Comment 63

[Chris Cooper [:coop] (he/him)]

Comment on attachment 433579 [details] [diff] [review]
buildbotcustom changes to create partials on slaves, v7
[Checkin: Comment 63]

http://hg.mozilla.org/build/buildbotcustom/rev/cbb6d748a5de

Comment 64

[Chris Cooper [:coop] (he/him)]

All changes have been pushed live. Partial update generation on slaves is currently enabled only for mozilla-central, pending approval for other branches.

I'll be monitoring tomorrow's nightlies on m-c and making adjustments as necessary.

Comment 65

[Chris Cooper [:coop] (he/him)]

Attachment: Don't copy partial mars to latest in post_upload.py [Checkin: Comment 66]

This change is already in place on stage to prevent more partials from ending up in latest-* like we saw this morning.

Comment 66

[Chris Cooper [:coop] (he/him)]

Comment on attachment 433808 [details] [diff] [review]
Don't copy partial mars to latest in post_upload.py
[Checkin: Comment 66]

http://hg.mozilla.org/build/tools/rev/1178f35ab548

Comment 67

[Chris Cooper [:coop] (he/him)]

Attachment: Touch the platform dir on the AUS server as the final step in uploading snippets

Comment 68

[Chris Cooper [:coop] (he/him)]

Attachment: Touch the platform dir on the AUS server as the final step in uploading snippets, v2 [Checkin: Comment 70]

Added the touch to the previous step to avoid an extra step.

Comment 69

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 434351 [details] [diff] [review]
Touch the platform dir on the AUS server as the final step in uploading snippets, v2
[Checkin: Comment 70]

Looks fine.

Comment 70

[Chris Cooper [:coop] (he/him)]

Comment on attachment 434351 [details] [diff] [review]
Touch the platform dir on the AUS server as the final step in uploading snippets, v2
[Checkin: Comment 70]

http://hg.mozilla.org/build/buildbotcustom/rev/ab0eb581b33f

production-masters have been reconfig-ed with this changed.

Comment 71

[Chris Cooper [:coop] (he/him)]

Latest patch seems to have worked. I got my update on the first attempt this morning.

Now we just need those approvals for the other branches.

Comment 72

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #66)
> (From update of attachment 433808 [details] [diff] [review])
> http://hg.mozilla.org/build/tools/rev/1178f35ab548

Updated staging-stage with this.

Comment 73

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 432984 [details] [diff] [review]
Upload l10n partial mars when they exist
[Checkin: Comment 61 & 74]

a=beltzner for whatever :)

Comment 74

[Chris Cooper [:coop] (he/him)]

Comment on attachment 432984 [details] [diff] [review]
Upload l10n partial mars when they exist
[Checkin: Comment 61 & 74]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/043ff4072e03
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/f6b1bf4e8ff1

Comment 75

[Chris Cooper [:coop] (he/him)]

Attachment: [buildbot-configs] Generate updates on slaves for 1.9.2 [Checkin: Comment 77]

Comment 76

[Armen [:armenzg]]

Comment on attachment 435713 [details] [diff] [review]
[buildbot-configs] Generate updates on slaves for 1.9.2
[Checkin: Comment 77]

Looks good.

Beside the flags I see you are changing build/0 for incoming/2.

Comment 77

[Chris Cooper [:coop] (he/him)]

Comment on attachment 435713 [details] [diff] [review]
[buildbot-configs] Generate updates on slaves for 1.9.2
[Checkin: Comment 77]

http://hg.mozilla.org/build/buildbot-configs/rev/d71c7c15a467

Comment 78

[Chris Cooper [:coop] (he/him)]

Turned this on for 1.9.2 this morning.

Comment 79

[Robert Kaiser]

Attachment: [buildbotcustom] make this work for CC* builders [Checkin: Comment 81]

Here's the patch needed to make partial creation work on CC* builders (SeaMonkey/Thunderbird).
It's mainly just a workdir correction and some adjustment to the inheritance chain for CCNightlyBuildFactory. I also updates one comment as I now can explain why the override there is needed.

Comment 80

[Chris Cooper [:coop] (he/him)]

Comment on attachment 435972 [details] [diff] [review]
[buildbotcustom] make this work for CC* builders
[Checkin: Comment 81]

Looks good, land this at your leisure. I'll plan to reconfig our masters with the change tomorrow morning.

Comment 81

[Robert Kaiser]

Comment on attachment 435972 [details] [diff] [review]
[buildbotcustom] make this work for CC* builders
[Checkin: Comment 81]

Thanks, landed as http://hg.mozilla.org/build/buildbotcustom/rev/da9e888f3189

Comment 82

[Mike Beltzner [:beltzner, not reading bugmail]]

Moving flag to .10-fixed as per comment 74

Comment 83

[Nick Thomas [:nthomas] (UTC+12)]

We never turned this on for 1.9.1 but looks like all the machinery landed. Fair enough that we don't need l10n nightly updates for 1.9.1 (511510), but we could get en-US off prometheus-vm. Am I missing something ?

Comment 84

[Chris Cooper [:coop] (he/him)]

(In reply to comment #83)
> We never turned this on for 1.9.1 but looks like all the machinery landed. Fair
> enough that we don't need l10n nightly updates for 1.9.1 (511510), but we could
> get en-US off prometheus-vm. Am I missing something ?

Sure, that would leave us with only Fx 3.0 and Tb 2.0 using prometheus-vm, and hopefully we'll be able to shut those off soon.

Comment 85

[Chris Cooper [:coop] (he/him)]

Attachment: Generate en-US nightly updates on slaves

Comment 86

[Chris Cooper [:coop] (he/him)]

Comment on attachment 453170 [details] [diff] [review]
Generate en-US nightly updates on slaves

http://hg.mozilla.org/build/buildbot-configs/rev/a05483b93162

Comment 87

[Chris Cooper [:coop] (he/him)]

en-US partials for 1.9.1 were generated on the slaves last night.