773290 - multiple partial MAR support in release automation

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Comment 1

[[e-mail] (:bhearsum)]

Attachment: factory.py, for marking up

Comment 2

[[e-mail] (:bhearsum)]

Attachment: factory.py, for marking up

Comment 3

[[e-mail] (:bhearsum)]

In terms of the release configs I think we should get rid of the old* variables in the release configs and replace them with a 'partialUpdates' dict. These variables are exclusively used for update generation and generating bouncer product names, so partialUpdates is pretty accurate. An example of that dict is:
partialUpdates = {
  '13.0.1': {
    'buildNumber': 1,
    'baseTag': FIREFOX_13_0_1',
    'mostRecent': True
  },
  '12.0': {
    'buildNumber': 1,
    'baseTag': FIREFOX_12_0',
  },
}

appVersion should also be allowed, but it should default to the key name. mostRecent is needed because we need to know what to set "from" to in the patcher config, and that's going to continue to be a singular value for now.

Speaking of patcher, we need to adjust that config format too. I don't want to make any changes to it that require changing patcher, because that's always risky and not really worth the effort. I think we should add a new 'partials' section to the <current-update> block. Each subsection inside of it would represent a version we want a partial for, and would contain the necessary URLs. For example:
<partials>  
    <12.0>
        betatest-url   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-12.0-14.0.partial.mar
        esrtest-url   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-12.0-14.0.partial.mar
        path   firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-12.0-14.0.partial.mar
        url   http://download.mozilla.org/?product=firefox-14.0-partial-12.0&os=%bouncer-platform%&lang=%locale%
    </12.0>
    <13.0.1>
        betatest-url   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-13.0.1-14.0.partial.mar
        esrtest-url   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-13.0.1-14.0.partial.mar
        path   firefox/nightly/14.0-candidates/build1/update/%platform%/%locale%/firefox-13.0.1-14.0.partial.mar
        url   http://download.mozilla.org/?product=firefox-14.0-partial-13.0.1&os=%bouncer-platform%&lang=%locale%
    </13.0.1>
</partials>

The update verify configs already support testing partials from any version, so we don't need to adjust those at all.

Comment 4

[[e-mail] (:bhearsum)]

I'm going to be heading up this effort.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Sounds like a good plan to me. What do we use baseTag for on the older releases? Maybe it can be dropped now.

Is the new python script going to generate all the snippets or just the extra partials? The new partials block looks fine for the former, but doesn't need the most recent info for the latter. It occurs to me that using two separate utils might be a source of problems and the new script might as well do all snippets, assuming we already have partials and checksums from build time.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> Sounds like a good plan to me. What do we use baseTag for on the older
> releases? Maybe it can be dropped now.

It's still used in ReleaseUpdatesFactory during the patcher config bump process - we need it to pull the correct shipped-locales for the previous release.

> Is the new python script going to generate all the snippets or just the
> extra partials? The new partials block looks fine for the former, but
> doesn't need the most recent info for the latter. It occurs to me that using
> two separate utils might be a source of problems and the new script might as
> well do all snippets, assuming we already have partials and checksums from
> build time.

That's a good point. I might even do you one better and suggest that if we're generating all of the partials through the new tool, that we should consider generating the completes through it, too. I'll look at both of these as part of this work.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #6)
> (In reply to Nick Thomas [:nthomas] from comment #5)
> > Sounds like a good plan to me. What do we use baseTag for on the older
> > releases? Maybe it can be dropped now.
> 
> It's still used in ReleaseUpdatesFactory during the patcher config bump
> process - we need it to pull the correct shipped-locales for the previous
> release.

It occurs to me that you just have been talking about release that we'll be generating partials for that aren't the mostRecent one. Those ones could omit baseTag AFAICT.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 641476 [details] [diff] [review]
factory.py, for marking up

Review of attachment 641476 [details] [diff] [review]:
-----------------------------------------------------------------

Most of the buildbotcustom changes are in factory.py, which I've outlined below. Additionally, a bunch of arg passing in release.py needs to be updated, as well as ripping out MajorUpdateFactory from it. scheduler.py needs updated arg passing, too, in TriggerBouncerCheck.poll().

::: factory.py
@@ +2789,5 @@
> +class ReleaseBuildFactory(MercurialBuildFactory):
> +    def __init__(self, env, version, buildNumber, brandName=None,
> +            unittestMasters=None, unittestBranch=None, talosMasters=None,
> +            usePrettyNames=True, enableUpdatePackaging=True, oldVersion=None,
> +            oldBuildNumber=None, appVersion=None, **kwargs):

Anything currently using oldVersion needs to accept a new 'partialVersions' variable instead. See comments on the release config patch for details on it. oldBuildNumber needs to be removed completely.

@@ +2823,5 @@
> +                               fileType=None, maxDepth=1, haltOnFailure=False):
> +        # We don't need to do this for release builds.
> +        pass
> +
> +    def addCreatePartialUpdateSteps(self):

Roughly, this method needs to be modified to loop over the new partialVersions variable instead of just oldVersion.

@@ +4584,5 @@
> +    snippetStagingDir = '/opt/aus2/snippets/staging'
> +    def __init__(self, cvsroot, patcherToolsTag, patcherConfig, verifyConfigs,
> +                 appName, productName,
> +                 version, appVersion, baseTag, buildNumber,
> +                 oldVersion, oldAppVersion, oldBaseTag,  oldBuildNumber,

oldBaseTag and oldBuildNumber should be removed. Their information will be in the new partialVersions dict, which needs to be added here.

@@ +4668,5 @@
> +        self.shippedLocales = self.getShippedLocales(self.repository, baseTag,
> +                                                appName)
> +        self.oldShippedLocales = self.getShippedLocales(self.oldRepository,
> +                                                        self.oldBaseTag,
> +                                                        self.appName)

Needs to be turned into a dict with all of the old versions we need.

@@ +4672,5 @@
> +                                                        self.appName)
> +        self.candidatesDir = self.getCandidatesDir(productName, version, buildNumber)
> +        self.updateDir = 'build/temp/%s/%s-%s' % (productName, oldVersion, version)
> +        self.marDir = '%s/ftp/%s/nightly/%s-candidates/build%s' % \
> +          (self.updateDir, productName, version, buildNumber)

We don't do any MAR generation here anymore, we should just rip out marDir and everything that uses it.

@@ +4777,5 @@
> +         name='get_old_shipped_locales',
> +         command=['wget', '-O', 'old-shipped-locales', self.oldShippedLocales],
> +         description=['get', 'old-shipped-locales'],
> +         haltOnFailure=True
> +        ))

Needs to loop over self.partialVersions.

@@ +4787,5 @@
> +                       '-v', self.version, '-a', self.appVersion,
> +                       '-o', self.oldVersion, '-b', str(self.buildNumber),
> +                       '-c', WithProperties(self.patcherConfigFile),
> +                       '-t', self.stagingServer, '-f', self.ftpServer,
> +                       '-d', self.bouncerServer, '-l', 'shipped-locales']

This call needs to be updated to pass along all partialVersions.

@@ +4924,5 @@
> +         name='patcher_create_patches',
> +         command=command,
> +         description=['patcher:', 'create patches'],
> +         haltOnFailure=True
> +        ))

We need to call the new script (new TBD) right after this one.

@@ +4939,5 @@
> +             name='switch_appv_in_snippets',
> +             command=cmd,
> +             haltOnFailure=True,
> +             workdir=self.updateDir,
> +            ))

This one-off hack should be ripped out too.

@@ +4949,5 @@
> +                   '--product', self.productName,
> +                   '--app-name', self.appName,
> +                   '--version', self.version,
> +                   '--app-version', self.appVersion,
> +                   '--old-version', self.oldVersion,

Need to update this call.

@@ +4978,5 @@
> +         flunkOnFailure=False,
> +         workdir=self.updateDir
> +        ))
> +
> +    def uploadMars(self):

This method can be completely ripped out.

@@ +5077,5 @@
> +                '-n', str(self.buildNumber), '-a', self.ausServerUrl,
> +                '-s', self.stagingServer, '-c', verifyConfigPath,
> +                '-d', oldCandidatesDir, '-l', 'shipped-locales',
> +                '--old-shipped-locales', 'old-shipped-locales',
> +                '--pretty-candidates-dir', '--channel', self.testChannel]

This call needs updating too.

@@ +5092,5 @@
> +                                          self.buildNumber)
> +
> +
> +
> +class MajorUpdateFactory(ReleaseUpdatesFactory):

This factory isn't used anymore, it can go.

@@ +5229,5 @@
> +class TuxedoEntrySubmitterFactory(ReleaseFactory):
> +    def __init__(self, baseTag, appName, config, productName, version,
> +                 tuxedoServerUrl, enUSPlatforms, l10nPlatforms,
> +                 extraPlatforms=None, bouncerProductName=None, brandName=None,
> +                 oldVersion=None, credentialsFile=None, verbose=True,

Needs to accept partialVersions instead of oldVersion.

@@ +5260,5 @@
> +        cmd.extend(['--brand-name', brandName])
> +
> +        if oldVersion:
> +            cmd.append('--add-mars')
> +            cmd.extend(['--old-version', oldVersion])

...and pass along those versions to the script.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: support multiple partials in buildbotcustom; rip out a bunch of dead stuff

Easy stuff first...I ripped out a bunch of dead code while looking for references to CVS, including:
* BootstrapFactory (dead since 3.0 was EOL'ed)
* steps/tryserver.py (dead since try server was made a mozilla-central based branch)
* a few dead cvs-related variables in steps/unittest.py

As well as some updates related things:
* MajorUpdateFactory (dead since we did the last 3.6 MU.)
* MAR generation in ReleaseUpdatesFactory (dead since partials @ build time)
* binaryName/oldBinaryName (only used for the "Mozilla Developer Preview" builds, which are long dead)
* fakeMacInfoTxt (a transitionary switch, dead since 3.6 died)
* one-off hack for Firefox 11.0

Now, the actual changes:
* Replace old* with partialUpdates in ReleaseBuildFactory, loop over it when creating partials
* Use perl modules/patcher configs from build/tools - no more CVS \o/.
* Adjust patcher-config-bump.pl command in accordance with the patch from bug 775535.
* Adjust tuxedo factory for proposed changes in bug 775526.

Todo:
* Change update verify bump command once we have a patch for bug 775541

Comment 10

[[e-mail] (:bhearsum)]

Attachment: support multiple partials in l10n scripts; copy patcher configs/modules

Most of this patch is a simple import of the Perl modules that patcher-config-bump.pl still needs to function, minus the changes in bug 775535. The patcher-configs that we care about are imported as well, but with the <partial> block being replaced by <partials>, and the fixes from https://bug775236.bugzilla.mozilla.org/attachment.cgi?id=646696.

The only other changes are simple s/old*/partialUpdates/g, which is very similar to the ReleaseBuildFactory changes.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: change release configs, drop dead stuff

Really simple stuff here. Just removing dead variables (including everything related to CVS) and adding in partialUpdates.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 651754 [details] [diff] [review]
support multiple partials in buildbotcustom; rip out a bunch of dead stuff

Review of attachment 651754 [details] [diff] [review]:
-----------------------------------------------------------------

In overall looks good to me. See some comments inline.

::: process/factory.py
@@ -244,5 @@
>      for ss, user, passwd in signingServers:
>          cmd.extend(['-H', ss])
>      return ' '.join(cmd)
>  
> -class BootstrapFactory(BuildFactory):

DIAF!!!

@@ +3950,3 @@
>  
> +        # The patcher config bumper needs to know the exact previous version
> +        self.previousVersion = max(v for v in self.partialUpdates)

This may hit Y2K problem:
>>> max(['13.0b1', '13.0b3', '15.0b4', '9.0b5'])
'9.0b5'

You either need to use max(..., key=...) or just self.partialUpdates[-1] (or [0] depending on how we sort versions on configs).

Or even better, we should get rid of this parameter and "previous version" concept.

@@ +4052,5 @@
> +#            self.addStep(ShellCommand(
> +#             name='bump_verify_configs',
> +#             command=bumpCommand,
> +#             description=['bump', self.verifyConfigs[platform]],
> +#            ))

TODO: use new update verify bumper here.

@@ +4065,2 @@
>           command=['hg', 'commit', '-u', self.hgUsername, '-m',
> +                  'Automated configuration bump: update configs ' + \

What do you think about moving this step (and maybe diff) out of the loop so we have just one commit instead of one per platform?

@@ +4203,5 @@
>               waitForFinish=False
>              ))
>  
>      def getUpdateVerifyBumpCommand(self, platform):
> +        return ""

This could be dropped since the class is not inherited anymore by major update factory.

::: process/release.py
@@ +1360,5 @@
>              tuxedoServerUrl=releaseConfig['tuxedoServerUrl'],
>              enUSPlatforms=releaseConfig['enUSPlatforms'],
>              l10nPlatforms=releaseConfig['l10nPlatforms'],
>              extraPlatforms=releaseConfig.get('extraBouncerPlatforms'),
> +            partialUpdates=releaseConfig['partialUpdates'],

Could you use releaseConfig.get('partialUpdates', {}) for unification?

@@ +1396,5 @@
>                  milestone=releaseConfig['milestone'],
>                  tuxedoServerUrl=releaseConfig['tuxedoServerUrl'],
>                  enUSPlatforms=('win32-EUballot',),
>                  l10nPlatforms=None, # not needed
> +                partialUpdates=releaseConfig['partialUpdates'],

The same here.

::: steps/tryserver.py
@@ -1,2 @@
> -# ***** BEGIN LICENSE BLOCK *****
> -# Version: MPL 1.1

Die, old code, die!

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #12)
> > +        # The patcher config bumper needs to know the exact previous version
> > +        self.previousVersion = max(v for v in self.partialUpdates)
> 
> This may hit Y2K problem:
> >>> max(['13.0b1', '13.0b3', '15.0b4', '9.0b5'])
> '9.0b5'
> 
> You either need to use max(..., key=...) or just self.partialUpdates[-1] (or
> [0] depending on how we sort versions on configs).

There isn't any "ordering" we can assume because partialUpdates is a dict. Looks like distutils.version.LooseVersion will help us out here.

> Or even better, we should get rid of this parameter and "previous version"
> concept.

Yeah, in Balrog-land (or new-patcher-config-land) it won't be necessary. Right now we still use it during update path calculation.

> @@ +4065,2 @@
> >           command=['hg', 'commit', '-u', self.hgUsername, '-m',
> > +                  'Automated configuration bump: update configs ' + \
> 
> What do you think about moving this step (and maybe diff) out of the loop so
> we have just one commit instead of one per platform?

I'm pretty sure it already is out of the loop...

> @@ +4203,5 @@
> >               waitForFinish=False
> >              ))
> >  
> >      def getUpdateVerifyBumpCommand(self, platform):
> > +        return ""
> 
> This could be dropped since the class is not inherited anymore by major
> update factory.

Good point. I'll get rid of this for the next iteration.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum [:bhearsum] from comment #13)
> (In reply to Rail Aliiev [:rail] from comment #12)
> > > +        # The patcher config bumper needs to know the exact previous version
> > > +        self.previousVersion = max(v for v in self.partialUpdates)
> > 
> > This may hit Y2K problem:
> > >>> max(['13.0b1', '13.0b3', '15.0b4', '9.0b5'])
> > '9.0b5'
> > 
> > You either need to use max(..., key=...) or just self.partialUpdates[-1] (or
> > [0] depending on how we sort versions on configs).
> 
> There isn't any "ordering" we can assume because partialUpdates is a dict.
> Looks like distutils.version.LooseVersion will help us out here.

Isn't this what mostRecent in releaseConfig['partialUpdates'] was for ?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Rail Aliiev [:rail] from comment #12)
> @@ +4065,2 @@
> >           command=['hg', 'commit', '-u', self.hgUsername, '-m',
> > +                  'Automated configuration bump: update configs ' + \
> 
> What do you think about moving this step (and maybe diff) out of the loop so
> we have just one commit instead of one per platform?

+1. Bonus points for only tagging once.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #14)
> (In reply to Ben Hearsum [:bhearsum] from comment #13)
> > (In reply to Rail Aliiev [:rail] from comment #12)
> > > > +        # The patcher config bumper needs to know the exact previous version
> > > > +        self.previousVersion = max(v for v in self.partialUpdates)
> > > 
> > > This may hit Y2K problem:
> > > >>> max(['13.0b1', '13.0b3', '15.0b4', '9.0b5'])
> > > '9.0b5'
> > > 
> > > You either need to use max(..., key=...) or just self.partialUpdates[-1] (or
> > > [0] depending on how we sort versions on configs).
> > 
> > There isn't any "ordering" we can assume because partialUpdates is a dict.
> > Looks like distutils.version.LooseVersion will help us out here.
> 
> Isn't this what mostRecent in releaseConfig['partialUpdates'] was for ?

Yeah...I was going to add that in, and then realized that it could be calculated instead. I'm happy to go back to that if you think it's better that way.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 651764 [details] [diff] [review]
support multiple partials in l10n scripts; copy patcher configs/modules

>diff --git a/lib/python/build/l10n.py b/lib/python/build/l10n.py
>         make_incremental_update = '../../tools/update-packaging/make_incremental_update.sh'
>+        prevMar = partialUpdates[oldVersion]['mar']
>         prevMarDir = '../../../../'
>         if mozillaDir:
>             unwrap_full_update = '../../../../%stools/update-packaging/unwrap_full_update.pl' % mozillaDir
>             make_incremental_update = '../../../%stools/update-packaging/make_incremental_update.sh' % mozillaDir
>             prevMarDir = '../../../../../'

Nit: All except the prevMar can move outside the loop to next to the unwrap_full_update definition. Otherwise looks good.

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 651764 [details] [diff] [review]
support multiple partials in l10n scripts; copy patcher configs/modules

     .  o ..                  
     o . o o.o                
          ...oo               
            __[]__            
         __|_o_o_o\__         
         \""""""""""/         
          \. ..  . /          
     ^^^^^^^^^^^^^^^^^^^^ it!

Comment 19

[[e-mail] (:bhearsum)]

Attachment: unbitrot; rip out majorUpdate references

Should be the last iteration of this.

Comment 20

[[e-mail] (:bhearsum)]

Attachment: address final review comments in tools patch

Just addressing Nick's final comments in this version. Metadiff is:
--- /home/bhearsum/Mozilla/patches/multiple-partials2-tools.diff	2012-08-14 10:43:05.548984774 -0400
+++ /home/bhearsum/Mozilla/patches/multiple-partials2-tools-1.diff	2012-08-20 10:05:27.944301076 -0400
@@ -990,7 +990,7 @@
 +
 +1;
 diff --git a/lib/python/build/l10n.py b/lib/python/build/l10n.py
-index 047b8ed..47f9844 100644
+index 047b8ed..431a0b1 100644
 --- a/lib/python/build/l10n.py
 +++ b/lib/python/build/l10n.py
 @@ -55,18 +55,18 @@ def l10nRepackPrep(sourceRepoName, objdir, mozconfigPath,
@@ -1045,6 +1045,12 @@
 +    updateAbsDir = '%s/%s' % (posixDistDir, updateDir)
 +    current_mar = '%s/%s-%s.complete.mar' % (updateAbsDir, productName, version)
 +    unwrap_full_update = '../../../tools/update-packaging/unwrap_full_update.pl'
++    make_incremental_update = '../../tools/update-packaging/make_incremental_update.sh'
++    prevMarDir = '../../../../'
++    if mozillaDir:
++        unwrap_full_update = '../../../../%stools/update-packaging/unwrap_full_update.pl' % mozillaDir
++        make_incremental_update = '../../../%stools/update-packaging/make_incremental_update.sh' % mozillaDir
++        prevMarDir = '../../../../../'
 +    env['MAR'] = mar
 +    env['MBSDIFF'] = mbsdiff
 +
@@ -1063,15 +1069,15 @@
 -        run_cmd(['rm', '-rf', previous, current])
 -        run_cmd(['mkdir', previous, current])
 -        unwrap_full_update = '../../../tools/update-packaging/unwrap_full_update.pl'
+-        make_incremental_update = '../../tools/update-packaging/make_incremental_update.sh'
+-        prevMarDir = '../../../../'
+-        if mozillaDir:
+-            unwrap_full_update = '../../../../%stools/update-packaging/unwrap_full_update.pl' % mozillaDir
+-            make_incremental_update = '../../../%stools/update-packaging/make_incremental_update.sh' % mozillaDir
+-            prevMarDir = '../../../../../'
 +        run_cmd(['rm', '-rf', previous])
 +        run_cmd(['mkdir', previous])
-         make_incremental_update = '../../tools/update-packaging/make_incremental_update.sh'
 +        prevMar = partialUpdates[oldVersion]['mar']
-         prevMarDir = '../../../../'
-         if mozillaDir:
-             unwrap_full_update = '../../../../%stools/update-packaging/unwrap_full_update.pl' % mozillaDir
-             make_incremental_update = '../../../%stools/update-packaging/make_incremental_update.sh' % mozillaDir
-             prevMarDir = '../../../../../'
          run_cmd(['perl', unwrap_full_update, '%s/%s' % (prevMarDir, prevMar)],
                  cwd=path.join(nativeDistDir, 'previous'), env=env)
 -        run_cmd(['perl', unwrap_full_update, current_mar],


Carrying forward r+.

Comment 21

[[e-mail] (:bhearsum)]

Attachment: address last review comments; call new update verify bumper

Comment 22

[[e-mail] (:bhearsum)]

Attachment: add previousReleasesStagingServer to staging configs

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 653828 [details] [diff] [review]
address last review comments; call new update verify bumper

Review of attachment 653828 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +1078,5 @@
>          )
>  
>          builders.append({
>              'name': builderPrefix('updates'),
> +            'slavenames': branchConfig['platforms']['linux']['slaves'],

Can you file a followup bug to test and re-enable linux64 slaves?

Comment 24

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #23)
> Comment on attachment 653828 [details] [diff] [review]
> address last review comments; call new update verify bumper
> 
> Review of attachment 653828 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/release.py
> @@ +1078,5 @@
> >          )
> >  
> >          builders.append({
> >              'name': builderPrefix('updates'),
> > +            'slavenames': branchConfig['platforms']['linux']['slaves'],
> 
> Can you file a followup bug to test and re-enable linux64 slaves?

Filed this in bug 784470.

Comment 25

[Chris Cooper [:coop] (he/him)]

In production as of this afternoon.

Comment 26

[[e-mail] (:bhearsum)]

I broke mobile repacks. They're failing in Beta 6 with:
Traceback (most recent call last):
  File "scripts/scripts/mobile_l10n.py", line 501, in <module>
    single_locale.run()
  File "/builds/slave/rel-m-beta-andrd-rpk-1/scripts/mozharness/base/script.py", line 620, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "/builds/slave/rel-m-beta-andrd-rpk-1/scripts/mozharness/base/script.py", line 591, in _possibly_run_method
    return getattr(self, method_name)()
  File "scripts/scripts/mobile_l10n.py", line 331, in setup
    env = self.query_repack_env()
  File "scripts/scripts/mobile_l10n.py", line 133, in query_repack_env
    rc = self.query_release_config()
  File "/builds/slave/rel-m-beta-andrd-rpk-1/scripts/mozharness/mozilla/release.py", line 41, in query_release_config
    self.release_config['old_version'] = rc['oldVersion']
KeyError: 'oldVersion'

Comment 27

[[e-mail] (:bhearsum)]

bug 784577 tracked an issue that broke the repacks when a new locale was added. The fix landed in https://hg.mozilla.org/build/tools/rev/f17982eb8cef.

Comment 28

[[e-mail] (:bhearsum)]

Attachment: update the patcher configs in build/tools

Thunderbird updates failed because I forgot grab new versions after the betas last week.

Comment 29

[[e-mail] (:bhearsum)]

Worked fine in Firefox 15.0b6/Thunderbird 15.0b5.