480081 - Generate partial & complete mars for nightly l10n builds on m-c

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We're just creating exe/zip/tar.bz2/dmg at the moment, but need complete mars to do another test in bug 459878. This is a mod to the system bsmedberg originally set up.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Need to
* configure with --enable-update-packaging, RepackFactory's extraConfigureArgs can help
* make in nsprpub, config and modules/libmar to get dist/host/mar
* set MOZ_MAKE_COMPLETE_MAR=1 when calling make installers-ab-CD in order to get a mar file
* locate complete mar's in objdir and adjust master.cfg's shipFiles to upload them

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: "buildbotcustom" patch

This is a patch against the fork of buildbotcustom
  http://hg.mozilla.org/users/bsmedberg_mozilla.com/buildbotcustom/
which adds a completeMar option to RepackFactory, then gets conditional-happy using it.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: master.cfg changes

Changes to the buildbot config to support complete mar generation - basically just update the manifest of files to upload and flip the boolean.

These patches work well on Linux and Windows, where I've checked tar.bz2 vs mar file contents, and zip vs exe vs mar. Didn't let the files upload, but it should work. 

On mac, bm-l10n-pmac-01 doesn't have the build environment of our other slaves so configure gives you
  configure: error: Library requirements (libIDL-2.0 >= 0.8.0) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them.
If we want to go ahead with this (dependent on b3 start timing) then we'd need to fix this up by rsyncing over /opt/ from another slave (to get all macports stuff).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I suspect this is INVALID for bsmedberg's machines, and I don't need nightly complete mars for bug 459878 now that 3.1b3 got started. Pushing it back to the pool for triage against on our future plans for l10n.

Comment 5

[Chris Cooper [:coop] (he/him)]

This might be a good one for Armen to pick up. cc-ing Axel as well for his thoughts on l10n updates.

Comment 6

[Axel Hecht [:Pike]]

This surely blocks updates on the nightly channel, aka bug 449828.

We probably want different build paths for dep builds and nightlies for this, so that we don't waste time and bandwidth creating the mars all day, when we only want them for the nightly.

I'd personally block this on a property-dependent factory, haven't filed a bug on that yet.

Comment 7

[Chris Cooper [:coop] (he/him)]

Armen and I discussed him taking this on yesterday, but he may not have time to get to it for a few weeks.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) Summary updated: Wouldnt we also want partial mars or am I missing something? 

2) Before we can have people getting nightly updates like this, we need to know that they are getting updated to *working* builds - ie a build that is not missing any l10n entities. Should that be a blocking bug?

Comment 9

[Axel Hecht [:Pike]]

Yes, and yes.

Comment 10

[Axel Hecht [:Pike]]

That'd be bug 478436 for compare-locales/l10n merge.

Comment 11

[Armen [:armenzg]]

After the whole week in this bug I would like to give an update of where I am and what are the problems that I am encountering. I will do this first briefly and after into details. My apologies for throwing so much at once.

I have accomplished:
- automate with buildbot the generation of mar files (thanks nthomas!)
- create manually a complete.update.snippet (thanks bhearsum - contents of one at the bottom of this comment)

I need to:
- test if the manually created snippet if uploaded to aus2-stage (production server) will be able to generate *partial updates* (complete updates can be tested in the staging AUS server without problems). I cannot do this by myself due to my lack of knowledge nthomas and bhearsum could be helping me with this
- automate the generation of snippets

#######################

PROBLEMS:
The generation of snippets for en-US goes like this:
1) call the CreateCompleteUpdateSnippet step - http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#669
2) this class basically generates a text file "complete.update.snippet" by pulling information from propeties - http://mxr.mozilla.org/build/source/buildbotcustom/steps/updates.py#62
3) these properties are set through SetMozillaBuildProperties - http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#619
4) this class calls a LoggedRemoteCommand - http://mxr.mozilla.org/build/source/buildbotcustom/steps/misc.py#174 - and loads a bunch of properties after it is run - http://mxr.mozilla.org/build/source/buildbotcustom/steps/misc.py#183
5) the LoggedRemoteCommand (which leaves in buildbot instead of buildbotcustom) loads the following properties that will be later used to create the snippet:
  - buildid
  - appVersion
  - packageFilename
  - packageHash
  - packageSize
  - installerFilename
  - installerHash
  - installerSize
  - completeMarFilename
  - completeMarHash
  - completeMarSize
6) as I mentioned once loaded this properties the CreationCompleteUpdateSnippet can generate the snippet

PROPOSED SOLUTION
We already load properties from the build system through "make ident" which gives us the en-US's checkout changeset. For fennec we have extended this target to pull two properties at once (showing that multiple of them could be pulled - I have used this locally to pull the buildid) http://mxr.mozilla.org/mobile-browser/source/locales/Makefile.in#93

I propose that the build system should do more rather than relying on custom buildbot code (I am not proposing change to the en-US system). I would like to extend the make ident target (or have another target) to pull all the mentioned properties that I need to generate the snippets.

My second proposal would be to have the build system to generate the snippet for us but this has less rush than pulling the properties. Which our previous releasing product (tinderbox) knew how to do it as well - http://mxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#751

###########################

The needed properties according to generateSnippet (http://mxr.mozilla.org/build/source/buildbotcustom/steps/updates.py#58) are:
- downloadURL
- completeMarHash
- completeMarSize
- buildid
- appVersion

Contents of a snippet for the curious:
------------------
complete
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/05/2009-05-19-03-mozilla-central/firefox-3.6a1pre.en-US.mac.complete.mar
sha1
d6134eda47910ff0bb67aa21eb1da1709b345dd6
16953371
20090519032945
3.6a1pre
3.6a1pre
--------------------

Comment 12

[Armen [:armenzg]]

Regarding snippet generation for en-US, do we generate snippets for mozilla-central and mozilla-1.9.1? If yes, do we want updates for locales in both of these branches?

Comment 13

[Robert Kaiser]

We surely provide snippets and updates for all differently versioned branches, and I think it would be good to have that for L10n nightlies as well.

Comment 14

[Armen [:armenzg]]

A quick update, I have filed bug 496196 to have a python script that generates the snippets for a locale. If anyone is interested to extend it for using it for en-US nightly snippet generation feel welcome to bring the discussion there.

I will be now working on making nightly repackages to be uploaded to firefox/tinderbox-builds rather that firefox/nightly/latest-blah.

After that I should be posting the patches making modifications to buildbotcustom/process/factopry.py and buildbot-configs/mozilla2-staging/master-main.cfg to enable the l10n nightly updates (which needs the snippet generation and uploading to AUS)

Comment 15

[Armen [:armenzg]]

I have run a test run in staging and the complete MAR file has been uploaded correctly to "latest":
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.1-l10n/firefox-3.5pre.fr.linux-i686.complete.mar
Here is the snippet that was uploaded:
http://staging-stage.build.mozilla.org/aus2/build/0/Firefox/mozilla-1.9.1/Linux_x86-gcc3/20090605031224/fr/complete.txt
which has an incorrect URL:
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/2009/06/2009-06-05-03-mozilla-1.9.1/firefox-3.5pre.fr.linux-i686.complete.mar
since it should point to staging-stage and in this run I did not create any dated folder and after mozilla-1.9.1 there should be a "-l10n".

Here is the log of the repackage (BTW I am testing "fr" and only Linux since I have to do local patching in the slave for the python script):
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaStaging/1244227404.1244227470.28923.gz&fulltext=1

Tested:
* MAR uploaded to latest
* Snippet uploaded correctly

Missing:
* MAR uploaded to dated dir
* Rectify URL

Comment 16

[Armen [:armenzg]]

Here is the dated folder:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/2009-06-05-03-mozilla-1.9.1-l10n/

I would give a log of it but the tbox page is gone. *sigh*

GLOBAL TODO:
- test repack-on-change scenario locally
- test repack-on-change scenario on staging-master to test proper uploading
- work on the python script to be called directly by buildbot rather than makefile
- fix the URL generated in the snippet

Comment 17

[Chris Cooper [:coop] (he/him)]

Just a few points of clarification:

* we're only targeting 3.5+ l10n nightlies for updates. We're explicitly *not* going to provide updates for 3.0 l10n nightlies.

* complete updates are easier to provide than partial updates, so it is likely that we will begin by providing complete updates, and then start providing partial updates sometime later. This will likely be in Q3, and may be dependent on improvements to the nightly update system as a whole (en-US included).

Comment 18

[Armen [:armenzg]]

Adding the dependency since I would like to see if the updates would work if downloaded from staging-stage before reaching production (Bug 497679)

Comment 19

[Armen [:armenzg]]

Attachment: moz2-staging - it enables l10n nightly updates

Comment 20

[Armen [:armenzg]]

Attachment: buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MARs and updates

This is what has been running on staging-master properly but
I am not asking for review until I test that it works for the repackage-on-change scenario. The same goes with the previous patch.

Comment 21

[Armen [:armenzg]]

Attachment: moz2-staging - it enables l10n nightly updates

Comment 22

[Armen [:armenzg]]

Attachment: buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

Comment 23

[Armen [:armenzg]]

The previous two patches have been tested that they detect the repackages on change as well.

Comment 24

[Chris Cooper [:coop] (he/him)]

Comment on attachment 383975 [details] [diff] [review]
moz2-staging - it enables l10n nightly updates

>-builders = Firefox\ mozilla-1.9.1\ linux\ l10n Firefox\ mozilla-1.9.1\ macosx\ l10n Firefox\ mozilla-1.9.1\ win32\ l10n
>+builders = Firefox\ mozilla-1.9.1\ linux\ l10n\ build Firefox\ mozilla-1.9.1\ macosx\ l10n\ build Firefox\ mozilla-1.9.1\ win32\ l10n\ build

Is there a reason behind the builder name change here (and elsewhere)?

Comment 25

[Chris Cooper [:coop] (he/him)]

Comment on attachment 383976 [details] [diff] [review]
buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

>+            # this is a tad ugly because we need to python interpolation
>+            # as well as WithProperties

Change to "we need to do python" <- missing "do."

r+ with nit fixed.

Comment 26

[Robert Kaiser]

Comment on attachment 383976 [details] [diff] [review]
buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

Just as a not, the factory.py change here will bitrot the patch from bug 494577 if it goes in earlier than that one.

Comment 27

[Armen [:armenzg]]

(In reply to comment #24)
> (From update of attachment 383975 [details] [diff] [review])
> >-builders = Firefox\ mozilla-1.9.1\ linux\ l10n Firefox\ mozilla-1.9.1\ macosx\ l10n Firefox\ mozilla-1.9.1\ win32\ l10n
> >+builders = Firefox\ mozilla-1.9.1\ linux\ l10n\ build Firefox\ mozilla-1.9.1\ macosx\ l10n\ build Firefox\ mozilla-1.9.1\ win32\ l10n\ build
> 
> Is there a reason behind the builder name change here (and elsewhere)?
>
Yes, one of the builders will do the nightly l10n repackages and the one ending with "build" will do the repackages on change. The disadvantage is to have twice the mozilla-central and 1.9.1 checked out. This will be fixed when we have jhford's centralized souce checkout.

Comment 28

[Chris Cooper [:coop] (he/him)]

Comment on attachment 383975 [details] [diff] [review]
moz2-staging - it enables l10n nightly updates

>@@ -391,6 +399,33 @@
>                     'category': name,
>                 }
>                 c['builders'].append(mozilla2_l10n_nightly_builder)
>+                mozilla2_l10n_dep_factory = NightlyRepackFactory(
>+                    hgHost=HGHOST,
>+                    tree=branch['l10n_tree'],
>+                    project=branch['product_name'],
>+                    appName=branch['app_name'],
>+                    enUSBinaryURL=branch['enUS_binaryURL'],
>+                    nightly=False,
>+                    stageServer=STAGE_SERVER,
>+                    stageUsername=STAGE_USERNAME,
>+                    stageSshKey=STAGE_SSH_KEY,
>+                    repoPath=branch['repo_path'],
>+                    l10nRepoPath=branch['l10n_repo_path'],
>+                    buildToolsRepoPath=BUILD_TOOLS_REPO_PATH,
>+                    compareLocalesRepoPath=COMPARE_LOCALES_REPO_PATH,
>+                    compareLocalesTag=COMPARE_LOCALES_TAG,
>+                    buildSpace=2,
>+                    clobberURL=BASE_CLOBBER_URL,
>+                    clobberTime=clobberTime,
>+                )

Armen and I were discussing landing this morning, and what we really want here is a config toggle that we can use to enable/disable nightly updates entirely. That will allow us to land the buildbotcustom changes in advance of the make targets, and will provide a convenient switch for later (or for other projects that want to re-use this code). 

Something like, e.g.: BRANCHES['mozilla-central']['l10nNightlyUpdate'] = True

Comment 29

[Chris Cooper [:coop] (he/him)]

Comment on attachment 383976 [details] [diff] [review]
buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

See comment #28 for details.

Comment 30

[Chris Cooper [:coop] (he/him)]

Comment on attachment 383975 [details] [diff] [review]
moz2-staging - it enables l10n nightly updates

>+                    createSnippet=branch['create_snippet'],

We're currently using the branch-wide flag create_snippet to determine whether we run the nightly update code for l10n. I think that's a good eventual goal, i.e. have en-US and l10n nightly updates on par with each other, but having an extra flag (l10nNightlyUpdates or some such) for now will allow us to do the staged landings we need in advance of the make targets.

Comment 31

[Armen [:armenzg]]

Attachment: buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files v3

If the parameter l10NightlyUpdates is False (which is by default) it uses the system that we have been using so far (no dated dirs being used)

If the parameter is True it does exactly what the previous patch did

Comment 32

[Armen [:armenzg]]

Attachment: moz2-staging - it enables l10n nightly updates (using l10nNightlyUpdates parameter) v3

the only difference with previous patch is to make it explicit the usage of l10nNightlyUpdate in the factory

Comment 33

[Chris Cooper [:coop] (he/him)]

Comment on attachment 384417 [details] [diff] [review]
buildbotcustom - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files v3

>+        postUploadCmd =  ['post_upload.py']
>+
>+        if l10nNightlyUpdate:
>+            postUploadCmd += ['-i %(buildid)s',
>+                              '-p %s' % kwargs['project'],
>+                              '-b %s' % uploadDir]
>+
>+            self.env = {}
>+            if self.nightly:
>+                # We want to generate MAR files for the nightly scenario
>+                # and upload to 1) latest and 2) a dated directory
>+                self.env = {'MOZ_MAKE_COMPLETE_MAR': '1'}
>+                self.extraConfigureArgs = ['--enable-update-packaging']
>+                postUploadCmd += ['--release-to-latest',
>+                              '--release-to-dated']
>+            else:
>+                # For the repack-on-change scenario we just want to upload
>+                # to tinderbox dated directories
>+                postUploadCmd += ['--release-to-tinderbox-dated-builds']
>+            self.postUploadCmd = WithProperties(' '.join(postUploadCmd))
>+        else:
>+            self.postUploadCmd = 'post_upload.py ' + \
>+                                 '-p %s ' % kwargs['project'] + \
>+                                 '-b %s ' % uploadDir + \
>+                                 '--release-to-latest'

I'm not a fan of the factoring here. Can we move the "-p" and "-b" options outside the if/else since they are always used?

Also, let's use consistent syntax for setting self.postUploadCmd. There's no reason to start with "postUploadCmd =  ['post_upload.py']" outside the if clause if you are just going to concat a string in the else clause.

Comment 34

[Armen [:armenzg]]

Attachment: buildbotcustom v4 - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

Comment 35

[Armen [:armenzg]]

Attachment: moz2-production v0 - it enables l10n nightly updates (using l10nNightlyUpdates parameter)

Comment 36

[Chris Cooper [:coop] (he/him)]

Comment on attachment 385092 [details] [diff] [review]
moz2-production v0 - it enables l10n nightly updates (using l10nNightlyUpdates parameter)

Should set l10nNightlyUpdate for tracemonkey too to avoid key errors.

Comment 37

[Armen [:armenzg]]

Attachment: moz2-production v1 - it enables l10n nightly updates (using l10nNightlyUpdates parameter)

Comment 38

[Chris Cooper [:coop] (he/him)]

Comment on attachment 384660 [details] [diff] [review]
buildbotcustom v4 - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

http://hg.mozilla.org/build/buildbotcustom/rev/1b4a558f099d

Comment 39

[Chris Cooper [:coop] (he/him)]

We actually managed to get the make target changes landed successfully on m-c today, so I've landed the config changes with l10nNightlyUpdates=True for m-c. We might as well take proper advantage of the downtime tomorrow by turning this on.

http://hg.mozilla.org/build/buildbot-configs/rev/c7d761556e15

Comment 40

[Armen [:armenzg]]

sweet - I will have an eye over the weekend
We have to add a line on stage so we don't pile more than 2 days of dated l10n repackages

Comment 41

[Armen [:armenzg]]

This morning we hit minor problems when trying to deploy the patches and had to back them out.

The problem came from:
1) identToProperties for 1.9.1 won't work since the make ident patch has not landed in that tree. To avoid this until landing it I will leave fx_revision in place. The problem is that I tested the patches in slaves in which I had deployed manually the patches from the dependent bug. My bad.

2) one of the slaves had the mozilla-central folder but with nothing in it making the get en_US source step to fail. This was a coincidence. A bad clobbering from previous builds

3) l10n upload for the installers failed. This should have been caught in my testing of the post_upload.py default behaviour changes.

######

I have retested the scenarios and will soon repost the patches with the slight required changes.

Comment 42

[Armen [:armenzg]]

Attachment: moz2-staging v4 - it enables l10n nightly updates *only* for moz-central

Comment 43

[Armen [:armenzg]]

Attachment: buildbotcustom v6 - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

Comment 44

[Armen [:armenzg]]

Comment on attachment 385477 [details] [diff] [review]
buildbotcustom v6 - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

- I have added a step for removing dist/update
- I have fixed the post_upload.py problem
- I have changed the name of addUpdateSteps() to uploadSnippet() since the MAR file and the snippet are generated when we do "make installers-foo"

Comment 45

[Armen [:armenzg]]

Attachment: moz2-staging v5 - it enables l10n nightly updates and dated dirs *only* for moz-central

Comment 46

[Armen [:armenzg]]

Attachment: buildbotcustom v7 - it changes the factory to enable a nightly mode and adds the steps necessary to generate MAR and snippet files

added optional datedDirs parameter

Comment 47

[Armen [:armenzg]]

This has been running correctly in staging-master over the weekend.

mozilla-central with nightly updates and with dated dirs
mozilla-1.9.1 without nightly updates and without dated dirs (same as the old way)

Comment 48

[Chris Cooper [:coop] (he/him)]

Comment on attachment 385815 [details] [diff] [review]
moz2-staging v5 - it enables l10n nightly updates and dated dirs *only* for moz-central

Looks good, but I'm going to change 'datedDirs' to 'l10nDatedDirs' on landing for consistency.

Comment 49

[Armen [:armenzg]]

Attachment: moz2-production v2 - it enables l10n nightly updates

Comment 50

[Chris Cooper [:coop] (he/him)]

http://hg.mozilla.org/build/buildbot-configs/rev/523c7ce321dc
http://hg.mozilla.org/build/buildbotcustom/rev/51eea21af245

Comment 51

[Armen [:armenzg]]

Yesterday we turned on the l10n nightly updates for mozilla-central. The MARs are being generated but the updates are not working. The complete.txt and patial.txt files are empty because the snippets uploaded have not been generated properly by generatesnippet.py. Product is missing from the URL pointing to the MAR files

The line that is missing the product is http://mxr.mozilla.org/mozilla-central/source/tools/update-packaging/generatesnippet.py#88

The URL points to:
http://ftp.mozilla.org/pub/mozilla.org/nightly/2009/07/2009-07-02-03-mozilla-central-l10n/firefox-3.6a1pre.fy-NL.mac.complete.mar
when it should point to:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/07/2009-07-02-03-mozilla-central-l10n/firefox-3.6a1pre.fy-NL.mac.complete.mar

The empty snippets are in:
/opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_Universal-gcc3/20090702031635
The snippets that have been uploaded by the slaves:
/opt/aus2/build/0/Firefox/mozilla-central/Darwin_Universal-gcc3/20090702031635

## beginning of snippet
complete
http://ftp.mozilla.org/pub/mozilla.org/nightly/2009/07/2009-07-02-03-mozilla-central-l10n/firefox-3.6a1pre.fy-NL.mac.complete.mar
sha1
1df337718442cb9e55b2f1f8a9abdad4d32472ae
17580636
20090702031635
3.6a1pre
3.6a1pre
## end of snippet

Comment 52

[Robert Kaiser]

(In reply to comment #51)
> The line that is missing the product is
> http://mxr.mozilla.org/mozilla-central/source/tools/update-packaging/generatesnippet.py#88

Sounds to me like we should hand the script a --download-base-URL but you aren't doing that. We have DOWNLOAD_BASE_URL in the buildmaster's config.py and we should hand that over to the script.

Comment 53

[Armen [:armenzg]]

Attachment: DOWNLOAD_BASE_URL was not being passed properly to generatesnippet.py

Comment 54

[Chris Cooper [:coop] (he/him)]

Comment on attachment 386732 [details] [diff] [review]
DOWNLOAD_BASE_URL was not being passed properly to generatesnippet.py

http://hg.mozilla.org/build/buildbotcustom/rev/2c8a4ab0c551

Comment 55

[Chris Cooper [:coop] (he/him)]

Had to disable nightly update again:

http://hg.mozilla.org/build/buildbot-configs/rev/b8528f6d3703

The patcher script was getting confused with locales and we didn't really want to *start* debugging is at 7:00pm on Friday.

[6:23pm] nthomas: the problem is that it's trying to build an fr partial, and download an hr complete from 20090702
[6:24pm] nthomas: something extremely weird is going on

Comment 56

[Armen [:armenzg]]

Summary
#######
We are taking a step back and making sure the next time we try to enable on production we will be successful.

On bug 502612 we are going to set up the same system that is used on production to generate the partial updates (which is what was not working properly on last week's attempt). Everything run properly on the one run we did few weeks ago but this time it just did not work as expected.

Meanwhile Axel is getting landed the Makefile for 1.9.1 and I will be trying to get generatesnippet.py landed on 1.9.1.

Comment 57

[Armen [:armenzg]]

I am adding the dependency on bug 489313 since once it lands it will allow us to call the generatesnippet.py on 1.9.1.

Tomorrow's downtime will enable l10n nightly updates on mozilla-central and once bug 489313 lands we will be able to enable it on 1.9.1

Comment 58

[Chris Cooper [:coop] (he/him)]

Attachment: Enable l10n nightly updates for mozilla-central in production

Getting the patch ready for tomorrow's downtime.

Comment 59

[Chris Cooper [:coop] (he/him)]

Comment on attachment 393699 [details] [diff] [review]
Enable l10n nightly updates for mozilla-central in production

http://hg.mozilla.org/build/buildbot-configs/rev/28f1149c6eb8

Comment 60

[Armen [:armenzg]]

The l10n nightly repacks with their complete MAR files are in:
* Mac - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009-08-10-03-mozilla-central-l10n/ 
* Win32 - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009-08-10-05-mozilla-central-l10n/
* Linux - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009-08-11-03-mozilla-central-l10n/

Mac and Win32 are made off the en-US nightly of the 10th since that was what there was when they were triggered after the downtime.

Tomorrow in these folders we should have the partial MAR files.

Comment 61

[Nick Thomas [:nthomas] (UTC+12)]

Do we have a cron job in place on stage to remove older dated l10n dirs ? There's plenty of free space there at the moment so there's time to find a nice solution.

Comment 62

[Armen [:armenzg]]

The mentioned folders in comment 60 have their partial MAR files in them.

Partial updates are working and apply as expected.

Windows and Linux do update.
Mac does not update and the URL does not advertise any update.

For instance, if I have this URL https://aus2.mozilla.org/update/3/Firefox/3.6a2pre/20090810031114/Darwin_Universal-gcc3/af/nightly/Darwin%209.8.0/default/default/update.xml?force=1 I believe it should match to /opt/aus2/incoming/2/Firefox/mozilla-central/Darwin_Universal-gcc3/20090810031114/af in aus2-staging but it shows as if there was nothing even thought the snippets are there (complete.txt and partial.txt)

Comment 63

[Armen [:armenzg]]

I have filed as blocker the bug 510080 for the Mac l10n repackages not updating.

We also have 2 bugs related to turning on the l10 nightly updates but we won't be blocking on them. We will fix them without backing out the l10n nightly updates. Bugs 510042 and 509937.

Comment 64

[Axel Hecht [:Pike]]

Follow up bugs and regresssions are usually added as dependent bugs, i.e., blocked by this one. Does it make sense to add those here?

Comment 65

[Armen [:armenzg]]

(In reply to comment #64)
> Follow up bugs and regresssions are usually added as dependent bugs, i.e.,
> blocked by this one. Does it make sense to add those here?
>
I see what you mean. My thought in here was that if the Mac thing gets fixed and the updates would be *done* regardless if the other two came a day or two after.
Adding them since they are a regression (bug 510042) and a follow up (bug 509937).

Comment 66

[Armen [:armenzg]]

(In reply to comment #61)
> Do we have a cron job in place on stage to remove older dated l10n dirs ?
> There's plenty of free space there at the moment so there's time to find a nice
> solution.
>
I don't want to rely on our memory to do this. I have filed bug 510503

Comment 67

[Armen [:armenzg]]

Status update:
- All platforms are working
- Bug 510503 tracks the cronjobs that have to be fine tuned
- Bug 510042 and bug 509937 will be fixed soon

Comment 68

[Armen [:armenzg]]

We finished all the work on this bug to generate the partial and complete MAR files for mozilla-central.

The only work left is in the follow up bugs and enabling updates for 1.9.2.

Axel do the dependencies make a little more sense?
Can we resolve fixed and continue the work on the other bugs?

Comment 69

[Axel Hecht [:Pike]]

Yep, closing as FIXED.