Closed Bug 527989 Opened 10 years ago Closed 10 years ago

Add repack-on-change builders for Maemo L10n and Fennec Desktop L10n

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

(Whiteboard: [fennec l10n][l10n])

Attachments

(3 files, 1 obsolete file)

We currently don't have l10n builders that handle repacks-on-change for Fennec.

On the Firefox scenario we have two builders because we didn't want the repacks-on-change to be uploaded just to tinderbox-builds and not latest. Both builders had slightly different steps (e.g. no updates http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l1658).

For the Fennec scenario I can't see at first glance where things get uploaded for each scenario (to be taken care by who takes this) and what the differences on steps are.

# What to consider to get this done

* add the builder to TinderboxMailNotifier
* the scheduler is already taken care by s = SchedulerL10n("l10n", "l10nbuilds2.ini")
* add the factory for the repack-on-change (in NightlyRepackFactory we make it dependent through nightly=False; not sure about MobileDesktopBuildFactory and MaemoNightlyRepackFactory)
* add the builder

The files to work on are (if I am not mistaken):
* mobile_master.py (instead of misc.py like on Firefox)
* l10nbuilds2.ini

To add the new repack-on-change builders to tinderbox like this:
http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#247
to be fixed in here:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2-staging/mobile_master.py#84

NOTE: The naming on the Firefox is incorrect since it doesn't have the word "nightly" for the real nightly. We can fix this for the Fennec scenario in here (I think) http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2-staging/mobile_master.py#69 

Add the repack-on-change factory after this one:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2-staging/mobile_master.py#433

Specify which builders handle the repackages-on-change:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/l10nbuilds2.ini#23
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/l10nbuilds2.ini#35
Assignee: nobody → ccooper
OS: Mac OS X → All
Priority: -- → P3
Hardware: x86 → All
I think we should follow suite with Firefox in that the on-change builds upload to tinderbox-builds. We should create debs and language packs for maemo (and probably tar balls) as well as desktop builds.

We shouldn't bother creating repositories or fennec.install files for these, though, IMHO.
Taking this bug.
Assignee: ccooper → armenzg
Status: NEW → ASSIGNED
Priority: P3 → P2
Even if I am asking for review I don't want to deploy it before I have it running a whole weekend. I want to be sure that I don't cause any regressions to other builders.

I will bake this overnight.
Attachment #414555 - Flags: review?(ccooper)
The dependent builds for Maemo L10n and Linux Desktop L10n have been uploaded to:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-1.9.2-l10n/

For Maemo L10n:
* fennec-1.0b6pre.x-testing.langpack.xpi
* fennec-1.0b6pre.x-testing.linux-gnueabi-arm.tar.bz2
* x-testing/fennec_1.0b6pre_armel.deb 

For Linux Desktop L10n:
* fennec-1.0b6pre.x-testing.linux-i686.tar.bz2
* fennec-1.0b6pre.x-testing.langpack.xpi

Axel how does this look?

(In reply to comment #1)
> We should create debs and language packs for maemo (and
> probably tar balls) as well as desktop builds.
>
Aren't the fennec debs unusable without an associated xulrunner deb considering that the buildid is different from the nightly's?
Whiteboard: [fennec l10n][l10n]
The build ids are the same as the nightlies, as we're repacking the nightly.

The created blobs look right, though we have mac and win repacks now, so I guess those should be created on change, too.

Looking at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n/, it feels a tad funny to have builds in the dir, and then builds in the subdirs. Not really discoverable.
(In reply to comment #6)
> The build ids are the same as the nightlies, as we're repacking the nightly.
I take your word and I scratch testing it myself.

> 
> The created blobs look right, though we have mac and win repacks now, so I
> guess those should be created on change, too.
> 
I would like to deploy this first (Maemo and Linux Desktop) so I can spend some time on my other bugs.
I agree that we should have it as well.

> Looking at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n/,
> it feels a tad funny to have builds in the dir, and then builds in the subdirs.
> Not really discoverable.
I realized that when I wrote this previous comment. We will have to file a bug to fix it but it should happen after we use "make upload" targets correctly in all places.
Sounds good.
Attachment #414552 - Flags: review?(ccooper)
Attachment #414552 - Flags: review?(ccooper)
Comment on attachment 414555 [details] [diff] [review]
[buildbotcustom] allow to differentiate between nightly and dependent builds for MobileNightlyRepackFactory

Sorry for the canceling of the previous patch I got confused.

MobileNightlyRepackFactory is only used by MaemoNightlyRepackFactory and MobileDesktopNightlyRepackFactory as I initially thought.

http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#5208
http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#5127
Comment on attachment 414552 [details] [diff] [review]
[moz2-staging] enable repack-on-change for Maemo L10n and Linux Desktop L10n

>-               )
>+                )
>+                mobile_l10n_dep_factory = MaemoNightlyRepackFactory(
>+                    nightly = False,
>+                    hgHost=mainConfig['hghost'],
>+                    tree=branch['l10n_tree'],
>+                    project=branch['product_name'],
>+                    appName=branch['app_name'],
>+                    packageGlob='-r %(locale)s ' +
>+                                'fennec-*.%(locale)s.linux-gnueabi-arm.tar.bz2 ' +
>+                                'install/fennec-*.%(locale)s.langpack.xpi',
>+                    enUSBinaryURL=branch['enUS_binaryURL'],
>+                    stageServer=mainConfig['stage_server'],
>+                    stageUsername=mainConfig['stage_username'],
>+                    configSubDir=mainConfig['config_subdir'],
>+                    mozconfig=pf['mozconfig'],
>+                    configRepoPath=mainConfig['config_repo_path'],
>+                    stageSshKey=mainConfig['stage_ssh_key'],
>+                    stageBasePath=mainConfig['stage_base_path'],
>+                    repoPath=branch['repo_path'],
>+                    l10nRepoPath=branch['l10n_repo_path'],
>+                    mobileRepoPath=branch['mobile_repo_path'],
>+                    buildToolsRepoPath=mainConfig['build_tools_repo_path'],
>+                    compareLocalesRepoPath=mainConfig['compare_locales_repo_path'],
>+                    compareLocalesTag=mainConfig['compare_locales_tag'],
>+                    buildSpace=2,
>+                    baseWorkDir=pf['base_l10n_workdir'],
>+                    baseUploadDir='%s-l10n' % name,
>+                    clobberURL=mainConfig['base_clobber_url'],
>+                    clobberTime=clobberTime,
>+                )

I think your indentation might be off for this stanza, but otherwise it the patch looks fine.
Attachment #414552 - Flags: review?(ccooper) → review+
Attachment #414555 - Flags: review?(ccooper) → review+
(In reply to comment #7)
> > The created blobs look right, though we have mac and win repacks now, so I
> > guess those should be created on change, too.
> > 
> I would like to deploy this first (Maemo and Linux Desktop) so I can spend some
> time on my other bugs.
> I agree that we should have it as well.
> 
Funny. I looked at my patch and we might be getting this off the bat (check this out I am using an idiom - take that ESL!). All that will be required is to modify l10nbuilds2.ini to react upon changes since the builders (Win32 and OSX Desktop L10n) exist already.

(In reply to comment #10)
> (From update of attachment 414552 [details] [diff] [review])
> I think your indentation might be off for this stanza, but otherwise it the
> patch looks fine.
>
I am actually fixing it.
carrying over the review. This just adds the configuration changes for production.

I did another run on staging and it is all ready to roll-out.
Attachment #414552 - Attachment is obsolete: true
Comment on attachment 414555 [details] [diff] [review]
[buildbotcustom] allow to differentiate between nightly and dependent builds for MobileNightlyRepackFactory

http://hg.mozilla.org/build/buildbotcustom/rev/029be189e08c
Attachment #414555 - Flags: checked-in+
Comment on attachment 416167 [details] [diff] [review]
[buildbot-configs] enable repack-on-change for Maemo L10n and Linux Desktop L10n

http://hg.mozilla.org/build/buildbot-configs/rev/d5384542c04b
and fix missing line:
http://hg.mozilla.org/build/buildbot-configs/rev/b5df6a0783b8
Attachment #416167 - Flags: review+
Attachment #416167 - Flags: checked-in+
We can also see x-testing uploaded to tinderbox-builds for both builders:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-trunk-l10n

Let me also trigger a nightly for both builders to see if everything is as expected.
It worked.

Resolving FIXED.

I will follow repack-on-change for Win32 and OSX in another bug.
We can now fix bug 511616 as well.
Blocks: 511616
For repackages-on-change for Fennec on Win32 and OSX Desktop builds I have filed bug 533780.

(In reply to comment #16)
> Resolving FIXED.
I will really resolve FIXED when I actually see a locale change to trigger the builders.
It is not picking up the changes. I can't be looking actively into this since I am acting as build on duty this week.
2009/12/25 19:05 PST [HTTPPageGetter,client] scheduler: (DEBUG) addChange: Change 34267, File: mobile/chrome/browser.properties
        At: Fri 25 Dec 2009 19:04:57
        Changed By: channy@mozilla.or.kr
        Comments: http://hg.mozilla.org/releases/l10n-mozilla-1.9.2/ko/rev/730f3e771dcdf422c14d5f99ef337d8d4b8e60cb

        Properties: None
2009/12/25 19:05 PST [HTTPPageGetter,client] dispatcher.fennec10x_192: (DEBUG) changed locales/all-locales for fennec10x_192?
2009/12/25 19:05 PST [HTTPPageGetter,client] dispatcher.l10n.fennec10x_192: (DEBUG) adding change 34267
2009/12/25 19:05 PST [HTTPPageGetter,client] dispatcher.l10n.fennec10x_192: (DEBUG) locales list for fennec10x_192 not set up, ignoring change
It seems that the mobile dispatchers would never load the all-locales files since they were missing "/default" in the url.
This patch fixes it but I was lucky to decide to check Axel's buildbotcustom repository.
I will do more testing tomorrow and bake the patch over the weekend on staging and hopefully pick up a change.

Besides that, Axel I diffed the l10n.py in our repos and the one in yours and there is quite some differences. Any ideas of why the differences?
Comment on attachment 421719 [details] [diff] [review]
[buildbotcustom] fixes the problem that we try to load all-locales URL without 

Tested that it works and it doesn't break the Firefox repack-on-change system. No need for baking over the weekend.
Attachment #421719 - Flags: review?(l10n)
Attachment #421719 - Flags: review?(ccooper)
Comment on attachment 421719 [details] [diff] [review]
[buildbotcustom] fixes the problem that we try to load all-locales URL without 

I'm pretty sure this isn't right.

I'm not running the buildbotcustom of my user repo, but a patched version of the releng one.

That patch is actually in bug 525674, not sure if the patch needs merging, but that's the right fix.
Attachment #421719 - Flags: review?(l10n) → review-
Attachment #421719 - Flags: review?(ccooper)
The patch in bug 525674 has fixed this bug.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.