574731 - single locale mobile repacks on buildbot-0.8.0

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

we need single locale repacks for mobile builds running on scheduler db

Comment 1

[[e-mail] (:bhearsum)]

Which existing builders does this cover. It seems to be at least:
$platform Fennec Desktop $branch l10n
Maemo $branch l10n
Maemo 5 GTK $branch l10n

Are there others?

Comment 2

[Aki Sasaki (not active)]

win32 mobile desktop
linux mobile desktop
maemo5 gtk

on m-c and m-1.9.2.

win32 mobile desktop is currently broken in 0.8 (bug 591802, should be fixed by bug 614254). maemo5 gtk is in 0.8, but not multilocale (this may not prevent working on single locale repacks, however, but will block turning on maemo5 gtk builds in 0.8 on m-c and m-1.9.2).  so linux mobile desktop is probably the best starting point if you're doing this piecemeal.

Comment 3

[Aki Sasaki (not active)]

Oh, we'll need maemo 4 on m-1.9.2 at some point, but we could potentially also leave that on 0.7 until fennec 4.0 ships and 1.x is officially EOLed.

That's extra work b/c maemo 4 isn't in 0.8 atm, and any maemo5 gtk solution should be easily ported there, so probably out of the scope of this bug.

Comment 4

[[e-mail] (:bhearsum)]

This is heavily related to other recent work I've done on l10n stuff. I'm going to look at it.

Comment 5

[[e-mail] (:bhearsum)]

I'm getting along well here but I've hit a weird issue in the build system. When I call 'make upload' from mobile/locales I end up uploading the en-US build again, despite having AB_CD set in the environment. If I run it manually like so: 'make upload ... AB_CD=es-AR' it uploads the correct files. I'm not really sure what the right way to go about doing this is -- whether we need a thin wrapper in mobile-browser/locales/Makefile.in, whether I should be calling it from a different directory, or what have you. Kyle or Ted, any advice you can give on this matter?

Comment 6

[Axel Hecht [:Pike]]

make upload AB_CD=es-AR
works through bash logic, you probably don't invoke it that way?

Gotta tweak the env of the child process then directly.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> make upload AB_CD=es-AR
> works through bash logic, you probably don't invoke it that way?

Thanks, this ended up fixing me up.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: add nightly repack script

This patch adds a script which is capable of doing mobile repacks. It's been tested on Linux and Windows Desktop repacks, though I believe it's capable of doing other platforms, and probably non-desktop repacks as well. All of the code in lib/python is non-mobile specific, and nightly-mobile-repack.py handles the mobile-specific things, such as cloning the correct source repositories.

I ran through a full set of Windows and Linux desktop nightly en-US + repacks, and they went well. I tested running both of them and they seemed to fine -- no issues related to l10n-merge that I could see. You can see the results for yourself here:
http://staging-master.build.mozilla.org:8018/builders
http://staging-stage.build.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-linux-l10n/
http://staging-stage.build.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-win32-l10n/

There's accompanying buildbotcustom and buildbot-configs patches to be posted still.

Comment 9

[[e-mail] (:bhearsum)]

Oops, please ignore the hgtags and update verify hunks, just some funny stuff in my local repo there.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: buildbot-configs tweaks

This turns on mobile desktop nightly en-US+l10n in 0.8-land. It also adds mozconfigs for the l10n repacks and turns on "hg share" usage for the Linux repacks.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: buildbotcustom tweaks

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 500401 [details] [diff] [review]
buildbotcustom tweaks

I screwed up the triggered schedulers bit here, I'll repost.

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 500399 [details] [diff] [review]
add nightly repack script

r=me for the non-hgtag/update verify portions.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: disable linux and windows desktop builds on trunk on 0.7

Very hacky.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: updated buildbotcustom patch

This patch fixes the triggeredScheduler issues with the last one. The triggering bits aren't desktop specific, and _should_ work for any of the platforms processed in this function. It's controlled by the l10n_chunks attribute of a platform.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: combined buildbot-configs patch

This patch combines the 0.7 and 0.8 configs patches, since they need to land at the same time.

Comment 17

[[e-mail] (:bhearsum)]

Oh, one thing I should note here is that the 3/6 builder on both platforms is always going to fail, because of ja/ja-JP-mac always being busted (even on the existing 0.7 repacks). All the other locales seem to repack fine, but we should look at disabling the ja/ja-JP-mac ones IMHO.

Aki, if everything looks good to you I think these are ready to land. If you want to, please land them, if not I'll do it when I get back, around January 10.

Comment 18

[(not currently active) Ted Mielczarek]

(In reply to comment #6)
> make upload AB_CD=es-AR
> works through bash logic, you probably don't invoke it that way?

To clarify:

AB_CD=es-AR make upload # passes AB_CD as an environment variable
make upload AB_CD=es-AR # passes AB_CD as a makefile variable
export AB_CD=es-AR; make upload # passes AB_CD as an environment variable

They *should* all wind up as the same thing, but there are some fiddly differences, see http://www.gnu.org/software/make/manual/make.html#Values .

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 500522 [details] [diff] [review]
updated buildbotcustom patch

>+        l10nSchedulerName = "%s-%s-%s-mobile-l10n" % (mobile_repo_name,
>+                                                      name, platform)

I'm kind of concerned about this.
The mobile_repo_name of "mobile-browser" is shared across a great number of release and project branches; if we enable l10n repacks across any two of those, doesn't that cause problems?

Then again, I'm not entirely sure what "name" is here; maybe that saves us.

Comment 20

[Aki Sasaki (not active)]

(In reply to comment #19)
> Then again, I'm not entirely sure what "name" is here; maybe that saves us.

branch name.
I think this should be fine.

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 500399 [details] [diff] [review]
add nightly repack script

changeset:   1986:4903d4415882

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 500523 [details] [diff] [review]
combined buildbot-configs patch

Landed on default

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 500522 [details] [diff] [review]
updated buildbotcustom patch

Landed on default

Comment 24

[[e-mail] (:bhearsum)]

These got merged to production today. Tomorrow we should see en-US and l10n nightlies off of 0.8 instead of 0.7.

Comment 25

[[e-mail] (:bhearsum)]

Attachment: disable m-c desktop stuff in production, too

I somehow forgot to disable in 0.7 production in my last patch, whoops.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 503532 [details] [diff] [review]
disable m-c desktop stuff in production, too

changeset:   3768:46f451376cc2

Comment 27

[[e-mail] (:bhearsum)]

Other than locales that failed to build because of issues not resolved by l10n-merge (ga-IE, ja/ja-JP-mac/he), the desktop l10n nightlies worked fine.

Comment 28

[Axel Hecht [:Pike]]

All of these should build perfectly fine. The only locale with broken builds is km.

Comment 29

[Axel Hecht [:Pike]]

ah, this is fennec? sorry, yeah, those have brokeness.

I don't see any desktop nightlies on http://l10n.mozilla.org/~axel/nightlies/?date=2011-01-14, related?

Comment 30

[[e-mail] (:bhearsum)]

They were all broken the last time the 0.7 mobile desktop l10n nightlies ran, too, so there must be something not related to these patches busting them.  (Why do even try to build ja on mac, anyways?) Specific errors are:
ga-IE:
Traceback (most recent call last):
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 507, in <module>
    main()
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 504, in main
    localedirs=options.l10n_src)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 295, in makeJars
    jardir=jardir)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 240, in makeJar
    localedirs)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 361, in processJarSection
    outHelper, jf)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-2/build/mozilla-central/config/JarMaker.py", line 393, in _processEntryLine
    raise RuntimeError('File "%s" not found in %s' % (src, ', '.join(src_base)))
RuntimeError: File "wikipedia-ga-IE.xml" not found in e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-2\build\mozilla-central\mobile\locales\en-US\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-2\build\l10n-central\ga-IE\mobile\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-2\build\mozilla-central\obj-l10n\mobile\locales, .
make[1]: *** [searchplugins] Error 1
make: *** [libs-ga-IE] Error 2

he:
Traceback (most recent call last):
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 507, in <module>
    main()
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 504, in main
    localedirs=options.l10n_src)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 295, in makeJars
    jardir=jardir)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 240, in makeJar
    localedirs)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 361, in processJarSection
    outHelper, jf)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 393, in _processEntryLine
    raise RuntimeError('File "%s" not found in %s' % (src, ', '.join(src_base)))
RuntimeError: File "wikipedia-he.xml" not found in e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\mobile\locales\en-US\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\l10n-central\he\mobile\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\obj-l10n\mobile\locales, .
make[1]: *** [searchplugins] Error 1
make: *** [libs-he] Error 2

ja:
Traceback (most recent call last):
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 507, in <module>
    main()
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 504, in main
    localedirs=options.l10n_src)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 295, in makeJars
    jardir=jardir)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 240, in makeJar
    localedirs)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 361, in processJarSection
    outHelper, jf)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 393, in _processEntryLine
    raise RuntimeError('File "%s" not found in %s' % (src, ', '.join(src_base)))
RuntimeError: File "twitter-ja.xml" not found in e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\mobile\locales\en-US\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\l10n-central\ja\mobile\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\obj-l10n\mobile\locales, .
make[1]: *** [searchplugins] Error 1
make: *** [libs-ja] Error 2

ja-JP-mac:
Traceback (most recent call last):
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 507, in <module>
    main()
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 504, in main
    localedirs=options.l10n_src)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 295, in makeJars
    jardir=jardir)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 240, in makeJar
    localedirs)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 361, in processJarSection
    outHelper, jf)
  File "e:/builds/moz2_slave/cen-mb-br-w32-ntly-l10n-3/build/mozilla-central/config/JarMaker.py", line 393, in _processEntryLine
    raise RuntimeError('File "%s" not found in %s' % (src, ', '.join(src_base)))
RuntimeError: File "twitter-ja.xml" not found in e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\mobile\locales\en-US\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\l10n-central\ja-JP-mac\mobile\searchplugins, e:\builds\moz2_slave\cen-mb-br-w32-ntly-l10n-3\build\mozilla-central\obj-l10n\mobile\locales, .
make[1]: *** [searchplugins] Error 1
make: *** [libs-ja-JP-mac] Error 2

Comment 31

[Aki Sasaki (not active)]

Ben's finished win32 and linux; I'm taking maemo and mac.

Comment 32

[Aki Sasaki (not active)]

Attachment: enable mac builds+mac l10n in 0.8

So far my testing's going green (2/6). I'm going to let it finish.

Comment 33

[Aki Sasaki (not active)]

Attachment: disable 0.7 desktop builds/repacks

Other than release builds/repacks.

Looks like you disabled them in mozilla2-staging/ but forgot them in mozilla2/, which is something that trips me up on a regular basis.

Comment 34

[Aki Sasaki (not active)]

Looks like they all repacked successfully except he, which isn't a surprise.

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 513712 [details] [diff] [review]
disable 0.7 desktop builds/repacks

Sorry I forgot to do this.

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 513709 [details] [diff] [review]
enable mac builds+mac l10n in 0.8

http://hg.mozilla.org/build/buildbot-configs/rev/174d91a1ed97
Landed on default.

Comment 37

[Aki Sasaki (not active)]

Comment on attachment 513712 [details] [diff] [review]
disable 0.7 desktop builds/repacks

http://hg.mozilla.org/build/buildbot-configs/rev/dd265e4c1a91
Landed on default.

Comment 38

[Aki Sasaki (not active)]

Maemo single locale repacks and a couple r?s on Maemo multilocale builds (bug 525327 and bug 628571) are all that's blocking us from moving all mobile depend/nightly builds to 0.8.

Maemo single locale deb packaging is already broken (bug 630475).
They're currently generating useful tarballs of repacked Maemo builds, as well as the xpi.

Per our current repack direction, we're considering moving to xpi downloads, and we're under the impression that these are not platform-specific.

In that case, should I skip porting Maemo single locale repacks to 0.8?
The downside: we'll stop producing Maemo single locale repacks at all, just the en-US and multilocale builds.
The upside: we'll still have xpi's produced from mac, windows, and linux, and it'll save me maybe ~1 week of work.

Otherwise, I'll spend that week of work porting Maemo single locale repacks to 0.8, and they'll continue to run, though with broken deb packaging.

I'm open to either solution.

Comment 39

[Aki Sasaki (not active)]

Because:

* we are only going to ship the multi-locale build on Android and Maemo [1],
* we have single locale repacks for linux, windows, and now mac desktop Fennec builds,
* Maemo single locale deb packaging is broken [2],
* and since we have a plan to move forward with xpi langpacks for future mobile localization,

I am going to consider porting Maemo single locale repacks to our 0.8 automation to be outside the scope of this bug.

If we do need this, let's open another bug and prioritize accordingly. Otherwise moving forward with xpis sounds like a great solution to both packaging/automation issues and file download sizes.

[1] http://groups.google.com/group/mozilla.dev.l10n/browse_thread/thread/0f54fcaafb39ddd6#
[2] bug 630475

-> resolved fixed.