Closed Bug 525327 Opened 10 years ago Closed 9 years ago

Get rid of MultiNightlyL10n

Categories

(Release Engineering :: General, defect, P4)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: aki)

References

Details

(Whiteboard: [fennec l10n][l10n][mozharness])

Attachments

(6 files, 7 obsolete files)

30.37 KB, patch
Details | Diff | Splinter Review
11.33 KB, patch
Details | Diff | Splinter Review
3.57 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
30.64 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
743 bytes, patch
jhford
: review+
Details | Diff | Splinter Review
2.39 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
Two builds will be triggered at the same time.
One of the builds will the en-US build that triggers all individual locales repackages and the other will be the multi-locale build.

This is a step that will make things more clear and it will bring back the ability to trigger from the waterfall the Maemo nightly builds (only for the single locale).

I am planning on fixing this on Monday but anyone is welcome to work on it since I won't be around tomorrow Friday.

Find my WIP in progress attached.
NOTE: I know that it is not the right place to put the builder and the factory according to what we have been doing but it gives the idea.
Whiteboard: [fennec l10n]
Priority: -- → P1
Blocks: 525301
Priority: P1 → P3
Putting it into the queue since I am not going to touch this for the next 2 weeks.
Assignee: armenzg → nobody
Component: Release Engineering → Release Engineering: Future
Priority: P3 → --
Dealing with this soon.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P2
Assignee: nobody → armenzg
Blocks: maemo4
Would it be difficult to make the multi-locale step a repack instead of a builder?
Attachment #409191 - Attachment is obsolete: true
Attachment #423815 - Flags: review?(aki)
Attachment #409192 - Attachment is obsolete: true
Attachment #423816 - Flags: review?(aki)
(In reply to comment #4)
> Would it be difficult to make the multi-locale step a repack instead of a
> builder?
That is not a small chunk of work. We would have to download the en-US nightly, unpack, add all the other locales and then repack it. I guess this would make us not generate different xulrunner's (en-US and multi-locale would upload their own), the creation of tests and others.

For now I believe it is worth making the separation but we will wait for the final release of Fennec to be out.

For the en-US we uploaded (on staging we upload to "mobile" instead of "firefox":
http://staging-stage.build.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mobile-1.9.2/en-US/

* xulrunner-1.9.2.2pre.en-US.linux-gnueabi-arm.tar.bz2
* xulrunner-1.9.2.2pre.en-US.linux-gnueabi-arm.tests.tar.bz2
* xulrunner_1.9.2.2pre-20100127084817_armel.deb
* fennec-1.0pre.en-US.linux-gnueabi-arm.tar.bz2
* fennec_1.0~20100127091037_armel.deb
* deb_name.txt

For the multi-locale we uploaded:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/?C=M;O=D

* xulrunner-1.9.2.2pre.multi.linux-gnueabi-arm.tests.tar.bz2 
* xulrunner_1.9.2.2pre-20100127092653_armel.deb  
* xulrunner-1.9.2.2pre.multi.linux-gnueabi-arm.tar.bz2
* fennec_1.0~20100127095255_armel.deb
* fennec-1.0pre.multi.linux-gnueabi-arm.tar.bz2
* deb_name.txt
Attachment #423815 - Flags: review?(aki)
Attachment #423816 - Flags: review?(aki)
Cancelling since it makes sense what aki
After chatting on IRC with aki this makes sense to give it a try.
Blocks: 542579
I don't think I see anything wrong with these, other than wanting a single build that's repacked.
Depends on: 543489
Depends on: 506989
Aki this addresses your request of making the multi-locale as triggered step.
I still have to work a little more on it but it has worked for me.

Notice that I have not yet removed the functions of the multi-locale job from MaemoBuildFactory. I assume you want this, correct?

Aki could you please have a quick look to the patch? I would like to work on this on Monday and give you a patch then.
Aki I am putting this bug on the side for a while since there are other bugs that are needing more energy for this end of the quarter. You can give me your feedback at your leisure and I will deal with it when I pick this up in few weeks.

I have left the patch in a working state. I can trigger manually a multi-locale that behaves like repack. It finishes from beginning to end.

I have not tested that it is triggered by the Maemo nightly (I will paste the configs changes) but it should from what I remember in my initial stages.

I tried to used "objdir" but it seems that "make -f client.mk configure" simply doesn't work inside of scratchbox. Therefore, I didn't push it that way.

I also have an idea on how to allow triggering a Multi-locale through the waterfall (I got it to work but it is not part of this patch).
Attachment #430626 - Attachment is obsolete: true
Attachment #431708 - Flags: feedback?(aki)
Assignee: armenzg → nobody
Status: ASSIGNED → NEW
Priority: P2 → P4
Assignee: nobody → armenzg
These were the files uploaded:
* deb_name.txt
* fennec-1.1a2pre.multi.linux-gnueabi-arm.tar.bz2   
* fennec-1.1a2pre.multi.linux-gnueabi-arm.tests.tar.bz2  
* fennec-1.1a2pre.multi.linux-gnueabi-arm.txt 
* fennec_1.1~a2~_armel.deb

http://staging-stage.build.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mobile-1.9.2-multi/1268257162/
Thanks Armen, that's awesome!
And yes, this isn't as high priority as a number of other items on our list.

One thing -- we're missing the buildid in the deb name =P
I'll try to track that down when I read the patch.
Blocks: 526154
Comment on attachment 431707 [details] [diff] [review]
[WIP][buildbotcustom] multi-locale as a triggered job

I think I'd want to see a logfile from this in conjunction with the patch; I'm not sure I'm grokking everything you're doing =)

>+    def addPackageSteps(self):
>+        extraArgs='AB_CD=multi'

I think you might need to specify DEB_PKG_NAME here.  You'll get this from the sendchange and/or the deb_name.txt uploaded by the en-US builder... That, and the absence of the xulrunner deb, is probably why the buildid is missing from the deb name.

The make wget-DEB_PKG_NAME target (with EN_US_BINARY_URL set) would give this to you from deb_name.txt, and then you can specify DEB_PKG_NAME=fennec....deb in the make steps afterwards.

>+        self.addStep(ShellCommand,
>+            name='make_pkg_tests',

We're probably ok just running tests on the en-US, but if this works it probably doesn't hurt... not that much disk or time spent I don't think.

When this is ready to land we need to make sure the appropriate changes are made to the Maemo release builders.  If you want that, I can help; if you're tired of this I can pick that up.

This is sweet; great progress, not too far from a final patch!
No longer blocks: 551310
Comment on attachment 431708 [details] [diff] [review]
[WIP] multi-locale as a triggered job

(most of my feedback was re: the custom patch)
Attachment #431708 - Flags: feedback?(aki) → feedback+
There is not even a chance I can have time this quarter to deal with this.
I believe aki has taken either the whole thing or some pieces of it in other patches. My memory could be failling me.
Assignee: armenzg → nobody
Yeah, I'll fall on this grenade.
Assignee: nobody → aki
Thanks aki. Sorry I couldn't get to it.
Duplicate of this bug: 525301
Status: NEW → ASSIGNED
I have separated this out into its own script, which bypasses all the MultiL10n scheduler bits in buildbot.

I repacked the multi-locale deb in two ways:

1) make package/make deb without actually building fennec, and add the locales. This creates a multi-locale deb that has no bits in it, but all the locales and the multi-locale mobile-l10n.js.   Then I extract the en-US deb into a tmpdir, extract the "hollow" multi-locale deb on top, and create a new deb.

2) repack the en-US deb similar to the single-locale repacks.  Manually copy the multi-locale mobile-l10n.js into defaults/pref/.

I tried running both of those and neither one launches.  The en-US and multi-locale deb (built, not repacked) both launch.

I'm wondering if I missed something in the above steps.  Right now the fastest way to splitting the builders and getting rid of the MultiL10n scheduler appears to be adding a multi-locale builder that runs parallel to the en-US builder, and running the locale portion of it (at least) via a slave-side build script.
Ok, I tried updating DEBIAN/md5sums to see if that was the culprit for the non-launch.  No go.

Back to the multiple builder idea.  Not ideal, as single-locale deb binaries can differ from multi-locale debs (not just string diffs), but if that's what works and gets us away from MultiL10n, then that's what we do for now.
I think we have three options here:

1) somehow get the multi-locale repack to work.  As in, launch and be the same browser, but with more strings.
2) have two separate compile-and-package builders: one single-locale en-US, and one multi-locale.
3) continue with the way Armen wrote them (en-US, then upload, then continue with multi-locale in the same factory), but put that logic slave-side so we get rid of all the negatives associated with MultiNightlyL10n (namely, force build breakage + buildbot-0.8.0 blockage).

Discussed this with joduinn. We ruled out #2 off the bat, since this doubles QA's workload and causes issues with updates and build ids.

Option #1 is nice to have, and we might want to file a followup bug to do this. However, the work associated with that is open-ended and most likely lengthy, and we want to move to buildbot-0.8.0 asap.

I already have a script that groks multi-locales slave-side, so I'll use that to add the multi-locale steps to the existing MaemoBuildFactory, allowing us to ditch MultiNightlyL10n and move to buildbot-0.8.0.

Morphing the bug.
Summary: Separate the multi-locale build into two builders (one for the single locale and one for the multi-locale) → Get rid of MultiNightlyL10n
At some point I had thought of having a make target that generate (and I might have a POC on original bug) the multi-locale and the build-system always has the latest maemo-locales at hand. I am pretty rusty but something like this:
1) make single-locale
2) make upload (or scp)
3) clobber single-locale only
4) make multi-locale (it loops based on maemo-locales)
5) make upload (or scp)
That's cool.

If we had that target, we'd need a way to specify a different set of locales, since release builds use the l10n-changesets json rather than maemo-locales.

Also, I *think* I got a repack with multiple locales in there (albeit with hacks) and it wouldn't launch. I'm wondering if there's something else in the build that is different.  I also wouldn't be surprised if I were doing something wrong.
maemo-locales should be a subset of all the locales and the build, and those locales that are wanted for the multi-locale build, in particular in beta releases.

Also, it'd had really poor build reporting.

I'd favor to get a step-per-locale process that can report well and be understood.
Ok, I'll try to generate a summary at the end of the script that reports on errors/warnings per-locale.
Depends on: 574473
Status: ASSIGNED → NEW
Whiteboard: [fennec l10n] → [fennec l10n][l10n][mozharness]
Duplicate of this bug: 574732
Depends on: 628571
This is my wip patch for buildbotcustom, 0.7.
I'm testing maemo nightlies atm; will move on to release builds, then port to 0.8 and test there.  Also verify that I'm not breaking depend builds or Android builds.
I should probably do a quick 1.9.2 run as well.
I'm fairly confident in my changes & progress at this point, but testing+cleanup will probably run over the weekend.
Attachment #423815 - Attachment is obsolete: true
Attachment #423816 - Attachment is obsolete: true
Attachment #431707 - Attachment is obsolete: true
Attachment #431708 - Attachment is obsolete: true
Also need to test failure scenario for bug 531869.
Forgot to mention: I'm building functional multilocale debs with the above patches. (Installed on n900, ran fennec en-US; switched locales to de; rebooted; ran fennec de)
To avoid disrupting the mobile release schedule ( https://wiki.mozilla.org/Mobile/Planning/2.0#Beta_5 ): I'm thinking I should

a) get this running on 0.8
b) get maemo l10n repacks on 0.8
c) move all nightlies to 0.8, but leave 0.7 releases alone
d) work towards 0.8 releases, but only switch after:
  d.1) fennec 4 is shipped, or
  d.2) we have so much confidence in it that there is a worldwide confidence shortage
Attached patch 0.8 configsSplinter Review
As mentioned above, switching tacks to move all nightlies to 0.8, then develop mobile releases in parallel with 0.7 releases. This allows us to finish this code without putting the mobile 4.0 betas and RCs at risk.

This patch doesn't turn on Maemo m-c builds; that's dependent on getting Maemo single locale repacks.  Other than that, we have macosx and macosx l10n repacks; I believe bhearsum has patches for that.

This code is dependent on bug 524519 and bug 628571.
Also tears out the currently unused 0.8 MaemoReleaseBuildFactory -- all it really did was alter the upload portions of MaemoBuildFactory, and that needs to change to use make upload.

Tested a Maemo m-c multilocale nightly, Android m-c multilocale nightly.  Testing a Maemo tracemonkey non-multilocale nightly to verify I didn't break anything.

On a different version of these patches I tested macosx mobile + l10n repacks -- we're missing an l10n mozconfig at the very least, so I took that out.

I tested installing the multilocale deb on the 0.7 phase of testing.
Comment on attachment 512114 [details] [diff] [review]
0.8 configs

If you want to hold off on the next two test results (non-multilocale build, verify that breaking the multil10n step in a multilocale build will haltOnFailure for bug 531869) that's cool.  Pretty sure this is good, though.
Attachment #512114 - Flags: review?(bhearsum)
Attachment #512116 - Flags: review?(bhearsum)
Comment on attachment 512116 [details] [diff] [review]
buildbotcustom portion

I'm not a domain expert here, but this seems good to me.
Attachment #512116 - Flags: review?(bhearsum) → review+
Attachment #512114 - Flags: review?(bhearsum) → review+
Verified that bug 531869 (maemo multilocale builds should abort on failed repacks) is fixed with this patch + the mozharness 0.3 patch.  (Kicked off a build, nuked all the files in es-ES mid-multil10n step, went red and went straight to reboot.)
Attachment #517234 - Flags: review?(bhearsum)
Attachment #517234 - Flags: review?(bhearsum) → review+
Attachment #517230 - Flags: review?(jhford) → review+
Comment on attachment 517230 [details] [diff] [review]
enable maemo5-gtk and maemo5-qt m-c builds in 0.8

http://hg.mozilla.org/build/buildbot-configs/rev/acfb969742ab
Attachment #517230 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.