Closed Bug 414622 Opened 12 years ago Closed 12 years ago

setup linux mobile builds

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

(Keywords: mobile)

Attachments

(10 files, 4 obsolete files)

932 bytes, text/plain
Details
1.55 KB, text/plain
Details
27.71 KB, application/x-gzip
Details
9.94 KB, patch
Details | Diff | Splinter Review
4.34 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
1.08 KB, patch
Details | Diff | Splinter Review
11.92 KB, patch
Details | Diff | Splinter Review
4.83 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
451 bytes, patch
Details | Diff | Splinter Review
1.92 KB, patch
Details | Diff | Splinter Review
This is to track setting up a mobile build slave which could be driven by the 1.9/trunk buildbot master.

As much as possible, we'd like to use a ref platform linux machine as basis for this, as it would help reduce possible buildtool weirdness. Details of refplatform are at: http://wiki.mozilla.org/ReferencePlatforms
Priority: -- → P2
What is the intended target? Are we trying to do a normal Linux build --with-embedding-profile=basic or are we doing a cross-compile to ARM or some other trickier target?
I think the initial stuff we would like to push is xulrunner w/ basic profile for linux arm (maemo chinook) in a deb package. 
(In reply to comment #3)
> I think the initial stuff we would like to push is xulrunner w/ basic profile
> for linux arm (maemo chinook) in a deb package. 

Doug - I thought the idea was to build on CentOS, adding additional tools & compilers if needed. Please clarify, do you *require* Debian?
There is currently not any build machinery to create .debs. That should be accomplished as a separate bug before asking release to actually build them.
i am not sure if we require Debian to build.  The requirement I mentioned above was that we wanted to create a debian installer as the binary we push out.  I am fairly sure that you can do this (create a deb package) on any platform.
bsmedberg, brad created the packaging scripts.
Are they in the tree? What is the command used to create the .deb? Probably either "make package" or "make deb" would be best.
on the tinderbox we run this script with:

cat /home/blassey/buildDebMoz.sh | scratchbox -s -p
the build script assumes this will be in the scratchbox user's home directory.
Benjamin, 
These files aren't checked into the tree yet, but I've attached them to the bug
Is there anything special I need to do to install Scratchbox? Is there any specific version I need?
I'll take a look at the build config side of this next week.
Depends on: 417028
Buildbot is up and running at: http://staging-master.build:8020/waterfall
The builds are still going red for now while I solve a problem with the upload step.

Rob/Nick: The mobile builds need to prepend each command with '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail out. I could either subclass to fix this or add a generic 'commandPrefix' variable to it. Which approach do you think is best?
Point: GetBuildID() won't work in it's current state either.
(In reply to comment #16)
> Buildbot is up and running at: http://staging-master.build:8020/waterfall
> The builds are still going red for now while I solve a problem with the upload
> step.
> 
> Rob/Nick: The mobile builds need to prepend each command with
> '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail
> out. I could either subclass to fix this or add a generic 'commandPrefix'
> variable to it. Which approach do you think is best?

Hrm, subclass seems kind of heavyweight, commandPrefix sounds reasonable I guess.. not sure if anything else will ever use it, but that's ok :)

(In reply to comment #17)
> Point: GetBuildID() won't work in it's current state either.

Since you have access to the srcdir and objdir, why not use ${objdir}/config/build_number instead of profile.ini (assuming that the mobile build puts it here too)?
(In reply to comment #18)
> (In reply to comment #16)
> > Buildbot is up and running at: http://staging-master.build:8020/waterfall
> > The builds are still going red for now while I solve a problem with the upload
> > step.
> > 
> > Rob/Nick: The mobile builds need to prepend each command with
> > '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail
> > out. I could either subclass to fix this or add a generic 'commandPrefix'
> > variable to it. Which approach do you think is best?
> 
> Hrm, subclass seems kind of heavyweight, commandPrefix sounds reasonable I
> guess.. not sure if anything else will ever use it, but that's ok :)
> 

It turns out that this won't work anyways. Scratchbox chokes on the '&&'s. Given that this is just the upload step I can actually run it outside of Scratchbox, though. This seems to be working fine. I've got a patch for commandPrefix anyways, but there's no reason to check it in now.


(In reply to comment #19)
> (In reply to comment #17)
> > Point: GetBuildID() won't work in it's current state either.
> 
> Since you have access to the srcdir and objdir, why not use
> ${objdir}/config/build_number instead of profile.ini (assuming that the mobile
> build puts it here too)?
> 

Hm! I didn't know that existed. Over in the moz2 bug (bug 377929) Ted suggested  '$(PYTHON) $(srcdir)/config/printconfigsetting.py $(DIST)/bin/application.ini App BuildID'. I've got a patch that switches to that...but I really don't care which method GetBuildID uses.
From IRC, this is the conclusion we came to:
MozillaStageUpload will use the buildid when hourly granularity is needed (ReleaseToDated) and the build start time when finer granularity is needed (DependToDated).
Attached patch mobile master.cfg + mozconfig (obsolete) — Splinter Review
This is the current version of the Mobile Buildbot config. It's pretty straightforward, albeit a little ugly :(. The SboxGetBuildID class is basically a full re-implementation. I didn't put effort into making it good because I _really_ think the correct solution (and correct way to de-ugly this config) is to get Buildbot running inside of Scratchbox. This config depends on a not-yet-landed-change to MozillaStageUpload (namely, tinderboxBuildsDir), I'll have a patch for that up shortly, after I run a couple more tests.
Attachment #303581 - Flags: review?(rhelmer)
Comment on attachment 303581 [details] [diff] [review]
mobile master.cfg + mozconfig

>diff -r 000000000000 master.cfg
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/master.cfg	Fri Feb 15 14:34:47 2008 -0500
>+c['buildbotURL'] = "http://staging-masterbuild.mozilla.org:8020/"

s/masterbuild/master\.build/ :)

Yeah seeing "scratchbox ..." prepended to everything gets pretty old and the get build ID method is a bit ugly, but it looks like it should work ok, and it's easy to see what's going on by just reading the config.
Attachment #303581 - Flags: review?(rhelmer) → review+
changeset:   2:060ad80d71c7
tag:         tip
user:        bhearsum@mozilla.com
date:        Fri Feb 15 14:50:48 2008 -0500
files:       mobile/linux-arm/mozconfig mobile/master.cfg
Attachment #303581 - Attachment is obsolete: true
This patch makes MozillaStageUpload support:
1) Storing more than one dep build at a time (using the same method/format as bug 291167).
2) Enable an override for the tinderbox-builds directory name. It defaults to 'buildername', but can be overridden, which is useful when you want dep builds and nightly builds in the same tinderbox-builds directory.

There's some uploads from the moz2/mobile builders on stage in ~ffxbld/.
Attachment #304213 - Flags: review?(nrthomas)
Comment on attachment 304213 [details] [diff] [review]
[checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names

So nice and clean in comparison to tinderbox.
Attachment #304213 - Flags: review?(nrthomas) → review+
Comment on attachment 304213 [details] [diff] [review]
[checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names

Checking in steps/transfer.py;
/cvsroot/mozilla/tools/buildbotcustom/steps/transfer.py,v  <--  transfer.py
new revision: 1.2; previous revision: 1.1
done
Attachment #304213 - Attachment description: make releaseToTinderboxBuilds support dependToDated and custom directory names → [checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names
I'd like to confirm a few things:
1) What format do you want the packages in? I'd really prefer to use a tarball but if .deb is important I can do that.
2) Where do you want packages pushed to?
3) How do you want the product built? There was talk of Xulrunner based builds above, but I saw some disagreement about that on IRC. Let me know what the team decides on and we can do that.
4) Here's the mozconfig that is being used: http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig. Can someone have a look over this and OK it? I'm not sure if there's any special things you guys use.

Apologies for this taking longer than expected, it's my #1 priority now, though.
(In reply to comment #28)
> I'd like to confirm a few things:
> 1) What format do you want the packages in? I'd really prefer to use a tarball
> but if .deb is important I can do that.

.deb would be great. What name will you be using?

> 2) Where do you want packages pushed to?

If we could get a directory on ftp.mozilla.org e.g. mobile?

> 3) How do you want the product built? There was talk of Xulrunner based builds
> above, but I saw some disagreement about that on IRC. Let me know what the team
> decides on and we can do that.

Please build xulrunner.

> 4) Here's the mozconfig that is being used:
> http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig.
> Can someone have a look over this and OK it? I'm not sure if there's any
> special things you guys use.

It looks fine except it needs to be changed to use xulrunner and we should not disable jemalloc.

> 
> Apologies for this taking longer than expected, it's my #1 priority now,
> though.
> 

Thanks. 
(In reply to comment #29)
> (In reply to comment #28)
> > I'd like to confirm a few things:
> > 1) What format do you want the packages in? I'd really prefer to use a tarball
> > but if .deb is important I can do that.
> 
> .deb would be great. What name will you be using?
> 

It doesn't matter to me. For now I'll use firefox-3.0b4pre-linux-arm.deb. If you want something else just let me know.

> > 2) Where do you want packages pushed to?
> 
> If we could get a directory on ftp.mozilla.org e.g. mobile?
> 

Looks like there's already a mobile directory on ftp. I believe that I need the 'mobilebld' keys (public+private) to upload to here. Can someone ping me on IRC to give me that?

> > 3) How do you want the product built? There was talk of Xulrunner based builds
> > above, but I saw some disagreement about that on IRC. Let me know what the team
> > decides on and we can do that.
> 
> Please build xulrunner.
> 

OK. To confirm, these are the only flags I need to build with, right?:
mk_add_options MOZ_BUILD_PROJECTS="xulrunner browser"
ac_add_app_options browser --with-libxul-sdk=../xulrunner/dist --enable-libxul
ac_add_app_options browser --disable-installer

> > 4) Here's the mozconfig that is being used:
> > http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig.
> > Can someone have a look over this and OK it? I'm not sure if there's any
> > special things you guys use.
> 
> It looks fine except it needs to be changed to use xulrunner and we should not
> disable jemalloc.
> 

I had build issues with jemalloc on, if someone could help me debug them that would be really helpful.

> > 
> > Apologies for this taking longer than expected, it's my #1 priority now,
> > though.
> > 
> 
> Thanks. 
> 

Attachment #300577 - Attachment mime type: application/octet-stream → text/plain
I think I'm confused here. I was under the impression that you want Firefox on Xulrunner builds done. However, the build scripts posted build browser. What am I supposed to be building here?
OK. I just chatted with Doug on the phone. We are building Xulrunner for Linux-arm right now. That's all.

I don't know anything about debian build/packaging scripts; I'm going to need someone to modify them to work with Xulrunner builds. Right now, it seems to be stuck (or just taking a long time) on the 'dpkg-source' target.
From IRC:
For now, we are simply going to be building xulrunner on Linux-arm. Nothing else, no packaging.

We will add support for 'make deb' in the build system. Once that is in place we will use that to create a package and upload to stage.

The former is running and will show up on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mobile shortly.
Attached patch updated mozconfig, master.cfg (obsolete) — Splinter Review
This config supports both nightly and dep builds. It explicitly does not package or upload builds as per stated in previous comments. I think it's pretty straightforward.

Doug, I realize you're probably unfamiliar with Buildbot config files I'd just like you to sign off on the mozconfig, though.
Attachment #304760 - Flags: review?(rhelmer)
Attachment #304760 - Flags: review?(dougt)
Comment on attachment 304760 [details] [diff] [review]
updated mozconfig, master.cfg

only reviewing the mozconfig.  the other stuff, i don't have experience with.

enable-extensions=softkb

do you want to do what bsmedberg said, and use a relative path to softkb so you don't have to move it and patch it?

other than that, it looks fine for now.
(In reply to comment #35)
> (From update of attachment 304760 [details] [diff] [review])
> only reviewing the mozconfig.  the other stuff, i don't have experience with.
> 
> enable-extensions=softkb
> 
> do you want to do what bsmedberg said, and use a relative path to softkb so you
> don't have to move it and patch it?
> 

Honestly, now that I have it working this way I'm not so inclined to change it. The patch would have to be updated too. I think the right solution here is to get whatever changes are necessary out of the patch and into the tree -- and then we can update the mozconfig if necessary.
i agree with you.
/var on mobile-linux-slave1 got mounted read-only this morning. We've seen this before and it is fixed with a kernel upgrade. I had to reboot before installing the new kernel and now (without even having a new kernel) scratchbox has stopped working. I get the following error:
ERROR: Scratchbox is not properly set up!

Here's an excerpt from the login script:
if [ ! -d $SANDBOX/scratchbox/tools/bin ] || [ ! -c $SANDBOX/dev/null ]; then
    exit_error "Scratchbox is not properly set up!"
fi

$SANDBOX/scratchbox/tools/bin translates to /scratchbox/users/cltbld/scratchbox/tools/bin. /scratchbox/users/cltbld/scratchbox is an empty directory. /scratchbox/users/cltbld/dev is also empty.

Not sure what to do here. Trying to debug it before resorting to re-installing scratchbox. I've got a funny feeling this is going to happen on every reboot though...
If anyone on the Mobile team knows how to fix this, please let me know. I'm reading the install scripts to see how these things are set-up in the first place..
as root:

/scratchbox/sbin/sbox_ctl start
This is a very, very, very basic script to start scratchbox on boot. Probably don't need it anywhere else right now, but I wanted it in this bug to make sure it doesn't get lost. To use it, place it in /etc/init.d and then do 'chkconfig --level 235 sbox add'.
Ben, did you get the build to run with jemalloc? 
christian -- it is the default linux configuration, so yes, it has jemalloc.
Comment on attachment 304760 [details] [diff] [review]
updated mozconfig, master.cfg

>diff -r 060ad80d71c7 mobile/linux-arm/mozconfig
>--- a/mobile/linux-arm/mozconfig	Fri Feb 15 14:50:48 2008 -0500
>+++ b/mobile/linux-arm/mozconfig	Thu Feb 21 12:43:41 2008 -0500

>+linux_arm_dep_factory.addStep(ShellCommand(
>+    command = ['/scratchbox/login', '-p', '-d', 'dep/mozilla',
>+               'patch -N -p0 -i ../softkb.diff'],
>+    description=['patching', 'for', 'softkb'],
>+    descriptionDone=['patched', 'for', 'softkb'],
>+    # this will retrun 1 on subsequent patches, we have to ignore it

>+linux_arm_nightly_factory.addStep(ShellCommand(
>+    command = ['/scratchbox/login', '-p', '-d', 'nightly/mozilla',
>+               'patch -N -p0 -i ../softkb.diff'],
>+    description=['patching', 'for', 'softkb'],
>+    descriptionDone=['patched', 'for', 'softkb'],
>+    # this will retrun 1 on subsequent patches, we have to ignore it

Please fix the typo in the comments :)

>+#linux_arm_nightly_factory.addStep(ShellCommand(
>+#    command = ['/scratchbox/login', '-p', '-d', 'mozilla/%s' % OBJDIR,
>+#               'make package'],
>+#    description=['make package'],
>+#    descriptionDone=['make package'],
>+#    haltOnFailure=True
>+#))
>+#linux_arm_nightly_factory.addStep(SboxGetBuildID(
>+#    basedir='mozilla'
>+#))
>+#linux_arm_nightly_factory.addStep(MozillaStageUpload(
>+#    objdir='/scratchbox/users/cltbld/home/cltbld/mozilla/%s' % OBJDIR,
>+#    username=STAGE_USERNAME,
>+#    milestone='mobile',
>+#    remoteHost=STAGE_SERVER,
>+#    remoteBasePath=STAGE_BASE_PATH,
>+#    platform='linux',
>+#    group=STAGE_GROUP,
>+#    sshKey=STAGE_SSH_KEY,
>+#    releaseToLatest=False,
>+#    releaseToDated=False,
>+#    releaseToTinderboxBuilds=True,
>+#    tinderboxBuildsDir='mobile-linux-arm',
>+#    dependToDated=True
>+#))
>+#

Why is this (and other) steps commented out? Do you plan on re-enabling them as-is? If not, I'd say just remove them.
Attachment #304760 - Flags: review?(rhelmer) → review+
(In reply to comment #43)
> christian -- it is the default linux configuration, so yes, it has jemalloc.
> 

Actually, the mozconfig has --disable-jemalloc at the moment. Sounds like I should turn it back on and have someone investigate the compile failures.
changeset:   5:09de4f042179
tag:         tip
user:        bhearsum@mozilla.com
date:        Tue Feb 26 10:20:31 2008 -0500
files:       mobile/linux-arm/mozconfig mobile/master.cfg
Attachment #304760 - Attachment is obsolete: true
Attachment #304760 - Flags: review?(dougt)
(In reply to comment #45)
> (In reply to comment #43)
> > christian -- it is the default linux configuration, so yes, it has jemalloc.
> > 
> 
> Actually, the mozconfig has --disable-jemalloc at the moment. Sounds like I
> should turn it back on and have someone investigate the compile failures.
> 

dougt has a patch for it.
Can you get that checked in?
Christian and I chatted on IRC about --disable-jemalloc. Until dougt's patch is landed we'll leave --disable-jemalloc in the mozconfig. We can flip the switch on it after that.
Priority: P2 → P3
Whiteboard: waiting for jemalloc patch to be landed
youre not blocked on he jemalloc stuff, right?
The only thing left to do in this bug is flip on jemalloc. Packaging/uploading is being tracked in bug 418852 and is blocked by debian packaging support.
(In reply to comment #51)
> The only thing left to do in this bug is flip on jemalloc. 
Doug/Brad: any update on this?


> Packaging/uploading
> is being tracked in bug 418852 and is blocked by debian packaging support.
Do we *require* deb packaging support in place before we can close this bug#414622 or are the mobile builds we have now enough?
john,
can't we just disable jemalloc in the mozconfig until jemalloc supports the platform we are targeting.  There will be other tweaks we will have to make to the build config as time goes one.  jemalloc can just be one of hem.
dougt, can't you check-in your patch since it is ARM-only?
> > Packaging/uploading
> > is being tracked in bug 418852 and is blocked by debian packaging support.
> Do we *require* deb packaging support in place before we can close this
> bug#414622 or are the mobile builds we have now enough?
> 

As mentioned in comment #51, packaging/uploading is being tracked in a follow-up bug, 418852.

(In reply to comment #53)
> john,
> can't we just disable jemalloc in the mozconfig until jemalloc supports the
> platform we are targeting.  There will be other tweaks we will have to make to
> the build config as time goes one.  jemalloc can just be one of hem.
> 

jemalloc is disabled right now, I left this bug open to track it so we didn't forget about it. I'm OK with closing the bug, though, so if you want to, feel free.
@christian: see bug 418509
@doug: I am aware of that bug but the patch you have right now will fix our immediate problem and allow us to do the build on Maemo. Can't we then keep the other bug open to make it work across other environments?
we can build _now_ w/o jemalloc.  I have expressed multiple times to do that in this bug.

(In reply to comment #58)
> we can build _now_ w/o jemalloc.  I have expressed multiple times to do that in
> this bug.
> 

I think you're implying that we're not building mobile now? Apologies if that's not what you meant, but we _are_ building it. I mentioned this in comment #33, but perhaps not clearly enough. You can see results here: http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mobile
1) jemalloc is not part of this bug.

2) there is a bug in QEMU..  you are going to have to remove /tmp/qemu.log.  This file grow unbounded.
3) could we also push tar.gz of release

4) the clobber builds are red, not sure if they have ever gone green
Status: ASSIGNED → NEW
Whiteboard: waiting for jemalloc patch to be landed
I'll probably have time to take care of that stuff this week. I was just looking over things and realized that I don't have the mobilebld keys -- can you get them to me somehow?
This patch does a few things:
* Change objdir name (s/fx/xr/) to reflect what is actually being built
* Remove unused code/imports (buildbotcustom stuff)
* Add BuildStep to remove qemu log at the start of a build
* Add steps to package and upload tar.bz2 packages to stage.
Attachment #308901 - Flags: review?(rhelmer)
Priority: P3 → P2
Alright, Brad sent me over the mobilebld keys today.
Whiteboard: waiting on review
Attachment #308901 - Flags: review?(rhelmer) → review+
Comment on attachment 308901 [details] [diff] [review]
[checked in] updates to address dougt's comments

changeset:   9:40e41ba19bba
tag:         tip
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Mon Mar 24 08:00:34 2008 -0400
files:       mobile/master.cfg
Attachment #308901 - Attachment description: updates to address dougt's comments → [checked in] updates to address dougt's comments
I've updated the buildbot master with these changes -- it's not spitting out builds right now though (see bug 424754) so we can't fully test it yet.
Status: NEW → ASSIGNED
Depends on: 424754
Priority: P2 → P3
Whiteboard: waiting on review
Attachment #313205 - Flags: review?
Attachment #313206 - Flags: review?(robert)
Attachment #313205 - Attachment is obsolete: true
Attachment #313205 - Flags: review?
Comment on attachment 313206 [details] [diff] [review]
final mobile config update, for realz

"-p" should make mkdir exit nicely if the dir already exists so the "haltOnFailure" doesn't need to change.

r+ with that :)
Attachment #313206 - Flags: review?(robert) → review+
changeset:   16:bcce69b28148
tag:         tip
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Wed Apr 02 15:19:51 2008 -0700
files:       mobile/master.cfg
Attachment #313206 - Attachment is obsolete: true
And we are done here.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Keywords: mobile
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.