414622 - setup linux mobile builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This is to track setting up a mobile build slave which could be driven by the 1.9/trunk buildbot master.

As much as possible, we'd like to use a ref platform linux machine as basis for this, as it would help reduce possible buildtool weirdness. Details of refplatform are at: http://wiki.mozilla.org/ReferencePlatforms

Comment 2

[Benjamin Smedberg]

What is the intended target? Are we trying to do a normal Linux build --with-embedding-profile=basic or are we doing a cross-compile to ARM or some other trickier target?

Comment 3

[Doug Turner (:dougt)]

I think the initial stuff we would like to push is xulrunner w/ basic profile for linux arm (maemo chinook) in a deb package. 

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> I think the initial stuff we would like to push is xulrunner w/ basic profile
> for linux arm (maemo chinook) in a deb package. 

Doug - I thought the idea was to build on CentOS, adding additional tools & compilers if needed. Please clarify, do you *require* Debian?

Comment 5

[Benjamin Smedberg]

There is currently not any build machinery to create .debs. That should be accomplished as a separate bug before asking release to actually build them.

Comment 6

[Doug Turner (:dougt)]

i am not sure if we require Debian to build.  The requirement I mentioned above was that we wanted to create a debian installer as the binary we push out.  I am fairly sure that you can do this (create a deb package) on any platform.

Comment 7

[Doug Turner (:dougt)]

bsmedberg, brad created the packaging scripts.

Comment 8

[Benjamin Smedberg]

Are they in the tree? What is the command used to create the .deb? Probably either "make package" or "make deb" would be best.

Comment 9

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: script to build for maemo

on the tinderbox we run this script with:

cat /home/blassey/buildDebMoz.sh | scratchbox -s -p

Comment 10

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: this is the script to create .deb's and push to the ftp server

Comment 11

[Brad Lassey [:blassey] (use needinfo?)]

Attachment: these are the files required to set up a debian package

the build script assumes this will be in the scratchbox user's home directory.

Comment 12

[Brad Lassey [:blassey] (use needinfo?)]

Benjamin, 
These files aren't checked into the tree yet, but I've attached them to the bug

Comment 13

[[e-mail] (:bhearsum)]

Is there anything special I need to do to install Scratchbox? Is there any specific version I need?

Comment 14

[(not currently active) Ted Mielczarek]

I'll take a look at the build config side of this next week.

Comment 15

[[e-mail] (:bhearsum)]

http://wiki.mozilla.org/Mobile/Build/Maemo_Build_Instructions

Comment 16

[[e-mail] (:bhearsum)]

Buildbot is up and running at: http://staging-master.build:8020/waterfall
The builds are still going red for now while I solve a problem with the upload step.

Rob/Nick: The mobile builds need to prepend each command with '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail out. I could either subclass to fix this or add a generic 'commandPrefix' variable to it. Which approach do you think is best?

Comment 17

[[e-mail] (:bhearsum)]

Point: GetBuildID() won't work in it's current state either.

Comment 18

[Robert Helmer [:rhelmer]]

(In reply to comment #16)
> Buildbot is up and running at: http://staging-master.build:8020/waterfall
> The builds are still going red for now while I solve a problem with the upload
> step.
> 
> Rob/Nick: The mobile builds need to prepend each command with
> '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail
> out. I could either subclass to fix this or add a generic 'commandPrefix'
> variable to it. Which approach do you think is best?

Hrm, subclass seems kind of heavyweight, commandPrefix sounds reasonable I guess.. not sure if anything else will ever use it, but that's ok :)

Comment 19

[Robert Helmer [:rhelmer]]

(In reply to comment #17)
> Point: GetBuildID() won't work in it's current state either.

Since you have access to the srcdir and objdir, why not use ${objdir}/config/build_number instead of profile.ini (assuming that the mobile build puts it here too)?

Comment 20

[[e-mail] (:bhearsum)]

(In reply to comment #18)
> (In reply to comment #16)
> > Buildbot is up and running at: http://staging-master.build:8020/waterfall
> > The builds are still going red for now while I solve a problem with the upload
> > step.
> > 
> > Rob/Nick: The mobile builds need to prepend each command with
> > '/scratchbox/login -p -d mozilla', this is causing MozillaStageUpload to fail
> > out. I could either subclass to fix this or add a generic 'commandPrefix'
> > variable to it. Which approach do you think is best?
> 
> Hrm, subclass seems kind of heavyweight, commandPrefix sounds reasonable I
> guess.. not sure if anything else will ever use it, but that's ok :)
> 

It turns out that this won't work anyways. Scratchbox chokes on the '&&'s. Given that this is just the upload step I can actually run it outside of Scratchbox, though. This seems to be working fine. I've got a patch for commandPrefix anyways, but there's no reason to check it in now.


(In reply to comment #19)
> (In reply to comment #17)
> > Point: GetBuildID() won't work in it's current state either.
> 
> Since you have access to the srcdir and objdir, why not use
> ${objdir}/config/build_number instead of profile.ini (assuming that the mobile
> build puts it here too)?
> 

Hm! I didn't know that existed. Over in the moz2 bug (bug 377929) Ted suggested  '$(PYTHON) $(srcdir)/config/printconfigsetting.py $(DIST)/bin/application.ini App BuildID'. I've got a patch that switches to that...but I really don't care which method GetBuildID uses.

Comment 21

[[e-mail] (:bhearsum)]

From IRC, this is the conclusion we came to:
MozillaStageUpload will use the buildid when hourly granularity is needed (ReleaseToDated) and the build start time when finer granularity is needed (DependToDated).

Comment 22

[[e-mail] (:bhearsum)]

Attachment: mobile master.cfg + mozconfig

This is the current version of the Mobile Buildbot config. It's pretty straightforward, albeit a little ugly :(. The SboxGetBuildID class is basically a full re-implementation. I didn't put effort into making it good because I _really_ think the correct solution (and correct way to de-ugly this config) is to get Buildbot running inside of Scratchbox. This config depends on a not-yet-landed-change to MozillaStageUpload (namely, tinderboxBuildsDir), I'll have a patch for that up shortly, after I run a couple more tests.

Comment 23

[Robert Helmer [:rhelmer]]

Comment on attachment 303581 [details] [diff] [review]
mobile master.cfg + mozconfig

>diff -r 000000000000 master.cfg
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/master.cfg	Fri Feb 15 14:34:47 2008 -0500
>+c['buildbotURL'] = "http://staging-masterbuild.mozilla.org:8020/"

s/masterbuild/master\.build/ :)

Yeah seeing "scratchbox ..." prepended to everything gets pretty old and the get build ID method is a bit ugly, but it looks like it should work ok, and it's easy to see what's going on by just reading the config.

Comment 24

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch as landed (had to fix an URL in the master.cfg)

changeset:   2:060ad80d71c7
tag:         tip
user:        [e-mail]
date:        Fri Feb 15 14:50:48 2008 -0500
files:       mobile/linux-arm/mozconfig mobile/master.cfg

Comment 25

[[e-mail] (:bhearsum)]

Attachment: [checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names

This patch makes MozillaStageUpload support:
1) Storing more than one dep build at a time (using the same method/format as bug 291167).
2) Enable an override for the tinderbox-builds directory name. It defaults to 'buildername', but can be overridden, which is useful when you want dep builds and nightly builds in the same tinderbox-builds directory.

There's some uploads from the moz2/mobile builders on stage in ~ffxbld/.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 304213 [details] [diff] [review]
[checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names

So nice and clean in comparison to tinderbox.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 304213 [details] [diff] [review]
[checked in] make releaseToTinderboxBuilds support dependToDated and custom directory names

Checking in steps/transfer.py;
/cvsroot/mozilla/tools/buildbotcustom/steps/transfer.py,v  <--  transfer.py
new revision: 1.2; previous revision: 1.1
done

Comment 28

[[e-mail] (:bhearsum)]

I'd like to confirm a few things:
1) What format do you want the packages in? I'd really prefer to use a tarball but if .deb is important I can do that.
2) Where do you want packages pushed to?
3) How do you want the product built? There was talk of Xulrunner based builds above, but I saw some disagreement about that on IRC. Let me know what the team decides on and we can do that.
4) Here's the mozconfig that is being used: http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig. Can someone have a look over this and OK it? I'm not sure if there's any special things you guys use.

Apologies for this taking longer than expected, it's my #1 priority now, though.

Comment 29

[Christian Sejersen]

(In reply to comment #28)
> I'd like to confirm a few things:
> 1) What format do you want the packages in? I'd really prefer to use a tarball
> but if .deb is important I can do that.

.deb would be great. What name will you be using?

> 2) Where do you want packages pushed to?

If we could get a directory on ftp.mozilla.org e.g. mobile?

> 3) How do you want the product built? There was talk of Xulrunner based builds
> above, but I saw some disagreement about that on IRC. Let me know what the team
> decides on and we can do that.

Please build xulrunner.

> 4) Here's the mozconfig that is being used:
> http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig.
> Can someone have a look over this and OK it? I'm not sure if there's any
> special things you guys use.

It looks fine except it needs to be changed to use xulrunner and we should not disable jemalloc.

> 
> Apologies for this taking longer than expected, it's my #1 priority now,
> though.
> 

Thanks. 

Comment 30

[[e-mail] (:bhearsum)]

(In reply to comment #29)
> (In reply to comment #28)
> > I'd like to confirm a few things:
> > 1) What format do you want the packages in? I'd really prefer to use a tarball
> > but if .deb is important I can do that.
> 
> .deb would be great. What name will you be using?
> 

It doesn't matter to me. For now I'll use firefox-3.0b4pre-linux-arm.deb. If you want something else just let me know.

> > 2) Where do you want packages pushed to?
> 
> If we could get a directory on ftp.mozilla.org e.g. mobile?
> 

Looks like there's already a mobile directory on ftp. I believe that I need the 'mobilebld' keys (public+private) to upload to here. Can someone ping me on IRC to give me that?

> > 3) How do you want the product built? There was talk of Xulrunner based builds
> > above, but I saw some disagreement about that on IRC. Let me know what the team
> > decides on and we can do that.
> 
> Please build xulrunner.
> 

OK. To confirm, these are the only flags I need to build with, right?:
mk_add_options MOZ_BUILD_PROJECTS="xulrunner browser"
ac_add_app_options browser --with-libxul-sdk=../xulrunner/dist --enable-libxul
ac_add_app_options browser --disable-installer

> > 4) Here's the mozconfig that is being used:
> > http://hg.mozilla.org/build/buildbot-configs/?file/7b01640de0e3/mobile/linux-arm/mozconfig.
> > Can someone have a look over this and OK it? I'm not sure if there's any
> > special things you guys use.
> 
> It looks fine except it needs to be changed to use xulrunner and we should not
> disable jemalloc.
> 

I had build issues with jemalloc on, if someone could help me debug them that would be really helpful.

> > 
> > Apologies for this taking longer than expected, it's my #1 priority now,
> > though.
> > 
> 
> Thanks. 
> 

Comment 31

[[e-mail] (:bhearsum)]

I think I'm confused here. I was under the impression that you want Firefox on Xulrunner builds done. However, the build scripts posted build browser. What am I supposed to be building here?

Comment 32

[[e-mail] (:bhearsum)]

OK. I just chatted with Doug on the phone. We are building Xulrunner for Linux-arm right now. That's all.

I don't know anything about debian build/packaging scripts; I'm going to need someone to modify them to work with Xulrunner builds. Right now, it seems to be stuck (or just taking a long time) on the 'dpkg-source' target.

Comment 33

[[e-mail] (:bhearsum)]

From IRC:
For now, we are simply going to be building xulrunner on Linux-arm. Nothing else, no packaging.

We will add support for 'make deb' in the build system. Once that is in place we will use that to create a package and upload to stage.

The former is running and will show up on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mobile shortly.

Comment 34

[[e-mail] (:bhearsum)]

Attachment: updated mozconfig, master.cfg

This config supports both nightly and dep builds. It explicitly does not package or upload builds as per stated in previous comments. I think it's pretty straightforward.

Doug, I realize you're probably unfamiliar with Buildbot config files I'd just like you to sign off on the mozconfig, though.

Comment 35

[Doug Turner (:dougt)]

Comment on attachment 304760 [details] [diff] [review]
updated mozconfig, master.cfg

only reviewing the mozconfig.  the other stuff, i don't have experience with.

enable-extensions=softkb

do you want to do what bsmedberg said, and use a relative path to softkb so you don't have to move it and patch it?

other than that, it looks fine for now.

Comment 36

[[e-mail] (:bhearsum)]

(In reply to comment #35)
> (From update of attachment 304760 [details] [diff] [review])
> only reviewing the mozconfig.  the other stuff, i don't have experience with.
> 
> enable-extensions=softkb
> 
> do you want to do what bsmedberg said, and use a relative path to softkb so you
> don't have to move it and patch it?
> 

Honestly, now that I have it working this way I'm not so inclined to change it. The patch would have to be updated too. I think the right solution here is to get whatever changes are necessary out of the patch and into the tree -- and then we can update the mozconfig if necessary.

Comment 37

[Doug Turner (:dougt)]

i agree with you.

Comment 38

[[e-mail] (:bhearsum)]

/var on mobile-linux-slave1 got mounted read-only this morning. We've seen this before and it is fixed with a kernel upgrade. I had to reboot before installing the new kernel and now (without even having a new kernel) scratchbox has stopped working. I get the following error:
ERROR: Scratchbox is not properly set up!

Here's an excerpt from the login script:
if [ ! -d $SANDBOX/scratchbox/tools/bin ] || [ ! -c $SANDBOX/dev/null ]; then
    exit_error "Scratchbox is not properly set up!"
fi

$SANDBOX/scratchbox/tools/bin translates to /scratchbox/users/cltbld/scratchbox/tools/bin. /scratchbox/users/cltbld/scratchbox is an empty directory. /scratchbox/users/cltbld/dev is also empty.

Not sure what to do here. Trying to debug it before resorting to re-installing scratchbox. I've got a funny feeling this is going to happen on every reboot though...

Comment 39

[[e-mail] (:bhearsum)]

If anyone on the Mobile team knows how to fix this, please let me know. I'm reading the install scripts to see how these things are set-up in the first place..

Comment 40

[Doug Turner (:dougt)]

as root:

/scratchbox/sbin/sbox_ctl start

Comment 41

[[e-mail] (:bhearsum)]

Attachment: centos5 startup script for scratchbox

This is a very, very, very basic script to start scratchbox on boot. Probably don't need it anywhere else right now, but I wanted it in this bug to make sure it doesn't get lost. To use it, place it in /etc/init.d and then do 'chkconfig --level 235 sbox add'.

Comment 42

[Christian Sejersen]

Ben, did you get the build to run with jemalloc? 

Comment 43

[Doug Turner (:dougt)]

christian -- it is the default linux configuration, so yes, it has jemalloc.

Comment 44

[Robert Helmer [:rhelmer]]

Comment on attachment 304760 [details] [diff] [review]
updated mozconfig, master.cfg

>diff -r 060ad80d71c7 mobile/linux-arm/mozconfig
>--- a/mobile/linux-arm/mozconfig	Fri Feb 15 14:50:48 2008 -0500
>+++ b/mobile/linux-arm/mozconfig	Thu Feb 21 12:43:41 2008 -0500

>+linux_arm_dep_factory.addStep(ShellCommand(
>+    command = ['/scratchbox/login', '-p', '-d', 'dep/mozilla',
>+               'patch -N -p0 -i ../softkb.diff'],
>+    description=['patching', 'for', 'softkb'],
>+    descriptionDone=['patched', 'for', 'softkb'],
>+    # this will retrun 1 on subsequent patches, we have to ignore it

>+linux_arm_nightly_factory.addStep(ShellCommand(
>+    command = ['/scratchbox/login', '-p', '-d', 'nightly/mozilla',
>+               'patch -N -p0 -i ../softkb.diff'],
>+    description=['patching', 'for', 'softkb'],
>+    descriptionDone=['patched', 'for', 'softkb'],
>+    # this will retrun 1 on subsequent patches, we have to ignore it

Please fix the typo in the comments :)

>+#linux_arm_nightly_factory.addStep(ShellCommand(
>+#    command = ['/scratchbox/login', '-p', '-d', 'mozilla/%s' % OBJDIR,
>+#               'make package'],
>+#    description=['make package'],
>+#    descriptionDone=['make package'],
>+#    haltOnFailure=True
>+#))
>+#linux_arm_nightly_factory.addStep(SboxGetBuildID(
>+#    basedir='mozilla'
>+#))
>+#linux_arm_nightly_factory.addStep(MozillaStageUpload(
>+#    objdir='/scratchbox/users/cltbld/home/cltbld/mozilla/%s' % OBJDIR,
>+#    username=STAGE_USERNAME,
>+#    milestone='mobile',
>+#    remoteHost=STAGE_SERVER,
>+#    remoteBasePath=STAGE_BASE_PATH,
>+#    platform='linux',
>+#    group=STAGE_GROUP,
>+#    sshKey=STAGE_SSH_KEY,
>+#    releaseToLatest=False,
>+#    releaseToDated=False,
>+#    releaseToTinderboxBuilds=True,
>+#    tinderboxBuildsDir='mobile-linux-arm',
>+#    dependToDated=True
>+#))
>+#

Why is this (and other) steps commented out? Do you plan on re-enabling them as-is? If not, I'd say just remove them.

Comment 45

[[e-mail] (:bhearsum)]

(In reply to comment #43)
> christian -- it is the default linux configuration, so yes, it has jemalloc.
> 

Actually, the mozconfig has --disable-jemalloc at the moment. Sounds like I should turn it back on and have someone investigate the compile failures.

Comment 46

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch as landed

changeset:   5:09de4f042179
tag:         tip
user:        [e-mail]
date:        Tue Feb 26 10:20:31 2008 -0500
files:       mobile/linux-arm/mozconfig mobile/master.cfg

Comment 47

[Christian Sejersen]

(In reply to comment #45)
> (In reply to comment #43)
> > christian -- it is the default linux configuration, so yes, it has jemalloc.
> > 
> 
> Actually, the mozconfig has --disable-jemalloc at the moment. Sounds like I
> should turn it back on and have someone investigate the compile failures.
> 

dougt has a patch for it.

Comment 48

[[e-mail] (:bhearsum)]

Can you get that checked in?

Comment 49

[[e-mail] (:bhearsum)]

Christian and I chatted on IRC about --disable-jemalloc. Until dougt's patch is landed we'll leave --disable-jemalloc in the mozconfig. We can flip the switch on it after that.

Comment 50

[Doug Turner (:dougt)]

youre not blocked on he jemalloc stuff, right?

Comment 51

[[e-mail] (:bhearsum)]

The only thing left to do in this bug is flip on jemalloc. Packaging/uploading is being tracked in bug 418852 and is blocked by debian packaging support.

Comment 52

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #51)
> The only thing left to do in this bug is flip on jemalloc. 
Doug/Brad: any update on this?


> Packaging/uploading
> is being tracked in bug 418852 and is blocked by debian packaging support.
Do we *require* deb packaging support in place before we can close this bug#414622 or are the mobile builds we have now enough?

Comment 53

[Doug Turner (:dougt)]

john,
can't we just disable jemalloc in the mozconfig until jemalloc supports the platform we are targeting.  There will be other tweaks we will have to make to the build config as time goes one.  jemalloc can just be one of hem.

Comment 54

[Christian Sejersen]

dougt, can't you check-in your patch since it is ARM-only?

Comment 55

[[e-mail] (:bhearsum)]

> > Packaging/uploading
> > is being tracked in bug 418852 and is blocked by debian packaging support.
> Do we *require* deb packaging support in place before we can close this
> bug#414622 or are the mobile builds we have now enough?
> 

As mentioned in comment #51, packaging/uploading is being tracked in a follow-up bug, 418852.

(In reply to comment #53)
> john,
> can't we just disable jemalloc in the mozconfig until jemalloc supports the
> platform we are targeting.  There will be other tweaks we will have to make to
> the build config as time goes one.  jemalloc can just be one of hem.
> 

jemalloc is disabled right now, I left this bug open to track it so we didn't forget about it. I'm OK with closing the bug, though, so if you want to, feel free.

Comment 56

[Doug Turner (:dougt)]

@christian: see bug 418509

Comment 57

[Christian Sejersen]

@doug: I am aware of that bug but the patch you have right now will fix our immediate problem and allow us to do the build on Maemo. Can't we then keep the other bug open to make it work across other environments?

Comment 58

[Doug Turner (:dougt)]

we can build _now_ w/o jemalloc.  I have expressed multiple times to do that in this bug.

Comment 59

[[e-mail] (:bhearsum)]

(In reply to comment #58)
> we can build _now_ w/o jemalloc.  I have expressed multiple times to do that in
> this bug.
> 

I think you're implying that we're not building mobile now? Apologies if that's not what you meant, but we _are_ building it. I mentioned this in comment #33, but perhaps not clearly enough. You can see results here: http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mobile

Comment 60

[Doug Turner (:dougt)]

1) jemalloc is not part of this bug.

2) there is a bug in QEMU..  you are going to have to remove /tmp/qemu.log.  This file grow unbounded.
3) could we also push tar.gz of release

4) the clobber builds are red, not sure if they have ever gone green

Comment 61

[[e-mail] (:bhearsum)]

I'll probably have time to take care of that stuff this week. I was just looking over things and realized that I don't have the mobilebld keys -- can you get them to me somehow?

Comment 62

[[e-mail] (:bhearsum)]

Attachment: [checked in] updates to address dougt's comments

This patch does a few things:
* Change objdir name (s/fx/xr/) to reflect what is actually being built
* Remove unused code/imports (buildbotcustom stuff)
* Add BuildStep to remove qemu log at the start of a build
* Add steps to package and upload tar.bz2 packages to stage.

Comment 63

[[e-mail] (:bhearsum)]

Alright, Brad sent me over the mobilebld keys today.

Comment 64

[[e-mail] (:bhearsum)]

Comment on attachment 308901 [details] [diff] [review]
[checked in] updates to address dougt's comments

changeset:   9:40e41ba19bba
tag:         tip
user:        Ben Hearsum <[e-mail]>
date:        Mon Mar 24 08:00:34 2008 -0400
files:       mobile/master.cfg

Comment 65

[[e-mail] (:bhearsum)]

Attachment: [checked in] update group to match actual stage group ('mobilebld')

Comment 66

[[e-mail] (:bhearsum)]

I've updated the buildbot master with these changes -- it's not spitting out builds right now though (see bug 424754) so we can't fully test it yet.

Comment 67

[[e-mail] (:bhearsum)]

Attachment: final mobile config update, for realz

Comment 68

[[e-mail] (:bhearsum)]

Attachment: final mobile config update, for realz

Comment 69

[Robert Helmer [:rhelmer]]

Comment on attachment 313206 [details] [diff] [review]
final mobile config update, for realz

"-p" should make mkdir exit nicely if the dir already exists so the "haltOnFailure" doesn't need to change.

r+ with that :)

Comment 70

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch as landed (use mkdir -p, as per rhelmer's comment)

changeset:   16:bcce69b28148
tag:         tip
user:        Ben Hearsum <[e-mail]>
date:        Wed Apr 02 15:19:51 2008 -0700
files:       mobile/master.cfg

Comment 71

[[e-mail] (:bhearsum)]

And we are done here.