422206 - Setup Windows Mobile trunk tinderbox

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Doug Turner (:dougt)]

I will set up temp tinderbox to work out the process.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Reassigning after  chat on irc.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Bug#460777 created two VMs for DougT and Blassey to experiment on. They've written up the steps needed to modify existing win32 ref images so they could support WinCE builds. See details at: https://wiki.mozilla.org/Mobile/Build/StepsToModifyRefVM

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Brad did all his work in mobile-win32-experiment01, so I'm trying a clean setup in mobile-win32-experiment02. Most installed just fine into C, leaving 2.4gb free. Working through some questions with Brad.

Comment 4

[Aki Sasaki (not active)]

John -- let me know where you left off; I can work on this.

Comment 5

[Aki Sasaki (not active)]

I believe mobile-win32-experiment02 is set up (leaving the PATH setup to a buildbot step).  I was able to build xulrunner in /e/builds/moz2_slave/mozilla-central

The next steps would seem to be:

1) joduinn wants someone to verify that mobile-win32-experiment02 is set up correctly
2) determine what to do about mq.  if the patches are going to land soon, that's one solution; if we determine that enabling it on the win32 ref image is not harmful, that's another.
3) push changes to the ref image and existing win32 build slaves
4) get the builds going in staging
5) get the builds going in production

Comment 6

[Doug Turner (:dougt)]

1) how do I/blassey gain access to viewing mobile-win32-experiment02

2) i do not imagine that the MQ will go away any time soon.

Comment 7

[Brad Lassey [:blassey] (use needinfo?)]

I don't see how enabling MQ could be harmful

Comment 8

[Aki Sasaki (not active)]

Attachment: optimized mozconfig built from blassey's debug mozconfig

Looks like it's standard practice to check these into buildbot-configs... and there's a staging one.  I'm just wget'ing from people.m.c right now.

Comment 9

[Aki Sasaki (not active)]

Attachment: working staging patch for wince builds

Currently, this can only build on mobile-win32-experiment02 as the appropriate changes haven't been applied to the pool/ref image.

I've successfully run this on staging; results are available on the MozillaTest tinderbox page (and the staging-master buildbot waterfall).  Builds are being pushed to staging-stage:

http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-browser/
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mobile-browser-wince-arm/

with timestamps in the directory name for tinderbox-builds.  The package is a tar.gz which doesn't seem right but is the default for the 'make package'.

I've also added the code to potentially upload the Nokia debs into a timestamped directory as well (bug 469294), but we won't see if it works until we debug the compilation timeout.

If this looks good, I'll create and attach a production patch.

Comment 10

[Aki Sasaki (not active)]

Also: I'm building in /e/builds/moz2_slave/mozilla-central because I found building in /e/builds/moz2_slave/mobile-wince-arm-dep/build/mozilla-central broke with a popup "memory could not be read" error during linking.  If there's a better directory to use I can change this.

Comment 11

[Aki Sasaki (not active)]

Attachment: staging/production windows mobile tinderbox

I've moved the builds to /e/builds/mobile and create zip files instead of tar.gz files.  (When bug 474530 is fixed, we can create cab files as well.)

This patch will also create timestamped directories (bug 469294).  Tested on staging.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 359205 [details] [diff] [review]
staging/production windows mobile tinderbox

Don't want to step on Catlee's review now may be a good time to move this build logic into factory.py. Any chance you can do that?

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 359205 [details] [diff] [review]
staging/production windows mobile tinderbox

Yes, I would love to see this in its own factory.

Also, some minor style nits:
 - use spaces after commas in lists (e.g. ["mobile-linux-arm-dep","mobile-wince-arm-dep"] should be ["mobile-linux-arm-dep", "mobile-wince-arm-dep"])
 - there are a few instances where you're settings description = ['word1 word2'] as opposed to ['word1', 'word2'], is that on purpose?

Other than that, it looks good to me.

Comment 14

[Chris AtLee [:catlee]]

Comment on attachment 359205 [details] [diff] [review]
staging/production windows mobile tinderbox

On further thought, I agree with bhearsum; this is a good time to move this code into factory.py.

Comment 15

[Aki Sasaki (not active)]

Attachment: MobileBuildFactory patch

I've created a factory for mobile builds in factory.py, as requested.
There isn't much shared between the linux-arm and wince-arm builds, but this will help prevent duplicate code in production/staging mobile_master.py.

Tested on staging for both linux and wince.  patch 1 of 2.

Comment 16

[Aki Sasaki (not active)]

Attachment: buildbot-configs patch using MobileBuildFactory

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 359719 [details] [diff] [review]
buildbot-configs patch using MobileBuildFactory

When we deploy this on production this will have to change. We keep MSVC and the SDKs on /d.

>+mk_add_options AR=@topsrcdir@/build/wince/tools/bin/arm-wince-lib
>+
>+ac_add_options --target=arm-wince
>+ac_add_options --enable-win32-target=WINCE
>+ac_add_options --enable-default-toolkit=cairo-windows
>+
>+ac_add_options --with-wince-sdk="c:/program files/windows mobile 6 sdk/pocketpc"

Same with this.


This looks great. I'm happy to see this file be cleaned up. On a side note, when are we going to deploy the win32 mobile ref platform stuff onto the rest of our machines?

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 359718 [details] [diff] [review]
MobileBuildFactory patch

>diff -r e9c504ba889e process/factory.py

>+class MobileBuildFactory(MozillaBuildFactory):
>+    def __init__(self, configRepoPath, mobileRepoPath, platform,
>+                 configSubDir, mozconfig, objdir="objdir",
>+                 stageUsername=None, stageSshKey=None, stageServer=None,
>+                 stageBasePath=None, stageGroup=None, packageGlob=None,
>+                 patchRepoPath=None, baseWorkDir='build',
>+                 scratchboxPath="/scratchbox/moz_scratchbox",
>+                 **kwargs):

I think it'd be much better to do this with subclassing. It's mostly ok how it is for now, but we're only going to be adding more platforms as time goes on and the 'if platform == blah' conditionals are going to get more complicated. You can probably do better argument validation that way, too. Eg, patchRepoPath can become required for win32, packageGlob can be set sanely by default, et. al.

>+        """
>+    mobileRepoPath: the path to the mobileRepo (mobile-browser)
>+    platform: the mobile platform (linux-arm, wince-arm)
>+    patchRepoPath: the path to the MQ repo, if any.  To be phased out.
>+    baseWorkDir: the path to the default slave workdir...
>+                 SBOX_HOME for linux, /e/builds/mobile for wince.

Why do you need to work outside of the builder's directory on windows? On Linux we only do it because we have to build inside of scratchbox. Unless there's no other way please don't reach outside of the builder's directory.

>+    scratchboxPath: the path to moz_scratchbox for linux-arm builds.
>+        """
>+        MozillaBuildFactory.__init__(self, **kwargs)
>+        self.platform = platform
>+        self.configRepository = self.getRepository(configRepoPath)
>+        self.mobileRepository = self.getRepository(mobileRepoPath)
>+        self.mobileBranchName = self.getRepoName(self.mobileRepository)
>+        self.configSubDir = configSubDir
>+        self.mozconfig = mozconfig
>+        self.scratchboxPath = scratchboxPath
>+        self.stageUsername = stageUsername
>+        self.stageSshKey = stageSshKey
>+        self.stageServer = stageServer
>+        self.stageBasePath = stageBasePath
>+        self.stageGroup = stageGroup
>+        self.packageGlob = packageGlob
>+        self.patchRepository = None
>+        if patchRepoPath:
>+            self.patchRepository = self.getRepository(patchRepoPath)
>+        self.baseWorkDir = baseWorkDir
>+        self.objdir = objdir
>+        self.mozconfig = 'configs/%s/%s/mozconfig' % (self.configSubDir,
>+                                                      self.mozconfig)
>+
>+        assert self.platform in ('linux-arm', 'wince-arm')
>+        if self.platform == 'linux-arm':
>+            self.addLinuxBuildSteps()
>+        elif self.platform == 'wince-arm':
>+            self.addWinceBuildSteps()
>+        self.addUploadSteps()
>+
>+    def addLinuxBuildSteps(self):
>+        self.baseWorkDir = '%s/build' % self.baseWorkDir
>+        self.addStep(ShellCommand,
>+            command = 'rm -f /tmp/*_cltbld.log',
>+            description=['removing', 'logfile'],
>+            descriptionDone=['removed', 'logfile'],
>+            haltOnFailure=False,
>+            flunkOnFailure=False,
>+            warnOnFailure=False

These are all false by default, just drop them from this and the other steps.

>+        self.addStep(ShellCommand,
>+            command=['hg', 'clone', self.configRepository, 'configs'],
>+            workdir=self.baseWorkDir,
>+            description=['checking', 'out', 'configs'],
>+            descriptionDone=['checked', 'out', 'configs'],
>+            haltOnFailure=True
>+        )

I see a lot of duplicated code related to checking out the source. Please factor it somewhere common.

>diff -r e9c504ba889e steps/transfer.py
>--- a/steps/transfer.py	Thu Jan 29 16:41:34 2009 +1300
>+++ b/steps/transfer.py	Fri Jan 30 00:20:31 2009 -0800
>@@ -121,19 +121,20 @@
>                                      sshKey=sshKey,
>                                      releaseToDated=releaseToDated,
>                                      releaseToLatest=releaseToLatest,
>                                      releaseToTinderboxBuilds=releaseToTinderboxBuilds,
>                                      tinderboxBuildsDir=tinderboxBuildsDir,
>                                      dependToDated=dependToDated,
>                                      uploadCompleteMar=uploadCompleteMar,
>                                      uploadLangPacks=uploadLangPacks)
> 
>-        assert platform in ('win32', 'linux', 'macosx')
>+        if (packageGlob == None):
>+            assert platform in ('win32', 'linux', 'macosx')

I realize this works, but it really doesn't make sense. Please pass in one of the existing platforms or add a new one.


I think this still requires a bit of work, but the core logic is all there.

Comment 19

[Aki Sasaki (not active)]

Attachment: WinceBuildFactory/MaemoBuildFactory patch

> when are we going to deploy the win32 mobile ref platform stuff onto the rest
> of our machines?

The mobile team is working towards getting rid of the patch queue for various reasons.  One reason is there are objections to enabling mq on production build slaves.  Once the patchRepository goes away we can deploy the mobile sdk on our win32 slaves.  We can install on /d at that point unless there's something preventing us that I don't know about (joduinn?).

> I think it'd be much better to do this with subclassing.

Done.

> Why do you need to work outside of the builder's directory on windows?

I think (I may be wrong, but I strongly think) the following command is one character too long for Windows' fragile little mind:

/e/builds/moz2_slave/mobile-wince-arm-dep/mozilla-central/objdir/xulrunner/dist/sdk/bin/arm-wince-link [snip long cmdln argument list for legibility]

I say possibly one character because arm-wince-res and arm-wince-gcc commands finish successfully.  We're definitely within 7 characters of the limit, however, because moving /e/builds/moz2_slave to /e/moz2_slave fixes this.  Renaming the builddir from mobile-wince-arm-dep to wince-arm-dep also fixes this.

I'm doing the latter since it's less of a footprint (i.e. won't require changing the buildbot.tac on every single win32 build slave), and since 'wince' implies 'mobile'.  Moving moz2_slave to /e/moz2_slave could give us even more headroom if needed in the future.

> These are all false by default, just drop them from this and the other steps.

These were necessary for bare 'hg clone repository' commands; if the repository was previously cloned, this would error out and turn the tinderbox red.

However, by checking if the directory exists before running hg clone, the step no longer fails.  Removed.

> I see a lot of duplicated code related to checking out the source. Please
factor it somewhere common.

Done.  The patchRepository steps makes this a bit more complicated, but once the patch queue is retired we can remove that code.

> I realize this works, but it really doesn't make sense. Please pass in one of
> the existing platforms or add a new one.

I think it makes sense:  MozillaStageUpload only uses platform to hack a default packageGlob afaict.  But I'm not tied to it.  I'm passing win32 for wince-arm and linux for linux-arm now.

patch 1 of 2.

Comment 20

[Aki Sasaki (not active)]

Attachment: buildbot-configs patch using WinceBuildFactory/MaemoBuildFactory

Comment 21

[[e-mail] (:bhearsum)]

(In reply to comment #19)
> Created an attachment (id=359976) [details]
> WinceBuildFactory/MaemoBuildFactory patch
> 
> > when are we going to deploy the win32 mobile ref platform stuff onto the rest
> > of our machines?
> 
> The mobile team is working towards getting rid of the patch queue for various
> reasons.  One reason is there are objections to enabling mq on production build
> slaves.  Once the patchRepository goes away we can deploy the mobile sdk on our
> win32 slaves.  We can install on /d at that point unless there's something
> preventing us that I don't know about (joduinn?).
> 
Ok, that sounds fine. I totally agree w.r.t mq on production machines. We can leave the experimental VMs how they are, but we'll need to adjust the paths when we install VC on /d.

> I'm doing the latter since it's less of a footprint (i.e. won't require
> changing the buildbot.tac on every single win32 build slave), and since 'wince'
> implies 'mobile'.  Moving moz2_slave to /e/moz2_slave could give us even more
> headroom if needed in the future.
> 

Sounds good.

> > These are all false by default, just drop them from this and the other steps.
> 
> These were necessary for bare 'hg clone repository' commands; if the repository
> was previously cloned, this would error out and turn the tinderbox red.
> 
> However, by checking if the directory exists before running hg clone, the step
> no longer fails.  Removed.
> 
Yeah, sorry. I forgot to check ShellCommand/Mercurial when I was looking at defaults.


I'll review this patches shortly.

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 359976 [details] [diff] [review]
WinceBuildFactory/MaemoBuildFactory patch

Looks fine

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 359977 [details] [diff] [review]
buildbot-configs patch using WinceBuildFactory/MaemoBuildFactory

>+wince_arm_dep_factory = WinceBuildFactory(
>+    hgHost = HGHOST,
>+    repoPath = 'mozilla-central',
>+    configRepoPath = CONFIG_REPO_PATH,
>+    configSubDir = CONFIG_SUBDIR,
>+    mozconfig = "wince/mobile-browser/nightly",
>+    stageUsername = STAGE_USERNAME,
>+    stageGroup = STAGE_GROUP,
>+    stageSshKey = STAGE_SSH_KEY,
>+    stageServer = STAGE_SERVER,
>+    stageBasePath = STAGE_BASE_PATH,
>+    packageGlob = "xulrunner/dist/*.zip mobile/dist/*.zip",
>+    mobileRepoPath = 'mobile-browser',
>+    patchRepoPath = 'users/blassey_mozilla.com/wince-patches',
>+    platform = 'wince-arm',
>+    baseWorkDir = ".",
>+    buildToolsRepoPath = BUILD_TOOLS_REPO_PATH,
>+    buildSpace = 0

Any reason not to set a proper buildSpace value?

Looks fine otherwise.

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 359977 [details] [diff] [review]
buildbot-configs patch using WinceBuildFactory/MaemoBuildFactory

per IRC, we'll set buildSpace for the win32 builds when they move onto production machines

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 359976 [details] [diff] [review]
WinceBuildFactory/MaemoBuildFactory patch

changeset:   185:8a945f301a98

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 359977 [details] [diff] [review]
buildbot-configs patch using WinceBuildFactory/MaemoBuildFactory

changeset:   718:0271d3dc6863

Comment 27

[Aki Sasaki (not active)]

Attachment: mozconfig: s/topsrcdir/TOPSRCDIR/g

Oops -- looks like my mozconfigs in the patch were slightly stale.

wince builds are now working successfully, with this patch.

Comment 28

[Aki Sasaki (not active)]

Attachment: VS9 buildbot configs patch

These patches do the following:

1) enable VS9 builds
2) disable patch queue steps
3) create and upload cab files
4) add a nightly scheduler

The build slaves list for production isn't complete; I think we should add to the list as we roll out VS9.  When VS9 is fully rolled out we can remove and nuke the experimental VMs.

Works in staging.  Patch one of two.

Comment 29

[Aki Sasaki (not active)]

Attachment: VS9 buildbotcustom patch

Updates to factory.py

Comment 30

[[e-mail] (:bhearsum)]

Comment on attachment 362787 [details] [diff] [review]
VS9 buildbot configs patch

>diff -r 5bb75f072a3e mozilla2-staging/mobile_config.py

>+wince_dep_env = mobile_config.wince_arm_env.copy()
>+wince_dep_env['TOPSRCDIR'] = '/e/builds/moz2_slave/wince-arm-dep/mozilla-central'

Why are you setting this here and not in mobile_config.py?

This looks fine otherwise.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 362788 [details] [diff] [review]
VS9 buildbotcustom patch

>diff -r 96bec7fadee3 process/factory.py

>@@ -2140,19 +2113,19 @@
>             username=self.stageUsername,
>             milestone=self.mobileBranchName,
>             remoteHost=self.stageServer,
>             remoteBasePath=self.stageBasePath,
>             platform=platform,
>             group=self.stageGroup,
>             packageGlob=self.packageGlob,
>             sshKey=self.stageSshKey,
>             uploadCompleteMar=False,
>-            releaseToLatest=True,
>+            releaseToLatest=self.nightly,
>             releaseToDated=False,

Please releaseToDated for nightlies, too. 

Glad to see the mq stuff being dropped - r=bhearsum with that change.

Comment 32

[Aki Sasaki (not active)]

Attachment: VS9 buildbot configs patch 2

Moved the wince_dep_env and wince_nightly_env settings to mobile_config.py; otherwise unchanged.

For factory.py, releaseToDated=self.nightly would work.

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 362788 [details] [diff] [review]
VS9 buildbotcustom patch

changeset:   199:337119db4774

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 362921 [details] [diff] [review]
VS9 buildbot configs patch 2

changeset:   954:a809f6525165

Comment 35

[[e-mail] (:bhearsum)]

the master has been restarted for the two latest patches.

Comment 36

[Aki Sasaki (not active)]

Attachment: nightly tinderbox, add moz2-win32-slave22/23, change mozconfig for dougt

Ok.  Things are looking good.

Doug just asked me to update the mozconfig.
Also, nightly builders aren't showing up on the Mobile tinderbox page.
Also also, I'm adding moz2-win32-slave22 and 23 to the list of wince builders since they have VS9 installed.

I think I'll close this bug after this patch goes through and open up another for pushing VS9 onto the rest of the win32 slaves... we may be able to roll out VS9 and the win32 reboot changes at the same time.

Comment 37

[[e-mail] (:bhearsum)]

Comment on attachment 363392 [details] [diff] [review]
nightly tinderbox, add moz2-win32-slave22/23, change mozconfig for dougt

Looks fine to me

Comment 38

[Aki Sasaki (not active)]

[14:24]	<ted>	aki: presumably, if you put your mozconfigs in hg in the buildbot-configs repo, developers can update them when needed
	which is the best option
[14:25]	<aki>	oh, ok
	they're there
[14:27]	<ted>	anyone with hg access can push to buildbot-configs

removing mozconfig.wince dependency.

Comment 39

[Aki Sasaki (not active)]

Attachment: as above, with win32 slave22+23 in prod config.py

we'll need to update BuildSlaves.py too.

Comment 40

[Aki Sasaki (not active)]

Comment on attachment 363398 [details] [diff] [review]
as above, with win32 slave22+23 in prod config.py

obsoleting; i'll add more slaves in the next patch.

Comment 41

[Aki Sasaki (not active)]

Attachment: mobile_slaves + 10 vs9 slaves + above

Ok.  This patch:

 - adds moz2-linux-slave18 and 19 to the production pool (+ linux-arm builds) for bug 477182
 - adds moz2-win32-slave22 and 23 to the production pool (+ win32-arm builds) for bug 477178
 - pulls the mobile slave list from mobile_master.py to mobile_config.py.  This isn't anywhere near as complex as config.BRANCHES yet.  If/when we need to support multiple mobile branches and environments we'll probably need to go that route.
 - adds the 10 win32 build slaves that now have VS9 installed to the win32-arm pool.
 - adds the linux-arm and wince-arm nightly builds to the Mobile tinderbox waterfall.
 - updates the wince mozconfig per discussions on #mobile.

Comment 42

[[e-mail] (:bhearsum)]

Comment on attachment 363726 [details] [diff] [review]
mobile_slaves + 10 vs9 slaves + above

>diff -r 2f7bb2d3fef2 mozilla2-staging/mobile_config.py
>+    'wince-arm': [
>+        'mobile-win32-experiment01',
>+        'mobile-win32-experiment02',
>+        'moz2-win32-slave13',
>+        'moz2-win32-slave14',
>+        'moz2-win32-slave15',
>+        'moz2-win32-slave16',
>+        'moz2-win32-slave17',
>+        'moz2-win32-slave18',
>+        'moz2-win32-slave19',
>+        'moz2-win32-slave20',
>+        'moz2-win32-slave22',
>+        'moz2-win32-slave23',

I guess 21 and 1-12 aren't setup yet?

Comment 43

[[e-mail] (:bhearsum)]

Comment on attachment 363726 [details] [diff] [review]
mobile_slaves + 10 vs9 slaves + above

As per IRC, the aforementioned slaves are not setup yet. r=bhearsum, in that case.

Comment 44

[[e-mail] (:bhearsum)]

Comment on attachment 363726 [details] [diff] [review]
mobile_slaves + 10 vs9 slaves + above

changeset:   959:1feecd1ae3e4

Comment 45

[[e-mail] (:bhearsum)]

Aki,

I've imported the two MSIs needed into the mofo repo for future usage:
Checking in wm6-pro-sdk.msi;
/mofo/ref-platforms/win32/wm6-pro-sdk.msi,v  <--  wm6-pro-sdk.msi
initial revision: 1.1
done
RCS file: /mofo/ref-platforms/win32/wm6.1-pro-images.msi,v
done
Checking in wm6.1-pro-images.msi;
/mofo/ref-platforms/win32/wm6.1-pro-images.msi,v  <--  wm6.1-pro-images.msi
initial revision: 1.1
done

I'm also in the process of downloading the VS2008 VM to one of our datastores (/vmfs/volumes/netapp-c-001/iso-images) for easier installation in the future.

I've updated the ref platform doc for both of these changes.

Comment 46

[Aki Sasaki (not active)]

Attachment: add remaining moz2-win32 slaves; retire mobile-win32-experiment slaves

This should hopefully be the final patch.

Comment 47

[Aki Sasaki (not active)]

Comment on attachment 364350 [details] [diff] [review]
add remaining moz2-win32 slaves; retire mobile-win32-experiment slaves

oops.

Comment 48

[[e-mail] (:bhearsum)]

Comment on attachment 364350 [details] [diff] [review]
add remaining moz2-win32 slaves; retire mobile-win32-experiment slaves

yay!

Comment 49

[Aki Sasaki (not active)]

Comment on attachment 364350 [details] [diff] [review]
add remaining moz2-win32 slaves; retire mobile-win32-experiment slaves

Changeset:  964:3d87831f92d5

Comment 50

[Aki Sasaki (not active)]

done.