485109 - WinCE 5.0 builds: Firefox on m-c & m-1.9.2

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We need a base WinCE 5 Standard SDK tinderbox set up; there are some differences between Windows Mobile and CE 5 base that are tripping things up.

The CE5 Standard SDK is at http://www.microsoft.com/DownLoads/details.aspx?familyid=FA1A3D66-3F61-4DDC-9510-AE450E2318C3&displaylang=en -- if that can get installed to the image or a machine, then it's just a custom mozconfig.  I'll upload one in a bit.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Where are we going with this in the longer-term ? Removing the Mobile SDK in favour of the CE5 one ? Or building and testing against both because there is a mix of devices out there ?

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Both -- there's a mix of devices and they take slightly different codepaths (in particular, some things that are present on WinMo are optional/not available under base CE)

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #2)
> Both -- there's a mix of devices and they take slightly different codepaths (in
> particular, some things that are present on WinMo are optional/not available
> under base CE)


1) Errr... is it possible to have *both* Mobile SDK *and* CE5 on the same Windows VM? 

Note: we use one ref platform VM for doing win32 builds *and* Mobile SDK builds. Is it possible to install CE5 onto the same VM as Mobile SDK? Will installing CE5 cause any driver changes or binary compatibility changes for either the win32 builds or mobile SDK builds already being produced? Or are you asking us to spin up a new totally separate CE5-specific infrastructure?

2) You ask about builds here. How do you want us to handle unittests and talos runs?

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to comment #3)
> (In reply to comment #2)
> > Both -- there's a mix of devices and they take slightly different codepaths (in
> > particular, some things that are present on WinMo are optional/not available
> > under base CE)
> 
> 
> 1) Errr... is it possible to have *both* Mobile SDK *and* CE5 on the same
> Windows VM? 

Yep, there's no problem -- it installs into a separate self-contained location, so putting it on the same ref platform VM as win32/winmo/etc. is fine.  No driver/binary/etc. issues.

> 2) You ask about builds here. How do you want us to handle unittests and talos
> runs?

For now just builds would be great -- the main problem I'm trying to catch is code that gets checked in for Windows Mobile that doesn't compile with the base SDK.  Unit tests/talos runs would be a future thing; we'd have to figure out how to get some CE platforms that can run in that kind of environment as well.

Comment 5

[Aki Sasaki (not active)]

Clarification: Stuart was under the impression you meant CE6. Do you mean 5 or 6?  Rereading this bug, it seems you do mean 5.

Also, what time frame is this needed/wanted?  And do you need to actually use these builds or just see that it compiles?

Comment 6

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

For the SDK, explicitly the CE5 SDK.  Ideally, you would also copy:

/Program Files/Windows Mobile 6 SDK/PocketPC/Include/Armv4i/ddraw.h
/Program Files/Windows Mobile 6 SDK/PocketPC/Lib/Armv4i/ddraw.lib

to

/Program Files/Windows CE Tools/wce500/STANDARDSDK_500/Include/Armv4i
and ..../Lib/Armv4i, repsectively

(we need the updated ddraw header/lib as opposed to the old one that came with the CE5 SDK).

It would be great to have these builds published somewhere.

Comment 7

[Aki Sasaki (not active)]

* Nightly builds of Firefox and Fennec, for trunk and 1.9.1.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Vlad, could upload the promised mozconfig please.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I found the wiki page and adapted the mozconfig there. Getting nightly updates for for WinCE 5 builds depends on bug 458950.

Comment 10

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

> * Nightly builds of Firefox and Fennec, for trunk and 1.9.1.

This should just be Firefox on trunk.  Fennec as well at some point, but the initial demand is just Firefox -- also, just trunk, no 1.9.1.

Bugzilla was acting up on me last night, but one additional SDK needs to be installed, which is at:

http://fs/public/Users/vladimir/nvidia/TegraInstaller_KD3519362_BR89_042.msi

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Ok, Firefox on mozilla-central only. The outputs we want for nightly builds are
* a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  
* a complete update so that we can do nightly updates

Is that right ?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: WIP buildbot-configs

I'm working on moz2-win32-slave03 and 
  staging-master:/builds/build/nthomas-playpit
with
  http://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs/
  http://hg.mozilla.org/users/nthomas_mozilla.com/buildbotcustom/

This is the patch against buildbot-configs to add wince build to the master. We'll need to fix the XXX for the update platform, and remove the disable-updater later on.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: WIP buildbotcustom

Minor tweaks to buildbotcustom to support wince builds. 

There are also some local tweaks to the buildbot-configs on the master, in order to get WinCE builds on the single slave without reboots and other builds running. Port 8019 for the waterfall.

This and the previous patch get us through the compile step, and dolske reports that the resulting dist/bin works on a device.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Packaging work to be done in bug 503469.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Notes on SDK install

This is what's been done to moz2-win32-slave03. Sounds like having builds working in staging is going to be sufficient in the short term, so there's no rush to create an OPSI package and deploy yesterday. Based on the changes I observed on the c and d drives I don't think it'll affect anything else.

Note that I haven't put the three entries the Tegra installer adds to the PATH on the copy we use for the compile. Vlad says we don't need them.

Comment 16

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to comment #11)
> Ok, Firefox on mozilla-central only. The outputs we want for nightly builds are
> * a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  
> * a complete update so that we can do nightly updates
> 
> Is that right ?

Yep, that'd be great.

(In reply to comment #15)
> Note that I haven't put the three entries the Tegra installer adds to the PATH
> on the copy we use for the compile. Vlad says we don't need them.

Right -- I may ask you to tweak the mozconfig with a few lines, but that should be it.

Comment 17

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

> > * a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  

Oh -- forgot to mention.  The invocation of make_wince_cab.py needs -s passed for these cab files; WinCE can't install compressed cab files, and -s tells the python script to not pass /compress.  If that's complicated, .zip files would also be fine.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Thanks, I gave up on cab for now and settled for zip.

I landed another mozconfig change for testing the updater - http://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs/rev/e8b548975594, only that didn't work out. We'd probably want part of that, like the -j4 & MOZILLA_OFFICIAL.

In terms of using moz2-win32-slave03, there's also a local change to buildbotcustom to disable the mozilla-central pull because it blows away local changes (like the packaging stuff in bug 503469. I've just been doing that manually. Usually I just connect the slave when I want a build, then gracefully disconnect immediately. Haven't tested a nightly build yet.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #18)
> Thanks, I gave up on cab for now and settled for zip.

Per discussion with vlad/nthomas/myself, we've also given up on trying to produce updates for WinCE until dep bug#458950 is landed in mozilla-central.

Comment 20

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

To be clear -- we should if at all possible still produce mar files for updates, even if the infrastructure is not in place yet to actually perform an update on the target platform.

Comment 21

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #20)
> To be clear -- we should if at all possible still produce mar files for
> updates, even if the infrastructure is not in place yet to actually perform an
> update on the target platform.

Totally agreed. 

We continue to work on producing the mar files along with the zip. While we can confirm if the zips are done right by running the builds, we cannot confirm if those mar files are valid/usable until the updater patch lands, so we dont feel comfortable saying the mar files are "done".

Comment 22

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Attachment: patch for mozconfig

Here are some tweaks to the mozconfig for this.

Also, which tree are these builds showing up on?  I couldn't find them on any of the staging/etc. trees.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the mozconfig patch, I'll incorporate that into what I'm working on. 

Currently I've been working on a staging machine, so the steps to move forward are
* finish up testing to make sure we can create zip reliably in depend and nightly builds, and also mar for the latter
* test and deploy OPSI package for installing SDKs on pool of windows slaves
* check that nightly updates are working (partials generated, and can be applied)

I'm planning to work on that this week.

Comment 24

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Great, thanks!  Let me know if there's anywhere that I can help.

One thing -- in the mozconfig, I think I got the right dir for the nvidia sdk, but can you just verify that the platformlibs dir is in the right place, as specified in the config file?

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

That's the right path for where I installed the SDK.

Comment 26

[Mike Beltzner [:beltzner, not reading bugmail]]

Can we get an ETA update here? Looks like the review in bug 503469 (though we don't really need MAR support as discussed offline) is finished, and we've started hitting cases where people are checking in changes to mozilla-central which are breaking WindowsCE builds, and the cost of having to bisect in order to determine where that's failing is pretty expensive. :/

Nick: I think that, going forward, if things get at all clogged up for more than a day it would be great if someone could ping me or Vlad to make sure that we're all on the same page.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

The status here is
* the packaging fixes in bug 503469 are now landed
* next step is to deploy the two new SDKs in bug 508312. Firstly to finish up Ben's work on having auto-install packages, deploy them to staging and make sure other builds don't burn, then deploy to the production builders. This is likely 2+ days out
* in parallel, rework the buildbot-configs for the new config style in bug 508312

Vlad, do you want to track mozilla-central or mozilla-1.9.2 ?

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

And which tree would you like the WinCE build to report to ?

Comment 29

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Firefox tree, please, tracking m-c for now.. we'll need one for 1.9.2 as well once that branch gets going.

Comment 30

[Mike Beltzner [:beltzner, not reading bugmail]]

(Thanks for the update, Nick!)

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Status update:
* dep and nightly builds are running in development environment, zip and mar files are produced, and partial updates also get created (see MozillaTest tree and search for WINCE)
* need to modify config for above and add 1.9.2 branch support, then get review
* followup patch added to bug 503469 so that parallel make works
* need to check main staging environment for bustage from SDKs installs, and if no problems deploy to main pool of slaves

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: buildbot custom tweaks

Minor tweaks to support a "wince" platform where we don't build xulrunner.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: buildbot-configs

Add WinCE builds for m-c and m-1.9.2, mozconfigs are identical on both branches.

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: buildbotcustom tweaks (really!)

Minor tweaks to support a "wince" platform where we don't build xulrunner. (helps if you attach the right patch)

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 394468 [details] [diff] [review]
buildbotcustom tweaks (really!)

It would be nicer not to have this exception logic done here, but that's a much larger problem. This is totally fine.

Comment 36

[[e-mail] (:bhearsum)]

Comment on attachment 394467 [details] [diff] [review]
buildbot-configs 

>+import buildbotcustom.env
>+reload(buildbotcustom.env)
>+from buildbotcustom.env import MozillaEnvironments
>+

This is the second place this gets reloaded but I *think* it's ok since it's not a class. Eg, there won't be any old references to worry about. If you're super duper paranoid you can remove the reload in mobile_config.py.

>diff -r 6d589d507e8d mozilla2-staging/wince/mozilla-1.9.2/nightly/mozconfig
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/mozilla2-staging/wince/mozilla-1.9.2/nightly/mozconfig	Fri Aug 14 18:08:00 2009 +1200

Big mozconfig, wow.



Vlad noted that the splashscreen is missing from the build, I dunno if that's your issue or his though.

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Update VC9 environment with Tegra paths

Ben, one thing I'm not sure about is the changes the Tegra SDK makes to the environment:
* globally adds NVAPSDK = d:\sdks\tegra042   (harmless)
* appends to the PATH
    %NVAPSDK%\tools;
    %NVAPSDK%\platformlibs\bin\winxp\x86\release;
    %NVAPSDK%\3rdparty\bin\winxp\x86\release
It does this by modifying the registry.

The "winmo-arm" environment dict used for WinCE and WinMo builds overrides PATH and the WINCE builds don't get the changes in the environment. Apparently that's not fatal as the test WinCE builds have been compiling fine, but someone may want to use stuff on those paths in the future so this patch syncs up the VC9 environment.

Other (non-unittest) builds pick up the global changes to the MSYS environment, and I'm wondering if we need to care about that. The files in those three directories look fairly safe, in the sense that there doesn't seem to be any naming overlap with utils we use already, and anyway there's only d:\mozilla-build\moztools\bin following them in PATH. Do you think we're OK to leave it as is in the environment ?

Comment 38

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We're ok as-is; we should never depend on the tools there, we're really just installing that SDK to get headers and import libs for GLES2 stuff, which are actually cross-platform.

Comment 39

[[e-mail] (:bhearsum)]

(In reply to comment #37)
 Do you think we're OK to
> leave it as is in the environment ?


Given Vlad's comment and my own investigation, yeah, I think it's fine. If we're super paranoid we could get the OPSI package to remove them - I'll leave that up to you though.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 394468 [details] [diff] [review]
buildbotcustom tweaks (really!)

committed changeset 384:b8d6623fc2c0

(In reply to comment #35)
> It would be nicer not to have this exception logic done here, but that's a much
> larger problem. This is totally fine.

Would you prefer we defined something like BRANCH_LEVEL_VARS['xulrunner_platforms'] defined in config.py ?

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 394770 [details] [diff] [review]
Update VC9 environment with Tegra paths

committed changeset 385:9da322657f3c

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 394467 [details] [diff] [review]
buildbot-configs 

committed changeset 1432:8a50eacda69f

with the make -j4 re-enabled. I'm going to land attachment 394251 [details] [diff] [review] as the trigger for the first WinCE builds so they doesn't go boom.

(In reply to comment #36)
> This is the second place this gets reloaded but I *think* it's ok since it's
> not a class. Eg, there won't be any old references to worry about. If you're
> super duper paranoid you can remove the reload in mobile_config.py.

I seem to not be paranoid today (huzzah!) so I left this as is. Similarly left the environment with the unneeded the PATH changes.

> Vlad noted that the splashscreen is missing from the build, I dunno if that's
> your issue or his though.

I'll take a look into this. Probably just a something that needs to be added to packages-static.

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Status:
* dep. builds are green on m-c and m-1.9.2, they get published to 
 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-wince/
  and similar for m-c.
* they appear on Firefox3.6 and Firefox tinderbox waterfalls, and you get two B's for Windows on tinderboxpushlog
* nightlies will appear in the firefox/nightly/latest-mozilla-1.9.2 and latest-mozilla-central directories. They get triggered at 3am Pacific like other Firefox nightlies. hg.m.o just blew up under the nightly load, so they might not be out today
* updates should be available after two nightlies have been built
* splashscreen fix attached to bug 503469

Comment 44

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Nagios monitoring for nightly files

Comment 45

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Just tried the 1.9.2 build, seems to work great!  Only thing I noticed was the splash screen image isn't getting packaged, but that's something we can fix directly in m-c (I think it's just left out of the list or something; not a huge deal).

Comment 46

[Nick Thomas [:nthomas] (UTC+12)]

Great to hear Vlad. The splashscreen issue should be fixed in the next dep build and in tomorrow's nightlies - it landed from bug 503469 just now.

Once the nagios monitoring is landed I should be able to close this out.

Comment 47

[Nick Thomas [:nthomas] (UTC+12)]

Pushed 
  http://hg.mozilla.org/build/buildbot-configs/rev/f9315d9ac7c3
to back make off to -j2, since we've been getting alot of these errors:
  ARMV4I : fatal error LNK1104: cannot open file 'xpcom.lib'

Comment 48

[Nick Thomas [:nthomas] (UTC+12)]

I've removed parallel make from the mozconfig
 http://hg.mozilla.org/build/buildbot-configs/rev/a2fbf9c38253
because we continued to hit the xpcom.lib error occasionally.

Comment 49

[Nick Thomas [:nthomas] (UTC+12)]

Lets close this out now we have CABs & try server (bug 514332), and file anything else as a followup.

Comment 50

[Tony Chung [:tchung]]

Verified fix from bug 511662