Closed Bug 485109 Opened 11 years ago Closed 10 years ago

WinCE 5.0 builds: Firefox on m-c & m-1.9.2

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: vlad, Assigned: nthomas)

References

Details

Attachments

(5 files, 4 obsolete files)

We need a base WinCE 5 Standard SDK tinderbox set up; there are some differences between Windows Mobile and CE 5 base that are tripping things up.

The CE5 Standard SDK is at http://www.microsoft.com/DownLoads/details.aspx?familyid=FA1A3D66-3F61-4DDC-9510-AE450E2318C3&displaylang=en -- if that can get installed to the image or a machine, then it's just a custom mozconfig.  I'll upload one in a bit.
Where are we going with this in the longer-term ? Removing the Mobile SDK in favour of the CE5 one ? Or building and testing against both because there is a mix of devices out there ?
Both -- there's a mix of devices and they take slightly different codepaths (in particular, some things that are present on WinMo are optional/not available under base CE)
(In reply to comment #2)
> Both -- there's a mix of devices and they take slightly different codepaths (in
> particular, some things that are present on WinMo are optional/not available
> under base CE)


1) Errr... is it possible to have *both* Mobile SDK *and* CE5 on the same Windows VM? 

Note: we use one ref platform VM for doing win32 builds *and* Mobile SDK builds. Is it possible to install CE5 onto the same VM as Mobile SDK? Will installing CE5 cause any driver changes or binary compatibility changes for either the win32 builds or mobile SDK builds already being produced? Or are you asking us to spin up a new totally separate CE5-specific infrastructure?

2) You ask about builds here. How do you want us to handle unittests and talos runs?
Component: Release Engineering → Release Engineering: Future
(In reply to comment #3)
> (In reply to comment #2)
> > Both -- there's a mix of devices and they take slightly different codepaths (in
> > particular, some things that are present on WinMo are optional/not available
> > under base CE)
> 
> 
> 1) Errr... is it possible to have *both* Mobile SDK *and* CE5 on the same
> Windows VM? 

Yep, there's no problem -- it installs into a separate self-contained location, so putting it on the same ref platform VM as win32/winmo/etc. is fine.  No driver/binary/etc. issues.

> 2) You ask about builds here. How do you want us to handle unittests and talos
> runs?

For now just builds would be great -- the main problem I'm trying to catch is code that gets checked in for Windows Mobile that doesn't compile with the base SDK.  Unit tests/talos runs would be a future thing; we'd have to figure out how to get some CE platforms that can run in that kind of environment as well.
Clarification: Stuart was under the impression you meant CE6. Do you mean 5 or 6?  Rereading this bug, it seems you do mean 5.

Also, what time frame is this needed/wanted?  And do you need to actually use these builds or just see that it compiles?
For the SDK, explicitly the CE5 SDK.  Ideally, you would also copy:

/Program Files/Windows Mobile 6 SDK/PocketPC/Include/Armv4i/ddraw.h
/Program Files/Windows Mobile 6 SDK/PocketPC/Lib/Armv4i/ddraw.lib

to

/Program Files/Windows CE Tools/wce500/STANDARDSDK_500/Include/Armv4i
and ..../Lib/Armv4i, repsectively

(we need the updated ddraw header/lib as opposed to the old one that came with the CE5 SDK).

It would be great to have these builds published somewhere.
Assignee: nobody → aki
* Nightly builds of Firefox and Fennec, for trunk and 1.9.1.
Summary: Create WinCE 5.0 tinderbox → WinCE 5.0 builds: Firefox and Fennec on m-c and 1.9.1
Vlad, could upload the promised mozconfig please.
Assignee: aki → nthomas
I found the wiki page and adapted the mozconfig there. Getting nightly updates for for WinCE 5 builds depends on bug 458950.
Depends on: 458950
> * Nightly builds of Firefox and Fennec, for trunk and 1.9.1.

This should just be Firefox on trunk.  Fennec as well at some point, but the initial demand is just Firefox -- also, just trunk, no 1.9.1.

Bugzilla was acting up on me last night, but one additional SDK needs to be installed, which is at:

http://fs/public/Users/vladimir/nvidia/TegraInstaller_KD3519362_BR89_042.msi
Ok, Firefox on mozilla-central only. The outputs we want for nightly builds are
* a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  
* a complete update so that we can do nightly updates

Is that right ?
Summary: WinCE 5.0 builds: Firefox and Fennec on m-c and 1.9.1 → WinCE 5.0 builds: Firefox on m-c
Attached patch WIP buildbot-configs (obsolete) — Splinter Review
I'm working on moz2-win32-slave03 and 
  staging-master:/builds/build/nthomas-playpit
with
  http://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs/
  http://hg.mozilla.org/users/nthomas_mozilla.com/buildbotcustom/

This is the patch against buildbot-configs to add wince build to the master. We'll need to fix the XXX for the update platform, and remove the disable-updater later on.
Attached patch WIP buildbotcustom (obsolete) — Splinter Review
Minor tweaks to buildbotcustom to support wince builds. 

There are also some local tweaks to the buildbot-configs on the master, in order to get WinCE builds on the single slave without reboots and other builds running. Port 8019 for the waterfall.

This and the previous patch get us through the compile step, and dolske reports that the resulting dist/bin works on a device.
Packaging work to be done in bug 503469.
Depends on: 503469
Attached file Notes on SDK install
This is what's been done to moz2-win32-slave03. Sounds like having builds working in staging is going to be sufficient in the short term, so there's no rush to create an OPSI package and deploy yesterday. Based on the changes I observed on the c and d drives I don't think it'll affect anything else.

Note that I haven't put the three entries the Tegra installer adds to the PATH on the copy we use for the compile. Vlad says we don't need them.
(In reply to comment #11)
> Ok, Firefox on mozilla-central only. The outputs we want for nightly builds are
> * a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  
> * a complete update so that we can do nightly updates
> 
> Is that right ?

Yep, that'd be great.

(In reply to comment #15)
> Note that I haven't put the three entries the Tegra installer adds to the PATH
> on the copy we use for the compile. Vlad says we don't need them.

Right -- I may ask you to tweak the mozconfig with a few lines, but that should be it.
> > * a cab archive, eg firefox-3.6a1pre.en-US.wince-arm.cab  

Oh -- forgot to mention.  The invocation of make_wince_cab.py needs -s passed for these cab files; WinCE can't install compressed cab files, and -s tells the python script to not pass /compress.  If that's complicated, .zip files would also be fine.
Thanks, I gave up on cab for now and settled for zip.

I landed another mozconfig change for testing the updater - http://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs/rev/e8b548975594, only that didn't work out. We'd probably want part of that, like the -j4 & MOZILLA_OFFICIAL.

In terms of using moz2-win32-slave03, there's also a local change to buildbotcustom to disable the mozilla-central pull because it blows away local changes (like the packaging stuff in bug 503469. I've just been doing that manually. Usually I just connect the slave when I want a build, then gracefully disconnect immediately. Haven't tested a nightly build yet.
(In reply to comment #18)
> Thanks, I gave up on cab for now and settled for zip.

Per discussion with vlad/nthomas/myself, we've also given up on trying to produce updates for WinCE until dep bug#458950 is landed in mozilla-central.
To be clear -- we should if at all possible still produce mar files for updates, even if the infrastructure is not in place yet to actually perform an update on the target platform.
(In reply to comment #20)
> To be clear -- we should if at all possible still produce mar files for
> updates, even if the infrastructure is not in place yet to actually perform an
> update on the target platform.

Totally agreed. 

We continue to work on producing the mar files along with the zip. While we can confirm if the zips are done right by running the builds, we cannot confirm if those mar files are valid/usable until the updater patch lands, so we dont feel comfortable saying the mar files are "done".
Attached patch patch for mozconfig (obsolete) — Splinter Review
Here are some tweaks to the mozconfig for this.

Also, which tree are these builds showing up on?  I couldn't find them on any of the staging/etc. trees.
Thanks for the mozconfig patch, I'll incorporate that into what I'm working on. 

Currently I've been working on a staging machine, so the steps to move forward are
* finish up testing to make sure we can create zip reliably in depend and nightly builds, and also mar for the latter
* test and deploy OPSI package for installing SDKs on pool of windows slaves
* check that nightly updates are working (partials generated, and can be applied)

I'm planning to work on that this week.
Component: Release Engineering: Future → Release Engineering
Great, thanks!  Let me know if there's anywhere that I can help.

One thing -- in the mozconfig, I think I got the right dir for the nvidia sdk, but can you just verify that the platformlibs dir is in the right place, as specified in the config file?
That's the right path for where I installed the SDK.
Component: Release Engineering → Release Engineering: Future
Component: Release Engineering: Future → Release Engineering
Depends on: 508177
Can we get an ETA update here? Looks like the review in bug 503469 (though we don't really need MAR support as discussed offline) is finished, and we've started hitting cases where people are checking in changes to mozilla-central which are breaking WindowsCE builds, and the cost of having to bisect in order to determine where that's failing is pretty expensive. :/

Nick: I think that, going forward, if things get at all clogged up for more than a day it would be great if someone could ping me or Vlad to make sure that we're all on the same page.
Whiteboard: [ETA?]
The status here is
* the packaging fixes in bug 503469 are now landed
* next step is to deploy the two new SDKs in bug 508312. Firstly to finish up Ben's work on having auto-install packages, deploy them to staging and make sure other builds don't burn, then deploy to the production builders. This is likely 2+ days out
* in parallel, rework the buildbot-configs for the new config style in bug 508312

Vlad, do you want to track mozilla-central or mozilla-1.9.2 ?
And which tree would you like the WinCE build to report to ?
Firefox tree, please, tracking m-c for now.. we'll need one for 1.9.2 as well once that branch gets going.
(Thanks for the update, Nick!)
Status update:
* dep and nightly builds are running in development environment, zip and mar files are produced, and partial updates also get created (see MozillaTest tree and search for WINCE)
* need to modify config for above and add 1.9.2 branch support, then get review
* followup patch added to bug 503469 so that parallel make works
* need to check main staging environment for bustage from SDKs installs, and if no problems deploy to main pool of slaves
Whiteboard: [ETA?] → [ETA - Monday, 20090817]
Attached patch buildbot custom tweaks (obsolete) — Splinter Review
Minor tweaks to support a "wince" platform where we don't build xulrunner.
Attachment #387823 - Attachment is obsolete: true
Attachment #387826 - Attachment is obsolete: true
Attachment #390852 - Attachment is obsolete: true
Attachment #394466 - Flags: review?(bhearsum)
Add WinCE builds for m-c and m-1.9.2, mozconfigs are identical on both branches.
Attachment #394467 - Flags: review?(bhearsum)
Minor tweaks to support a "wince" platform where we don't build xulrunner. (helps if you attach the right patch)
Attachment #394466 - Attachment is obsolete: true
Attachment #394468 - Flags: review?(bhearsum)
Attachment #394466 - Flags: review?(bhearsum)
Comment on attachment 394468 [details] [diff] [review]
buildbotcustom tweaks (really!)

It would be nicer not to have this exception logic done here, but that's a much larger problem. This is totally fine.
Attachment #394468 - Flags: review?(bhearsum) → review+
Comment on attachment 394467 [details] [diff] [review]
buildbot-configs 

>+import buildbotcustom.env
>+reload(buildbotcustom.env)
>+from buildbotcustom.env import MozillaEnvironments
>+

This is the second place this gets reloaded but I *think* it's ok since it's not a class. Eg, there won't be any old references to worry about. If you're super duper paranoid you can remove the reload in mobile_config.py.

>diff -r 6d589d507e8d mozilla2-staging/wince/mozilla-1.9.2/nightly/mozconfig
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/mozilla2-staging/wince/mozilla-1.9.2/nightly/mozconfig	Fri Aug 14 18:08:00 2009 +1200

Big mozconfig, wow.



Vlad noted that the splashscreen is missing from the build, I dunno if that's your issue or his though.
Attachment #394467 - Flags: review?(bhearsum) → review+
Whiteboard: [ETA - Monday, 20090817] → [ETA - first half of the week beginning 20090817]
Ben, one thing I'm not sure about is the changes the Tegra SDK makes to the environment:
* globally adds NVAPSDK = d:\sdks\tegra042   (harmless)
* appends to the PATH
    %NVAPSDK%\tools;
    %NVAPSDK%\platformlibs\bin\winxp\x86\release;
    %NVAPSDK%\3rdparty\bin\winxp\x86\release
It does this by modifying the registry.

The "winmo-arm" environment dict used for WinCE and WinMo builds overrides PATH and the WINCE builds don't get the changes in the environment. Apparently that's not fatal as the test WinCE builds have been compiling fine, but someone may want to use stuff on those paths in the future so this patch syncs up the VC9 environment.

Other (non-unittest) builds pick up the global changes to the MSYS environment, and I'm wondering if we need to care about that. The files in those three directories look fairly safe, in the sense that there doesn't seem to be any naming overlap with utils we use already, and anyway there's only d:\mozilla-build\moztools\bin following them in PATH. Do you think we're OK to leave it as is in the environment ?
Attachment #394770 - Flags: review?(bhearsum)
We're ok as-is; we should never depend on the tools there, we're really just installing that SDK to get headers and import libs for GLES2 stuff, which are actually cross-platform.
(In reply to comment #37)
 Do you think we're OK to
> leave it as is in the environment ?


Given Vlad's comment and my own investigation, yeah, I think it's fine. If we're super paranoid we could get the OPSI package to remove them - I'll leave that up to you though.
Attachment #394770 - Flags: review?(bhearsum) → review+
Comment on attachment 394468 [details] [diff] [review]
buildbotcustom tweaks (really!)

committed changeset 384:b8d6623fc2c0

(In reply to comment #35)
> It would be nicer not to have this exception logic done here, but that's a much
> larger problem. This is totally fine.

Would you prefer we defined something like BRANCH_LEVEL_VARS['xulrunner_platforms'] defined in config.py ?
Attachment #394468 - Flags: checked-in+
Comment on attachment 394770 [details] [diff] [review]
Update VC9 environment with Tegra paths

committed changeset 385:9da322657f3c
Attachment #394770 - Flags: checked-in+
Comment on attachment 394467 [details] [diff] [review]
buildbot-configs 

committed changeset 1432:8a50eacda69f

with the make -j4 re-enabled. I'm going to land attachment 394251 [details] [diff] [review] as the trigger for the first WinCE builds so they doesn't go boom.

(In reply to comment #36)
> This is the second place this gets reloaded but I *think* it's ok since it's
> not a class. Eg, there won't be any old references to worry about. If you're
> super duper paranoid you can remove the reload in mobile_config.py.

I seem to not be paranoid today (huzzah!) so I left this as is. Similarly left the environment with the unneeded the PATH changes.

> Vlad noted that the splashscreen is missing from the build, I dunno if that's
> your issue or his though.

I'll take a look into this. Probably just a something that needs to be added to packages-static.
Attachment #394467 - Flags: checked-in+
Status:
* dep. builds are green on m-c and m-1.9.2, they get published to 
 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-1.9.2-wince/
  and similar for m-c.
* they appear on Firefox3.6 and Firefox tinderbox waterfalls, and you get two B's for Windows on tinderboxpushlog
* nightlies will appear in the firefox/nightly/latest-mozilla-1.9.2 and latest-mozilla-central directories. They get triggered at 3am Pacific like other Firefox nightlies. hg.m.o just blew up under the nightly load, so they might not be out today
* updates should be available after two nightlies have been built
* splashscreen fix attached to bug 503469
Summary: WinCE 5.0 builds: Firefox on m-c → WinCE 5.0 builds: Firefox on m-c & m-1.9.2
Attachment #395197 - Flags: review?(anodelman)
Just tried the 1.9.2 build, seems to work great!  Only thing I noticed was the splash screen image isn't getting packaged, but that's something we can fix directly in m-c (I think it's just left out of the list or something; not a huge deal).
Great to hear Vlad. The splashscreen issue should be fixed in the next dep build and in tomorrow's nightlies - it landed from bug 503469 just now.

Once the nagios monitoring is landed I should be able to close this out.
Pushed 
  http://hg.mozilla.org/build/buildbot-configs/rev/f9315d9ac7c3
to back make off to -j2, since we've been getting alot of these errors:
  ARMV4I : fatal error LNK1104: cannot open file 'xpcom.lib'
Attachment #395197 - Flags: review?(anodelman) → review+
Attachment #395197 - Flags: checked-in+
I've removed parallel make from the mozconfig
 http://hg.mozilla.org/build/buildbot-configs/rev/a2fbf9c38253
because we continued to hit the xpcom.lib error occasionally.
Whiteboard: [ETA - first half of the week beginning 20090817] → [builds running, need cabs?]
No longer blocks: 511662
Depends on: 511662
Lets close this out now we have CABs & try server (bug 514332), and file anything else as a followup.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [builds running, need cabs?]
Verified fix from bug 511662
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.