779902 - Enable nightly metro builds on elm

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Comment 1

[Armen [:armenzg]]

Passing it to coop in my absence.

Comment 2

[Jim Mathies [:jimm]]

We'll need to do update testing with the new metro interface, so having this fairly soon would be good. Note though we are still blocked on bug 755724 before we can integrate the install. Progress there is being made but isn't complete yet. For this bug end of aug. is a good target.

Comment 3

[Armen [:armenzg]]

I have adjusted the dependency to bug 755724 appropriately.

Please let us know when we're ready for this.

Comment 4

[Jim Mathies [:jimm]]

Can we just turn them on now?

Comment 5

[Armen [:armenzg]]

Mine again.

Comment 6

[Armen [:armenzg]]

I think we might have packaging collision with:
firefox-17.0a1.en-US.win32.zip
firefox-17.0a1.en-US.win32.installer.exe

I think filing a bug in Build:Config would do.

Comment 7

[Armen [:armenzg]]

Hi all,
I'm trying to enable nightly builds for metro on elm.

There are a couple of things I need some guidance with so we won't shoot our feet:

* update platform. What should it be? jimm says that we could reuse the win32 one since eventually they will be the same builds. I would prefer to have a separate one just in case down the road it is not the same story.
    
* packaging. It seems that packaging is the same for both the win32 and the metro builds. For builds on check-in this is not a problem but for nightly builds they actually collide when uploading to ftp [1][2]

My proposal would be to disable win32 *nightly* builds for elm and enable the metro *nightly* builds.
Packaging would not be an issue but, if we reuse the update platform then win32 users on elm (if any) would be updated to the metro build regardless if they run on Windows 8 or not (I think & could be easily wrong).

Suggestions?

[1] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-elm/
[2] http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/elm-win32-metro/1346288964/

Comment 8

[Robert Strong (they/them - no direct email)]

(In reply to Armen Zambrano G. [:armenzg] from comment #7)
> Hi all,
> I'm trying to enable nightly builds for metro on elm.
> 
> There are a couple of things I need some guidance with so we won't shoot our
> feet:
> 
> * update platform. What should it be? jimm says that we could reuse the
> win32 one since eventually they will be the same builds. I would prefer to
> have a separate one just in case down the road it is not the same story.
I'm ok with that as a just in case.

> * packaging. It seems that packaging is the same for both the win32 and the
> metro builds. For builds on check-in this is not a problem but for nightly
> builds they actually collide when uploading to ftp [1][2]
> 
> My proposal would be to disable win32 *nightly* builds for elm and enable
> the metro *nightly* builds.
> Packaging would not be an issue but, if we reuse the update platform then
> win32 users on elm (if any) would be updated to the metro build regardless
> if they run on Windows 8 or not (I think & could be easily wrong).
That should be fine though I'd like jimm's input on this.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano G. [:armenzg] from comment #7)
> if we reuse the update platform then
> win32 users on elm (if any) would be updated to the metro build regardless
> if they run on Windows 8 or not (I think & could be easily wrong).

Not quite. The update platform is basically our signifier for a particular target platform. For example, WINNT_x86-msvc is the update platform in all 32-bit Windows builds. It is independent of the OS that build runs on.

In any case, I would assert that a Metro-only build is different enough from a Desktop-only or Metro+Desktop that it warrants a different update platform. This is nearly identical to how we treat Mac builds. (Darwin_x86_64-gcc3 for a single arch, Darwin_x86_64-gcc3-u-i386-x86_64 for a universal).

Comment 10

[Robert Strong (they/them - no direct email)]

The main difference is that with Mac's Universal bundles the architecture can and sometimes is removed by the user which breaks updates so we went with more details. In the case of metro the plan is they will have the same files. If all goes as plan it will be the same as supporting a new platform as we do today.

Comment 11

[Jim Mathies [:jimm]]

One note I'd like to add, at some point the experimental win8 builders we have going on elm will go away. At which point we will abandon those builds as far as updates go. This is fine, we'll mention this in any public statement we make about the upcoming "preview" release. We don't want to be responsible for migrating these users to whatever build we finally end up with.

Also, I'd like to avoid (if we can) losing the current test coverage we have on elm using the normal win32 builds. Those test have been useful to us in catching bad landings that tweak general platform code.

However if ending normal builds/test runs on elm is unavoidable to get nightlies going off the win8 experimental builders, we can live with it.

Comment 12

[[e-mail] (:bhearsum)]

Jimm, Armen, and I just chatted about this in #developers. As best we can tell, there's an elegant solution to this. Now that these metro builds are shaping up to be a replacement for the plain "win32" builds, we should change the win32 mozconfigs to build them, rather than trying to get updates working for "win32-metro". To do this, we need to:
1) Limit the "win32" (and maybe "win32-debug") platform to the subset of machines that can currently build metro builds. (armen)
2) Replace https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/win32/nightly with https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/win32-metro/nightly (jimm)
3) Get rid of the "win32-metro" platform entirely, since it will be 100% the same as "win32". (armen)
4) Turn off tests on XP for "win32", because they don't work yet (armen)

Before this merges to m-c, we also need to:
* Make sure the entire build pool can build metro builds
* Turn XP tests back on on elm, to make sure things work there

And if we can't ship a single binary that works with XP and up, we'll need to revisit having a win32-metro or equivalent platform, including package-name.mk updates, update platform changes, etc.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Jimm, Armen, and I just chatted about this in #developers. As best we can
> tell, there's an elegant solution to this. Now that these metro builds are
> shaping up to be a replacement for the plain "win32" builds, we should
> change the win32 mozconfigs to build them, rather than trying to get updates
> working for "win32-metro". To do this, we need to:
> 1) Limit the "win32" (and maybe "win32-debug") platform to the subset of
> machines that can currently build metro builds. (armen)
> 2) Replace
> https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/
> win32/nightly with
> https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/
> win32-metro/nightly (jimm)
> 3) Get rid of the "win32-metro" platform entirely, since it will be 100% the
> same as "win32". (armen)
> 4) Turn off tests on XP for "win32", because they don't work yet (armen)
> 
> Before this merges to m-c, we also need to:
> * Make sure the entire build pool can build metro builds
> * Turn XP tests back on on elm, to make sure things work there
> 
> And if we can't ship a single binary that works with XP and up, we'll need
> to revisit having a win32-metro or equivalent platform, including
> package-name.mk updates, update platform changes, etc.

looping in asa and akeybl, lsblakk because of the WinXP support question.

Comment 14

[Asa Dotzler [:asa]]

(In reply to John O'Duinn [:joduinn] from comment #13) 
> looping in asa and akeybl, lsblakk because of the WinXP support question.

If we're just talking about Elm builds we can make for a Metro Preview ASAP, it is not important that those builds work at all on XP or Windows 7.  Am I missing the real question here?

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Asa Dotzler [:asa] from comment #14)
> (In reply to John O'Duinn [:joduinn] from comment #13) 
> > looping in asa and akeybl, lsblakk because of the WinXP support question.
> 
> If we're just talking about Elm builds we can make for a Metro Preview ASAP,
> it is not important that those builds work at all on XP or Windows 7.  Am I
> missing the real question here?

I don't think this bug requires further input right now. This bug is about enabling nightly updates for the metro builds on Elm, which we have a plan to do. There's open questions about the implementation of it based on whether or not we ship a single binary for all versions of Windows or we ship multiple binaries, but that doesn't affect this bug.

Comment 16

[Jim Mathies [:jimm]]

Armen, just checking in, how goes the switch over?

Comment 17

[Armen [:armenzg]]

This is my top priority this week. Last week I was on duty and had no time to look at it.

Comment 18

[Armen [:armenzg]]

Attachment: merge win32-metro mozconfig into win32

jimm, would this be the correct patch?

Comment 19

[Armen [:armenzg]]

I'm also hitting this our known issue after installing VS2012.

Would the mozconfig change make the difference?
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1347390592.1347390929.1462.gz&fulltext=1

make[7]: Entering directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32/vmwarerecordinghelper'
e:/builds/moz2_slave/elm-w32/build/obj-firefox/_virtualenv/Scripts/python.exe -O /e/builds/moz2_slave/elm-w32/build/build/cl.py cl -Fovmwarerecordinghelper.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrappers    -I/e/builds/moz2_slave/elm-w32/build/build/win32/vmwarerecordinghelper -I. -I../../../dist/include  -Ie:/builds/moz2_slave/elm-w32/build/obj-firefox/dist/include/nspr -Ie:/builds/moz2_slave/elm-w32/build/obj-firefox/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy- -MT            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT /e/builds/moz2_slave/elm-w32/build/build/win32/vmwarerecordinghelper/vmwarerecordinghelper.cpp
vmwarerecordinghelper.cpp

/bin/sh /e/builds/moz2_slave/elm-w32/build/build/msys-perl-wrapper /e/builds/moz2_slave/elm-w32/build/config/version_win.pl -QUIET 1 -DEPTH ../../.. -TOPSRCDIR /e/builds/moz2_slave/elm-w32/build -OBJDIR . -SRCDIR /e/builds/moz2_slave/elm-w32/build/build/win32/vmwarerecordinghelper -DISPNAME Nightly -APPVERSION 18.0a1 -OFFICIAL 1
Creating Resource file: module.res
rc.exe  -r   -I/e/builds/moz2_slave/elm-w32/build/build/win32/vmwarerecordinghelper -I. -I../../../dist/include  -Ie:/builds/moz2_slave/elm-w32/build/obj-firefox/dist/include/nspr -Ie:/builds/moz2_slave/elm-w32/build/obj-firefox/dist/include/nss     -Fomodule.res /e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32/vmwarerecordinghelper/module.rc
Microsoft (R) Windows (R) Resource Compiler Version 6.1.7600.16385

Copyright (C) Microsoft Corporation.  All rights reserved.


e:/builds/moz2_slave/elm-w32/build/obj-firefox/_virtualenv/Scripts/python.exe /e/builds/moz2_slave/elm-w32/build/config/pythonpath.py -I../../../config /e/builds/moz2_slave/elm-w32/build/config/expandlibs_exec.py --depend .deps/vmwarerecordinghelper.dll.pp --target vmwarerecordinghelper.dll --uselist -- link -NOLOGO -DLL -OUT:vmwarerecordinghelper.dll -PDB:vmwarerecordinghelper.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86  vmwarerecordinghelper.obj   ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV -DEBUG -OPT:REF -DEF:e:/builds/moz2_slave/elm-w32/build/build/win32/vmwarerecordinghelper/vmwarerecordinghelper.def       kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib   
   Creating library vmwarerecordinghelper.lib and object vmwarerecordinghelper.exp
NEXT ERROR LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
make[7]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32/vmwarerecordinghelper'
make[6]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32'
NEXT ERROR make[7]: *** [vmwarerecordinghelper.dll] Error 99
make[6]: *** [libs] Error 2
make[5]: *** [libs] Error 2
make[5]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build'
make[4]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
NEXT ERROR make[4]: *** [libs_tier_base] Error 2
make[3]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
NEXT ERROR make[3]: *** [tier_base] Error 2
make[2]: *** [default] Error 2
make[2]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
make[1]: Leaving directory `/e/builds/moz2_slave/elm-w32/build'
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
program finished with exit code 2

Comment 20

[Armen [:armenzg]]

This seems to be the same issue as in https://bugzilla.mozilla.org/show_bug.cgi?id=772989#c5

I will look into it.

Comment 21

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #20)
> This seems to be the same issue as in
> https://bugzilla.mozilla.org/show_bug.cgi?id=772989#c5
> 
> I will look into it.

Yes definitely. I think you're on the right track in bug 772989. For our metro builders, installing the 7.1 sdk seems fine if it fixes the issue. It looks like we are going to require that any way through bug 774910.

Comment 22

[Armen [:armenzg]]

Attachment: win32 builds to be done by subset of elm slaves

This patch cannot go in until bug 772989 is resolved.

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 660221 [details] [diff] [review]
win32 builds to be done by subset of elm slaves

Review of attachment 660221 [details] [diff] [review]:
-----------------------------------------------------------------

We want to rip out the 'win32-metro' platform at the same time as this, don't we?

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 660221 [details] [diff] [review]
win32 builds to be done by subset of elm slaves

Review of attachment 660221 [details] [diff] [review]:
-----------------------------------------------------------------

17:35 < armenzg_afk> bhearsum: the patch to remove metro completely will come in a following patch
17:35 < bhearsum> ok
17:35 < armenzg_afk> not yet though
17:35 < bhearsum> what's the reason not to do it at the same time, out of curiosity?
17:36 < bhearsum> with just this patch, we'll end up with two builds doing the same thing
17:36 < armenzg_afk> bhearsum: for now I want to see it running side by side
17:36 < bhearsum> ok, so it will come a day or two later, after things look ok?
17:36 < armenzg_afk> I can try remove it all at once but once I have seen it run properly on staging
17:36 < armenzg_afk> bhearsum: correct
17:36 < bhearsum> okay, that's fine with me

Comment 25

[Brian R. Bondy [:bbondy]]

One reason why it would be a good idea to have side by side nightly builds (w/ metro + vs2012) and (w/o MOZ_METRO + vs2010) is because we'll want to land on m-c soon and have nightly builds, but vs2012 builds can't target winxp yet.  MS says it's coming soon but will not give a date.

Comment 26

[[e-mail] (:bhearsum)]

(In reply to Brian R. Bondy [:bbondy] from comment #25)
> One reason why it would be a good idea to have side by side nightly builds
> (w/ metro + vs2012) and (w/o MOZ_METRO + vs2010) is because we'll want to
> land on m-c soon and have nightly builds, but vs2012 builds can't target
> winxp yet.  MS says it's coming soon but will not give a date.

When we talked about this a couple of weeks ago, Jimm wasn't sure whether or not we'd be able to target everyone with one build when we landed on mozilla-central. If we're reasonably sure we won't be able to, I fully agree.

Comment 27

[Brian R. Bondy [:bbondy]]

It looks like we'll probably land on m-c before MS adds XP support.

Comment 28

[Jim Mathies [:jimm]]

On Elm we need metro enabled nightlies w/crash reporting - so for that repo, whatever setup changes represent the shortest distance between what we have now and what we need is what we want to do.

Comment 29

[[e-mail] (:bhearsum)]

OK, so if we're going to do them in parallel, we need build system support for different package names at the very least. Otherwise they'll overwrite each other when they upload. I'd recommend getting the update platform set differently, too. The package name logic is all here. I think you just need to add some logic around here to make MOZ_PKG_PLATFORM different for metro builds: https://github.com/mozilla/mozilla-central/blob/master/toolkit/mozapps/installer/package-name.mk#L14

I'm not sure where the update platform is set, though.

Comment 30

[Jim Mathies [:jimm]]

(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Jimm, Armen, and I just chatted about this in #developers. As best we can
> tell, there's an elegant solution to this. Now that these metro builds are
> shaping up to be a replacement for the plain "win32" builds, we should
> change the win32 mozconfigs to build them, rather than trying to get updates
> working for "win32-metro". To do this, we need to:
> 1) Limit the "win32" (and maybe "win32-debug") platform to the subset of
> machines that can currently build metro builds. (armen)
> 2) Replace
> https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/
> win32/nightly with
> https://hg.mozilla.org/projects/elm/file/default/browser/config/mozconfigs/
> win32-metro/nightly (jimm)
> 3) Get rid of the "win32-metro" platform entirely, since it will be 100% the
> same as "win32". (armen)
> 4) Turn off tests on XP for "win32", because they don't work yet (armen)
> 
> Before this merges to m-c, we also need to:
> * Make sure the entire build pool can build metro builds
> * Turn XP tests back on on elm, to make sure things work there
> 
> And if we can't ship a single binary that works with XP and up, we'll need
> to revisit having a win32-metro or equivalent platform, including
> package-name.mk updates, update platform changes, etc.

Just to confirm, we are back on this plan.

Comment 31

[Armen [:armenzg]]

Attachment: win32 builds to be done by subset of elm slaves

I noticed that we were using TRY_SLAVES rather than SLAVES.

I will have results from staging by Monday.

Comment 32

[Armen [:armenzg]]

I'm testing switching from :
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/rev/475a2258d45c
(mainly:)
-  . $topsrcdir/build/win32/mozconfig.vs2010
+  . $topsrcdir/build/win32/mozconfig.vs2011

We should the results of it by slave w64-ix-slave40 under the builder "WINNT 5.2 elm build":
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

Comment 33

[Armen [:armenzg]]

jimm, I landed that patch [1] to test but I wonder if this patch would be better (much closer to m-c):
diff --git a/browser/config/mozconfigs/win32/nightly b/browser/config/mozconfigs/win32/nightly
--- a/browser/config/mozconfigs/win32/nightly
+++ b/browser/config/mozconfigs/win32/nightly
@@ -9,4 +9,5 @@ ac_add_options --enable-jemalloc
 ac_add_options --enable-signmar
 ac_add_options --enable-profiling
+ac_add_options --enable-metro
 
 # Nightlies only since this has a cost in performance
@@ -25,7 +26,7 @@ fi
 
 if test "$PROCESSOR_ARCHITECTURE" = "AMD64" -o "$PROCESSOR_ARCHITEW6432" = "AMD64"; then
-  . $topsrcdir/build/win32/mozconfig.vs2010-win64
+  . $topsrcdir/build/win32/mozconfig.vs2011-win64
 else
-  . $topsrcdir/build/win32/mozconfig.vs2010
+  . $topsrcdir/build/win32/mozconfig.vs2011
 fi

[1]
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/rev/475a2258d45c

Comment 34

[Armen [:armenzg]]

Hi jimm, I've found another issue.

Landing those mozconfig changes produce some builds that don't seem to be testable (I think).

Here's one of the logs:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1348062817.1348062925.1511.gz&fulltext=1

######################################
'python' 'c:/talos-slave/test/tools/buildfarm/utils/printbuildrev.py' 'firefox'
...
Traceback (most recent call last):
  File "c:/talos-slave/test/tools/buildfarm/utils/printbuildrev.py", line 18, in <module>
    buildid = appini.get('App', 'BuildID')
  File "c:\mozilla-build\python25\Lib\ConfigParser.py", line 511, in get
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'App'
...
'python' 'reftest/runreftest.py' '--appname=firefox/firefox.exe' '--utility-path=bin' '--extra-profile-file=bin/plugins' '--symbols-path=http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/elm-win32/1348060882/firefox-18.0a1.en-US.win32.crashreporter-symbols-full.crashreporter-symbols.zip' 'reftest/tests/layout/reftests/reftest.list'
....
Error: Path C:\talos-slave\test\build\firefox\firefox.exe doesn't exist.
Are you executing $objdir/_tests/reftest/runreftest.py?
program finished with exit code 1
elapsedTime=0.328000
TinderboxPrint: reftest<br/><em class="testfail">T-FAIL</em>
Unknown Error: command finished with exit code: 1

Comment 35

[Jim Mathies [:jimm]]

> Error: Path C:\talos-slave\test\build\firefox\firefox.exe doesn't exist.
> Are you executing $objdir/_tests/reftest/runreftest.py?
> program finished with exit code 1
> elapsedTime=0.328000
> TinderboxPrint: reftest<br/><em class="testfail">T-FAIL</em>
> Unknown Error: command finished with exit code: 1


Looks like fallout from the work in bug 755724. If reftests fail, so be it, we can patch this up after this is live on elm.

Comment 36

[Mike Hommey [:glandium]]

Nowhere in the log it downloads firefox-18.0a1.en-US.win32.zip or firefox-18.0a1.en-US.win32.installer.exe

Comment 37

[Jim Mathies [:jimm]]

(In reply to Mike Hommey [:glandium] from comment #36)
> Nowhere in the log it downloads firefox-18.0a1.en-US.win32.zip or
> firefox-18.0a1.en-US.win32.installer.exe

Ah nice catch. It should have done that before downloading firefox-18.0a1.en-US.win32.crashreporter-symbols-full.zip. That step is missing in this log.

Comparing to:

https://tbpl.mozilla.org/php/getParsedLog.php?id=15330793&tree=Elm&full=1

Comment 38

[Armen [:armenzg]]

Thanks glandium!

It seems there is something funky on the sendchanges that buildbot generates.
It should include the URL to the binary.
I will let you know once I figure it out.

URLs on the sendchanges are positional and that is why the symbols got downloaded instead of the binaries.

'python' 'e:/builds/moz2_slave/elm-w32/tools/buildfarm/utils/retry.py' '-s' '5' '-t' '1800' '-r' '1' '--stdout-regexp' 'change sent successfully' 'buildbot' 'sendchange' '--master' 'dev-master01.build.scl1.mozilla.com:9041' '--username' 'sendchange' '--branch' 'elm-win32-talos' '--revision' '475a2258d45c' '--property' 'buildid:20120919062122' '--property' 'pgo_build:False' '--property' 'builduid:525882d7515a4ef7a324e5964bf3754b' 'http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/elm-win32/1348060882/firefox-18.0a1.en-US.win32.crashreporter-symbols-full.zip'

Comment 39

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #33)
> jimm, I landed that patch [1] to test but I wonder if this patch would be
> better (much closer to m-c):
jimm said to worry about this later.

I think I will be able to fix the wrong packaging URL by adding this:
         elif m.endswith("crashreporter-symbols.zip"):
             retval['symbolsUrl'] = m
+        elif m.endswith("crashreporter-symbols-full.zip"):
+            retval['symbolsUrl'] = m

Once I verify I will ask for a review.

Comment 40

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #39)
> (In reply to Armen Zambrano G. [:armenzg] from comment #33)
> > jimm, I landed that patch [1] to test but I wonder if this patch would be
> > better (much closer to m-c):
> jimm said to worry about this later.
> 
> I think I will be able to fix the wrong packaging URL by adding this:
>          elif m.endswith("crashreporter-symbols.zip"):
>              retval['symbolsUrl'] = m
> +        elif m.endswith("crashreporter-symbols-full.zip"):
> +            retval['symbolsUrl'] = m
> 
> Once I verify I will ask for a review.

In case anyone wonders, we hit this in production for metro builds *but* we don't have unit tests triggers where it would have been noticeable.

FTR jimm said this:
"we can revert that change back to the normal package assuming nightlies are close. we only did that because we didn't have symbols on the symbol server to debug with."

Comment 41

[Jim Mathies [:jimm]]

I've been trying to track down the original symbols patch that landed, but I'm at a loss. It's somewhere down in toolkit and is related to packager. If that helps. :)

Comment 42

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #41)
> I've been trying to track down the original symbols patch that landed, but
> I'm at a loss. It's somewhere down in toolkit and is related to packager. If
> that helps. :)

http://mxr.mozilla.org/projects-central/source/elm/toolkit/mozapps/installer/packager.mk#1002

Comment 43

[Armen [:armenzg]]

Hi jimm,
I've found a new issue:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1348085685.1348092023.31166.gz&fulltext=1

It seems that it tries to go a PGO build for the nightly build but it fails to do so:
e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/_virtualenv/Scripts/python.exe /e/builds/moz2_slave/elm-w32-ntly/build/config/pythonpath.py -I../../config /e/builds/moz2_slave/elm-w32-ntly/build/config/expandlibs_exec.py --depend .deps/xul.dll.pp --target xul.dll --uselist -- link -NOLOGO -DLL -OUT:xul.dll -PDB:xul.pdb -SUBSYSTEM:WINDOWS -MACHINE:X86  dlldeps-xul.obj nsStaticXULComponents.obj nsDllMain.obj nsGFXDeps.obj dlldeps-zlib.obj nsUnicharUtils.obj nsBidiUtils.obj nsSpecialCasingData.obj nsUnicodeProperties.obj nsRDFResource.obj   ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV -DEBUG -OPT:REF -LTCG:PGINSTRUMENT    ../../toolkit/components/osfile/osfile_s.lib ../../toolkit/xre/xulapp_s.lib  ../../staticlib/components/necko.lib ../../staticlib/components/uconv.lib ../../staticlib/components/i18n.lib ../../staticlib/components/chardet.lib ../../staticlib/components/jar50.lib ../../staticlib/components/startupcache.lib ../../staticlib/components/pref.lib ../../staticlib/components/htmlpars.lib ../../staticlib/components/identity.lib ../../staticlib/components/imglib2.lib ../../staticlib/components/mediasniffer.lib ../../staticlib/components/gkgfx.lib ../../staticlib/components/gklayout.lib ../../staticlib/components/docshell.lib ../../staticlib/components/embedcomponents.lib ../../staticlib/components/webbrwsr.lib ../../staticlib/components/nsappshell.lib ../../staticlib/components/txmgr.lib ../../staticlib/components/commandlines.lib ../../staticlib/components/toolkitcomps.lib ../../staticlib/components/pipboot.lib ../../staticlib/components/pipnss.lib ../../staticlib/components/appcomps.lib ../../staticlib/components/jsreflect.lib ../../staticlib/components/composer.lib ../../staticlib/components/telemetry.lib ../../staticlib/components/jsinspector.lib ../../staticlib/components/jsdebugger.lib ../../staticlib/components/storagecomps.lib ../../staticlib/components/rdf.lib ../../staticlib/components/windowds.lib ../../staticlib/components/jsctypes.lib ../../staticlib/components/jsperf.lib ../../staticlib/components/gkplugin.lib ../../staticlib/components/windowsproxy.lib ../../staticlib/components/jsd.lib ../../staticlib/components/autoconfig.lib ../../staticlib/components/auth.lib ../../staticlib/components/cookie.lib ../../staticlib/components/permissions.lib ../../staticlib/components/universalchardet.lib ../../staticlib/components/places.lib ../../staticlib/components/tkautocomplete.lib ../../staticlib/components/satchel.lib ../../staticlib/components/pippki.lib ../../staticlib/components/imgicon.lib ../../staticlib/components/profiler.lib ../../staticlib/components/widget_windows.lib ../../staticlib/components/widget_winrt.lib ../../staticlib/components/accessibility.lib ../../staticlib/components/spellchecker.lib ../../staticlib/components/zipwriter.lib ../../staticlib/components/services-crypto.lib ../../staticlib/jsipc_s.lib ../../staticlib/domipc_s.lib ../../staticlib/domplugins_s.lib ../../staticlib/mozipc_s.lib ../../staticlib/mozipdlgen_s.lib ../../staticlib/ipcshell_s.lib ../../staticlib/gfxipc_s.lib ../../staticlib/hal_s.lib ../../staticlib/dombindings_s.lib ../../staticlib/xpcom_core.lib ../../staticlib/ucvutil_s.lib ../../staticlib/chromium_s.lib ../../staticlib/snappy_s.lib ../../staticlib/thebes.lib ../../staticlib/gl.lib ../../staticlib/ycbcr.lib   e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/mozjs.lib e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/crmf.lib         e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/smime3.lib         e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/ssl3.lib         e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/nss3.lib         e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/nssutil3.lib   ../../dist/lib/mozsqlite3.lib  ../../dist/lib/gkmedias.lib   e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/nspr4.lib e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/plc4.lib e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/dist/lib/plds4.lib  ../../dist/lib/mozalloc.lib -DELAYLOAD:psapi.dll -DELAYLOAD:dbghelp.dll -DELAYLOAD:rasapi32.dll -DELAYLOAD:rasdlg.dll -DELAYLOAD:comdlg32.dll -DELAYLOAD:winspool.drv -DELAYLOAD:secur32.dll -DELAYLOAD:wininet.dll -DELAYLOAD:uiautomationcore.dll  -DELAYLOAD:VCCORLIB110.DLL -DELAYLOAD:API-MS-WIN-CORE-WINRT-L1-1-0.DLL -DELAYLOAD:API-MS-WIN-CORE-WINRT-STRING-L1-1-0.DLL -DELAYLOAD:oleacc.dll -LIBPATH:../../dist/lib -NODEFAULTLIB:msvcrt -NODEFAULTLIB:msvcrtd -NODEFAULTLIB:msvcprt -NODEFAULTLIB:msvcprtd -DEFAULTLIB:mozcrt kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib shell32.lib ole32.lib version.lib winspool.lib comdlg32.lib imm32.lib msimg32.lib shlwapi.lib psapi.lib ws2_32.lib dbghelp.lib rasapi32.lib rasdlg.lib iphlpapi.lib uxtheme.lib setupapi.lib secur32.lib sensorsapi.lib portabledeviceguids.lib windowscodecs.lib wininet.lib oleacc.lib uiautomationcore.lib delayimp.lib  usp10.lib oleaut32.lib   
LINK : fatal error LNK1330: Profile Guided Optimization is not supported for Windows Store apps; remove PGINSTRUMENT from switch /LTCG and restart the link
make[6]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/toolkit/library'
make[5]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox'
make[4]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox'
make[3]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox'
make[2]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build'
make[1]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build'
make[6]: *** [xul.dll] Error 50
make[5]: *** [libs_tier_platform] Error 2
make[4]: *** [tier_platform] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [profiledbuild] Error 2
make: *** [build] Error 2
program finished with exit code 2
elapsedTime=6051.294000

Comment 44

[Jim Mathies [:jimm]]

looking...

Comment 45

[Jim Mathies [:jimm]]

This may well represent the final nail in the coffin for vs2012. There doesn't appear to be a way around this. We are still investigating.

In the mean time, how hard would it be to nix 'MOZ_PGO=1' from elm nightly builds?

'make' '-f' 'client.mk' 'build' 'MOZ_BUILD_DATE=20120919131458' 'MOZ_PGO=1'

Comment 46

[Armen [:armenzg]]

Attachment: [wip] disable pgo platforms

This is what this patch does when comparing the output of "python config.py":
< elm:  'pgo_platforms': ('linux', 'linux64', 'win32', 'win64'),
---
> elm:  'pgo_platforms': (),

I will run this through staging.

Comment 47

[[e-mail] (:bhearsum)]

Comment on attachment 663061 [details] [diff] [review]
[wip] disable pgo platforms

This will turn off PGO for 64-bit Windows too. Is that intentional?

Comment 48

[Armen [:armenzg]]

bhearsum, I will be disable win64.

jimm, here's what is left:

1) land attachment 661325 [details] [diff] [review] to lock slaves with VS2012 to win32 builds
2) land mozconfig changes (attachment 660200 [details] [diff] [review])
** the win32 build would become a metro enabled build
3) disable PGO (attachment 663061 [details] [diff] [review])
** so we can produce metro nightly builds
4) Remove win32-metro from our configs (not blocking though - it needs to be done though)
5) disable Windows XP tests as they don't work yet (not blocking though - it needs to be done though)

The problem I see is that pretty much all win7 and XP unit tests will start failing.
We can debug one by one on staging or we can do the switch on elm and let you figure each one out.
For instance, one of them is "Couldn't load XPCOM": http://cl.ly/Jal1

What would you like to do? Should we take care of 1, 2 & 3 tomorrow?
This way you could have your nightly builds and unit tests on Elm while I work on 4 & 5.

I can even loan you a win7 slave to figure out the test failures.

The tests show up in http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest
as "Rev3 WINNT {5,6}.1 elm opt <suite>".

NOTE: I have not even touched debug builds

Comment 49

[Jim Mathies [:jimm]]

Hmm, so I don't see why these tests are failing. From the reftest log data:

'unzip' '-o' 'firefox-18.0a1.en-US.win32.zip'
in dir c:\talos-slave\test\build

..
inflating: firefox/firefox.exe 
..
inflating: firefox/xpcom.dll  
..

So firefox.exe and the xpcom.dll are in the same directory. This is good, it's no different from our normal test runs on mc.

running the test:

args: ['c:\\talos-slave\\test\\build\\firefox\\firefox.exe', '-no-remote', '-profile', 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpwa5onv/', '-reftest', 'c:\\talos-slave\\test\\build\\reftest\\tests\\layout\\reftests\\reftest.list']

TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output

w/this screenshot: http://cl.ly/Jal1

That's the error dialog nsBrowserApp throws up when it can't find or load xpcom.dll.

So we should be able to track this down and fix it pretty easily. Either xpcom.dll can't be found by the browser stub, or the library is corrupt, or there's a bug in nsBrowserApp.

If you have access to one of these slaves, can you test something? What happens when you try to run c:\talos-slave\test\build\firefox.exe manually?

My feeling here is that something very basic in our setup isn't quite right. Once we sort that out these tests should run fine. If you want I can try to diagnose this if I can get access to one of these slaves.

Comment 50

[Mike Hommey [:glandium]]

(In reply to Jim Mathies [:jimm] from comment #49)
> So we should be able to track this down and fix it pretty easily. Either
> xpcom.dll can't be found by the browser stub, or the library is corrupt, or
> there's a bug in nsBrowserApp.

or one of the libraries xpcom.dll depends upon are missing, or dependentlibs.list contains garbage

Comment 51

[Armen [:armenzg]]

This is what I get:
http://cl.ly/Jabg

"The program can't start because api-ms-win-core-winrt-string-l1-1-0.dll is missing from your computer"

Comment 52

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #51)
> This is what I get:
> http://cl.ly/Jabg
> 
> "The program can't start because api-ms-win-core-winrt-string-l1-1-0.dll is
> missing from your computer"

I can fix this. The new packager is missing dummyvccorlib.dll. I'll land a fix on elm and post back with the cset.

Comment 53

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #52)
> (In reply to Armen Zambrano G. [:armenzg] from comment #51)
> > This is what I get:
> > http://cl.ly/Jabg
> > 
> > "The program can't start because api-ms-win-core-winrt-string-l1-1-0.dll is
> > missing from your computer"
> 
> I can fix this. The new packager is missing dummyvccorlib.dll. I'll land a
> fix on elm and post back with the cset.

https://hg.mozilla.org/projects/elm/rev/afd03b35fb62

Comment 54

[Armen [:armenzg]]

Attachment: handle crashreporter-full.zip naming

jimm you said that this was necessary until there was nightly builds for metro.

Should I adjust for it? or are you planning on reverting this naming change? (since we would get metro nightly builds soon)

If you want to keep it then I can pass this patch to one of us to review it.

Comment 55

[Armen [:armenzg]]

Attachment: use subset of VC2012 slaves for elm win32 builds + disable PGO for metro nightly builds (for win32-metro) on elm

Comment 56

[Jim Mathies [:jimm]]

Comment on attachment 663098 [details] [diff] [review]
handle crashreporter-full.zip naming

I think this will be useful in general, and I'd like to keep uploading dep build full symbols for a little while. We can disable this on elm when we're sure we don't need them anymore.

Comment 57

[Armen [:armenzg]]

Comment on attachment 663098 [details] [diff] [review]
handle crashreporter-full.zip naming

OK cool. Requesting now a releng review.

Comment 58

[[e-mail] (:bhearsum)]

Comment on attachment 663098 [details] [diff] [review]
handle crashreporter-full.zip naming

Review of attachment 663098 [details] [diff] [review]:
-----------------------------------------------------------------

You could also modify Makefile.in to include PDBs in the non-full symbol archive if you don't want to wait for a reconfig: https://mxr.mozilla.org/mozilla-central/source/Makefile.in#184

Comment 59

[Armen [:armenzg]]

Hi jimm,
I've got my first metro nightly builds; would you mind checking it out and see if we're good to go?
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-elm/

Comment 60

[Armen [:armenzg]]

I meant this:
http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip

Comment 61

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #60)
> I meant this:
> http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip

The revision is 475a2258d45c rather than:
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/rev/63bc2f1f96b4

I will post another build tomorrow morning or later tonight.

Comment 62

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #60)
> I meant this:
> http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip

Once I copy that missing dll I landed the manifest change for, it works fine.

Comment 63

[[e-mail] (:bhearsum)]

Comment on attachment 663103 [details] [diff] [review]
use subset of VC2012 slaves for elm win32 builds + disable PGO for metro nightly builds (for win32-metro) on elm

Review of attachment 663103 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me.

Comment 64

[Armen [:armenzg]]

jimm,
It seems that we have 2 application.ini on the mar files.
Is this expected?

I want to run this through you and figure out which one is the one we need:

We have a step that finds the application.ini:
> 'bash' '-c' 'find previous -maxdepth 4 -type f -name application.ini'
Which finds these:
> previous/application.ini
> previous/metro/application.ini

Unfortunately, in the next step things get messed up:
> 'python' 'build/config/printconfigsetting.py' 'build/obj-firefox/previous/application.ini\nprevious/metro/application.ini' 'App' 'BuildID'

I can't turn on the nightly builds unless we fix this issue as well.

##########################

On another note, it seems that win32 would be both the normal desktop builds plus the metro one but I see that the size is smaller:
firefox-18.0a1.en-US.win32.zip			20-Sep-2012 12:43 	22M
firefox-18.0a1.en-US.win32.installer.exe	20-Sep-2012 12:43 	17M

(mozilla-central)
firefox-18.0a1.en-US.win32.installer.exe	21-Sep-2012 05:55 	19M	 
firefox-18.0a1.en-US.win32.zip			21-Sep-2012 05:55 	25M

Comment 65

[Armen [:armenzg]]

Attachment: remove win32-metro

This partially removes win32-metro which reverts the changes from:
http://hg.mozilla.org/build/buildbot-configs/rev/bd409064c929

For now, we have to leave around the 'win64-metro' pool of slaves to create elm builds (VS2012 + metro).

Comment 66

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #64)
> jimm,
> It seems that we have 2 application.ini on the mar files.
> Is this expected?


Yep, two apps, two ini files. Long term I think the firefox one will move to /browser.


> We have a step that finds the application.ini:
> > 'bash' '-c' 'find previous -maxdepth 4 -type f -name application.ini'
> Which finds these:
> > previous/application.ini
> > previous/metro/application.ini
> Unfortunately, in the next step things get messed up:
> > 'python' 'build/config/printconfigsetting.py' 'build/obj-firefox/previous/application.ini\nprevious/metro/application.ini' 'App' 'BuildID'
> 
> I can't turn on the nightly builds unless we fix this issue as well.


I'll bet we could fix this like webapprt did, by renaming the application.ini file to something else. Let me look at it.


> On another note, it seems that win32 would be both the normal desktop builds
> plus the metro one but I see that the size is smaller:

firefox-18.0a1.en-US.win32.zip			20-Sep-2012 12:43 	22M
firefox-18.0a1.en-US.win32.installer.exe	20-Sep-2012 12:43 	17M

(mozilla-central)
firefox-18.0a1.en-US.win32.zip			21-Sep-2012 05:55 	25M
firefox-18.0a1.en-US.win32.installer.exe	21-Sep-2012 05:55 	19M	 

Hmm, not sure.

Comment 67

[Jim Mathies [:jimm]]

We should do a diff across the two zips, see what's different.

Maybe the new packager is optimizing things better. I ping'd glandium on irc to see if he has any ideas.

Comment 68

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #67)
> We should do a diff across the two zips, see what's different.
> 
> Maybe the new packager is optimizing things better. I ping'd glandium on irc
> to see if he has any ideas.

Most likely not the new packager, so we should investigate.

Comment 69

[Jim Mathies [:jimm]]

> > Unfortunately, in the next step things get messed up:
> > > 'python' 'build/config/printconfigsetting.py' 'build/obj-firefox/previous/application.ini\nprevious/metro/application.ini' 'App' 'BuildID'
> > 
> > I can't turn on the nightly builds unless we fix this issue as well.
> 
> I'll bet we could fix this like webapprt did, by renaming the
> application.ini file to something else. Let me look at it.
> 

https://hg.mozilla.org/projects/elm/rev/b2a65e1743e5

Comment 70

[Mike Hommey [:glandium]]

One possible (and known) source of difference between old and new packager is that the new packager (as it is on elm) doesn't handle startupcache population yet, so jsloader/ and jssubloader/ files are missing in the new omni.ja files.

Comment 71

[Armen [:armenzg]]

After speaking with bbondy, we have determined that:
* we should use application.ini to grab the buildid (as the normal win32 builds do, even though metroapp.ini -b2a65e1743e5 - also contains the same info)
* I have merged b2a65e1743e5 into users/armenzg_mozilla.com/elm
* I have triggered one more nightly build (on staging) and will trigger another one after a lapse
* we need someone to test that the mar got generated properly

I can probably test it on a win8 machine and win7 machine and see what happens.

After all that, unless I find more blockers, I could enable the metro nightly builds.

Comment 72

[Armen [:armenzg]]

I merge the latest changes from elm because I was getting a "make package" issue. It seems to be fixed by glandium:
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/graph

#######################################

On another note, I have noticed something interesting.

The elm builds that I'm generating on staging can run on Windows XP as well as on Windows 8.
The build is on dev-stage [1] but have moved it to my people account with the log:
http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip
http://people.mozilla.com/~armenzg/incoming/elm.build.log.txt

I have triggered clobber builds just to be sure. I'm very confused.

[1] http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/elm-win32/1348233924/

Comment 73

[Jim Mathies [:jimm]]

> http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip

This looks like a vs2010 build of Elm w/o enable-metro.

Comment 74

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #73)
> > http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip
> 
> This looks like a vs2010 build of Elm w/o enable-metro.

*sigh* I somehow commented out the line that was point all the staging builds to my user repo. I don't know when I did this. I think I did it this morning by mistake.

Re-triggering nightly builds and others.

Comment 75

[Armen [:armenzg]]

I have verified that the right mozconfig is being used:
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/file/default/browser/config/mozconfigs/win32/nightly
but I now get an error on "make package" (which I don't see happening on tbpl):
make[1]: Entering directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
make export
make[2]: Entering directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
make[2]: Nothing to be done for `export'.
make[2]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
make libs
make[2]: Entering directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
Packaging JavaScript Shell...
rm -f ../../dist/jsshell-win32.zip
zip -9j ../../dist/jsshell-win32.zip ../../dist/bin/js.exe ../../dist/bin/mozglue.dll  ../../dist/bin/nspr4.dll ../../dist/bin/msvcr110.dll
  adding: js.exe (172 bytes security) (deflated 59%)
  adding: mozglue.dll (172 bytes security) (deflated 41%)
  adding: nspr4.dll (172 bytes security) (deflated 48%)
  adding: msvcr110.dll (172 bytes security) (deflated 46%)
e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/_virtualenv/Scripts/python.exe /e/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py -DAB_CD=en-US -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/preferences -D_MSC_VER=1700 -DJAREXT= -DMOZ_ANGLE_RENDERER=1 -DMOZ_D3DX9_DLL=d3dx9_43.dll -DMOZ_D3DCOMPILER_DLL=D3DCompiler_43.dll -DMOZ_CHILD_PROCESS_NAME=plugin-container.exe -DMOZ_MSVC_REDIST=1700 -DMOZ_SHARED_MOZGLUE=1 -DMOZ_JSDEBUGGER -DNECKO_WIFI -DDLL_PREFIX= -DDLL_SUFFIX=.dll -DBIN_SUFFIX=.exe -DBINPATH=bin --format omni --removals /e/builds/moz2_slave/elm-w32-ntly/build/browser/installer/removed-files.in /e/builds/moz2_slave/elm-w32-ntly/build/browser/installer/package.manifest ../../dist/bin ../../dist/firefox
make[2]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
make[1]: Leaving directory `/e/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/browser/installer'
res/spacetrace.css was not registered
Missing file(s): ..\..\dist\bin\metro/application.ini
Traceback (most recent call last):
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 600, in <module>
    main()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 586, in main
    packager.finalize()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 547, in finalize
    Packager.finalize(self)
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 357, in finalize
    self.copier.finalize()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 180, in finalize
    raise Exception, 'Aborting due to errors'
Exception: Aborting due to errors
make[2]: *** [stage-package] Error 1
make[1]: *** [all] Error 2
make: *** [package] Error 2

Comment 76

[Jim Mathies [:jimm]]

Ugh, I think I can fix that on elm.

Comment 77

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #76)
> Ugh, I think I can fix that on elm.

https://hg.mozilla.org/projects/elm/rev/2ca465b67f15

Comment 78

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #77)
> (In reply to Jim Mathies [:jimm] from comment #76)
> > Ugh, I think I can fix that on elm.
> 
> https://hg.mozilla.org/projects/elm/rev/2ca465b67f15

Got it and triggered:
http://hg.mozilla.org/users/armenzg_mozilla.com/elm/rev/f9b9344504fd

Comment 79

[Armen [:armenzg]]

Here is the produced build:
http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip

I don't have any more issues that I would like to solve before we enable this on elm.
I will start landing unless there are any issues.

Comment 80

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #79)
> Here is the produced build:
> http://people.mozilla.com/~armenzg/incoming/firefox-18.0a1.en-US.win32.zip
> 
> I don't have any more issues that I would like to solve before we enable
> this on elm.
> I will start landing unless there are any issues.

That build seems fine based on some basic testing, so yeah lets do it.

Comment 81

[Brian R. Bondy [:bbondy]]

I noticed that if you run:
<unzippeddir>/firefox.exe and then update on windows, it will put the update inside <unzippeddir>/browser/firefox.exe.

I'm fairly confident this is just a toolkit/xre/nsUpdateDriver.cpp quick fix though, so please go ahead and land on elm and I'll fix that there in the context of another bug.

Comment 82

[Armen [:armenzg]]

Comment on attachment 663103 [details] [diff] [review]
use subset of VC2012 slaves for elm win32 builds + disable PGO for metro nightly builds (for win32-metro) on elm

http://hg.mozilla.org/build/buildbot-configs/rev/660543529632

Comment 83

[Armen [:armenzg]]

Comment on attachment 663386 [details] [diff] [review]
remove win32-metro

http://hg.mozilla.org/build/buildbot-configs/rev/26d0ebe251ea

Comment 84

[Armen [:armenzg]]

Comment on attachment 663098 [details] [diff] [review]
handle crashreporter-full.zip naming

http://hg.mozilla.org/build/buildbotcustom/rev/cd90d12a58fc

Comment 85

[Armen [:armenzg]]

Comment on attachment 660200 [details] [diff] [review]
merge win32-metro mozconfig into win32

Pushed to elm:
http://hg.mozilla.org/projects/elm/rev/c41946b26d4c

I have triggered a nightly build.

jimm, what are we going to be doing about the win32 debug mozconfig?
It is pointing to vs2010 right now.

NOTE: The XP slaves will be failing on tbpl from here on

Known remaining issues:
* Even though I have locked the "WINNT 5.2 elm build" builder to only 5 slaves. I can see w64-ix-slave84 taking a job which is not part of the subset of slaves. [1]

[1]
http://buildbot-master13.build.scl1.mozilla.com:8001/builders/WINNT%205.2%20elm%20build

Comment 86

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #85)
> Comment on attachment 660200 [details] [diff] [review]
> merge win32-metro mozconfig into win32
> 
> Pushed to elm:
> http://hg.mozilla.org/projects/elm/rev/c41946b26d4c
> 
> I have triggered a nightly build.
> 
> jimm, what are we going to be doing about the win32 debug mozconfig?
> It is pointing to vs2010 right now.

We can tweak those if we want to through mozconfig changes on elm correct? If that's the case lets let things sit as is and see how the release build / nightly come out.

Comment 87

[Jim Mathies [:jimm]]

The nightly seems to have died at:

========= Finished create aus previous upload dir failed (results: 2, elapsed: 17 secs) (at 2012-09-24 13:41:59.779752) =========

at the bottom of the latest nightly build log.

Comment 88

[Armen [:armenzg]]

For some odd reason, the w64 slaves that I had were missing the auspush key.
I have deployed that and re-triggered the nightly build.

Notice that some of the win32 builds are going to break until I figure out why are jobs being scheduled to the wrong slaves even though the patch I wrote it should work.

Comment 89

[Jim Mathies [:jimm]]

Looks like we have nightly builds showing up in the archive - 

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-elm/

I think we probably want to get those xp builds/tests stopped next. The test machines take a long time to bail so we're holding up slaves on each checkin.

Comment 90

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #88)
> Notice that some of the win32 builds are going to break until I figure out
> why are jobs being scheduled to the wrong slaves even though the patch I
> wrote it should work.

This does not seem to happen anymore. Yay! Perhaps it was just a matter that a scheduling object took a little longer to update.


I will work on a patch for the XP jobs. It should be not too much work.


Jimm, what else do you need help with? (in this bug; another bug) and how urgent is it?

Comment 91

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #90)
> (In reply to Armen Zambrano G. [:armenzg] from comment #88)
> > Notice that some of the win32 builds are going to break until I figure out
> > why are jobs being scheduled to the wrong slaves even though the patch I
> > wrote it should work.
> 
> This does not seem to happen anymore. Yay! Perhaps it was just a matter that
> a scheduling object took a little longer to update.

Yes, they seem to be clearing up pretty quick this morning.

> I will work on a patch for the XP jobs. It should be not too much work.

Great, thanks.

> Jimm, what else do you need help with? (in this bug; another bug) and how
> urgent is it?

If it isn't too much work clearing up bug 789335 for elm would be great, although it's not critical.

Comment 92

[Armen [:armenzg]]

Attachment: disable XP unit tests

jimm, do you want unit tests for XP on debug builds to be disabled as well?
I guess it depends on when you will be switching the debug builds to VS2012.

Comment 93

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #92)
> Created attachment 664514 [details] [diff] [review]
> disable XP unit tests
> 
> jimm, do you want unit tests for XP on debug builds to be disabled as well?
> I guess it depends on when you will be switching the debug builds to VS2012.

I'd like to keep those running for now so we can continue to track xp builds on elm. Lets just get rid of WinXP opt.

Comment 94

[Armen [:armenzg]]

Attachment: disable XP optimized unit tests on Elm

Comment 95

[Armen [:armenzg]]

Attachment: disable XP optimized unit tests on Elm

Comment 96

[[e-mail] (:bhearsum)]

Comment on attachment 664528 [details] [diff] [review]
disable XP optimized unit tests on Elm

Armen says an updated patch is coming.

Comment 97

[Armen [:armenzg]]

Attachment: (archive) this disables XP unit test fo 'xp'

I spoke with jimm and we won't need to disable the XP tests as they will become handy down the road.

This patch almost made the cut. It adds tegra_android as well for some odd reason.

Comment 98

[Armen [:armenzg]]

I think we're done in here. Let me know if I missed anything.