661910 - Windows XULRunner Aurora 6 SDK is missing

Status: VERIFIED FIXED

Description

[Alex Vincent [:WeirdAl]]

The company I work for requires a Windows SDK to compile a binary component for FF6.  Currently, the latest-mozilla-aurora directory for XULRunner only has Linux builds.  Please advise!

Comment 1

[[e-mail] (:bhearsum)]

A recent build failed in make package with:
updater.exe
xpcom.dll
xpcshell.exe
xpidl.exe
xul.dll
Linking XPT files...
xulrunner-stub.exe
xulrunner.exe
d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/cen-w32-xr/build/config/optimizejars.py --optimize /e/builds/moz2_slave/cen-w32-xr/build/obj-firefox/xulrunner/installer/../../jarlog/ ../../dist/bin/chrome ../../dist/xulrunner/chrome
Removing unpackaged files...
cd ../../dist/xulrunner; rm -rf xulrunner-config regchrome* regxpcom*  core bsdecho gtscc js js-config jscpucfg nsinstall viewer TestGtkEmbed codesighs* elf-dynstr-gc mangle* maptsv* mfc* mkdepend* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* ssltunnel* certutil* pk12util* winEmbed.exe chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM 
e:/builds/moz2_slave/cen-w32-xr/build/obj-firefox/config/nsinstall.exe -D ../../dist/
Compressing...
cd ../../dist && (cd xulrunner && rm -f omni.jar components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest && mv components.manifest components && zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  && true && d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/cen-w32-xr/build/config/optimizejars.py --optimize /e/builds/moz2_slave/cen-w32-xr/build/obj-firefox/xulrunner/installer/../../jarlog/ ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd xulrunner && d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/cen-w32-xr/build/config/createprecomplete.py) && zip -r9D xulrunner-7.0a1.en-US.win32.zip xulrunner
make[2]: Leaving directory `/e/builds/moz2_slave/cen-w32-xr/build/obj-firefox/xulrunner/installer'
make[1]: Leaving directory `/e/builds/moz2_slave/cen-w32-xr/build/obj-firefox/xulrunner/installer'
sed: can't read components/components.manifest: No such file or directory
make[2]: *** [make-package] Error 2
make[1]: *** [all] Error 2
make: *** [package] Error 2

Comment 2

[Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me)]

Took a quick look and in this instance it appears to be related to there being no components in the components directory after packaging on Windows unlike bug 653971 where on Mac the package has one binary component in the components directory. I'll look a bit more later.

Comment 3

[Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me)]

Attachment: patch in progress

Not sure this is the right approach... just putting this in the bug in case someone has time to look at this which I don't at the moment.

Comment 4

[Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me)]

Mike, fairly certain switching xulrunner to omnijar caused this bug / broke Win32 xulrunner builds.

Comment 5

[Aki Sasaki [:aki] (PTO, back May28)]

I wonder if this is related to bug 661792.  If it's the same problem, this should be fixed during the buildbot master reconfigure tomorrow.  If not, something else needs fixing.

Comment 6

[Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me)]

I saw that and think it might be related but it might not be fixed by it due to xulrunner not having any binary components on Windows.

Comment 7

[Mike Hommey [:glandium]]

An easy immediate fix would be to backout part 5 of bug 620931 (https://bug620931.bugzilla.mozilla.org/attachment.cgi?id=499282 )

That being said, I think the problem is that xulrunner doesn't have a package manifest, and as such, components manifests are not aggregated. I think it's about time we have a package manifest for the toolkit, that the browser package manifest would include. And then (finally) have one for xulrunner.

Comment 8

[Mike Hommey [:glandium]]

(In reply to comment #7)
> That being said, I think the problem is that xulrunner doesn't have a
> package manifest, and as such, components manifests are not aggregated.

meaning we have plenty of components/*.manifest files, and no components/components.manifest nor components/interfaces.manifest.

Comment 9

[Mike Hommey [:glandium]]

(In reply to comment #7)
> I think it's about time we have a package manifest for the toolkit, that the
> browser package manifest would include. And then (finally) have one for
> xulrunner.

That wouldn't solve the problem for other things not using a package manifest, though. So an easy first way out would be to iterate over components/*.manifest instead of dealing with components/components.manifest only.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Iterate over all component manifests when commenting out binary components during omnijar packing

Totally untested, but that should look like this.

Comment 11

[[e-mail] (:bhearsum)]

Based on the patch, looks like this belongs in Core: Build Config

Comment 12

[Alex Vincent [:WeirdAl]]

Could someone at least push this to tryserver, see if it fixes the build?

Comment 13

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Pushing it to tryserver isn't going to kick off a XULRunner SDK build ...

Comment 14

[Michael Wu [:mwu]]

(In reply to comment #7)
> That being said, I think the problem is that xulrunner doesn't have a
> package manifest, and as such, components manifests are not aggregated. I
> think it's about time we have a package manifest for the toolkit, that the
> browser package manifest would include. And then (finally) have one for
> xulrunner.

bug 526333

Comment 15

[Alex Vincent [:WeirdAl]]

I've tested the patch locally, and it works.

Comment 16

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 537710 [details] [diff] [review]
Iterate over all component manifests when commenting out binary components during omnijar packing

Comment 15 says this works locally (the best test we have for XR SDK). Seems fine to me, but sending to khuey for sanity checks.

Comment 17

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Comment on attachment 537710 [details] [diff] [review]
Iterate over all component manifests when commenting out binary components during omnijar packing

Looks reasonable enough, and one of the ten people on earth who care about XULRunner says it works, so let's do it.

Comment 18

[Alex Vincent [:WeirdAl]]

FWIW, mozilla-central XULRunner builds are broken for the same reason, according to Tinderbox, so we need to land this patch there too.

Comment 19

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

http://hg.mozilla.org/mozilla-central/rev/5bb18a13d2b9

Going to wait for tomorrow's XULRunner builds before closing this out.

Comment 20

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

XULRunner on m-c completed successfully this morning.

Comment 21

[Alex Vincent [:WeirdAl]]

Comment on attachment 537710 [details] [diff] [review]
Iterate over all component manifests when commenting out binary components during omnijar packing

XULRunner tinderbox shows this fixes the build on mozilla-central.  Requesting approval for Aurora.

Comment 22

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

This may have broken Mac XULRunner builds :-(
I'm going to back it out to see.

Comment 23

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

http://hg.mozilla.org/mozilla-central/rev/17219648b629

Comment 24

[Josh Matthews [:jdm]]

Attachment: Back out since it may have broken Mac XULRunner builds.

Comment 25

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Backing this out fixed Mac XULRunner builds, so we'll need a new patch here.

Comment 26

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Comment on attachment 537710 [details] [diff] [review]
Iterate over all component manifests when commenting out binary components during omnijar packing

Clearing approval request because this patch has unacceptable side effects.

Comment 27

[Mike Hommey [:glandium]]

For the record, the mac failure log is here:
http://tinderbox.mozilla.org/showlog.cgi?tree=XULRunner&errorparser=unix&logfile=1307959643.1307967855.17882.gz&buildtime=1307959643&buildname=OS%20X%2010.6.2%20mozilla-central%20xulrunner&fulltext=1

The relevant part:
/builds/slave/cen-osx64-xr/build/build/macosx/universal/unify \
          --unify-with-sort "\.manifest$" \
          --unify-with-sort "components\.list$" \
	  obj-firefox/i386/dist/xulrunner/XUL.framework \
	  obj-firefox/x86_64/dist/xulrunner/XUL.framework \
	  obj-firefox/i386/dist/universal/xulrunner/XUL.framework
/builds/slave/cen-osx64-xr/build/build/macosx/universal/unify: warning: makeUniversalDirectory: only in x86 obj-firefox/x86_64/dist/xulrunner/XUL.framework/Versions/7.0a1:
  xulrunner
Can't call method "path" on an undefined value at /builds/slave/cen-osx64-xr/build/build/macosx/universal/unify line 1092.
make[2]: *** [postflight_all] Error 255
make[1]: *** [realbuild] Error 2

There effectively no xulrunner file in one of the zip outputs, but I don't see how this relates to this patch...

Comment 28

[Mike Hommey [:glandium]]

(In reply to comment #27)
> There effectively no xulrunner file in one of the zip outputs, but I don't
> see how this relates to this patch...

Not zip, rsync.

Comment 29

[Alex Vincent [:WeirdAl]]

I just tried build XULRunner SDK on my Mac with the patch applied (i386 only), and it worked fine.  I can't build x86_64 as the MacBook is not a 64-bit machine (probably about 5 years old).

Comment 31

[Chris AtLee [:catlee]]

(In reply to comment #28)
> (In reply to comment #27)
> > There effectively no xulrunner file in one of the zip outputs, but I don't
> > see how this relates to this patch...
> 
> Not zip, rsync.

That's bug 667012.

Comment 32

[Chris AtLee [:catlee]]

This busted win32 xulrunner builds for 6.0 beta.

Comment 33

[Armen [:armenzg] (back on May 20th, 2019)]

The last time we saw a win32 xulrunner was June 13th
http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/latest-mozilla-central/?C=M;O=A
http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/latest-mozilla-aurora/?C=M;O=D


I see the following files for Mac (dated July 11th):
* xulrunner-8.0a1.en-US.mac-pkg.dmg
* xulrunner-8.0a1.en-US.mac-i386.sdk.tar.bz2
* xulrunner-8.0a1.en-US.mac-pkg.txt
* xulrunner-8.0a1.en-US.mac-x86_64.sdk.tar.bz2
* xulrunner-8.0a1.en-US.mac-pkg.checksums

I believe this got fixed for Mac with khuey's change:
http://hg.mozilla.org/mozilla-central/rev/518a79e90444cce2919b4e13a559b45aef080360

Does this mean that this is fixed for Mac but still broken for Win32?
Could we separate the two issues?

Does anyone know what the issue for Windows is? It seems that most comments are with regards to the Mac issue.

Comment 34

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

The fix for Windows broke Mac, so we backed it out.

I'll look into this if I ever get the Mac from Bug 664011.  Been waiting a month now ...

Comment 35

[Armen [:armenzg] (back on May 20th, 2019)]

(In reply to comment #34)
> The fix for Windows broke Mac, so we backed it out.
> 
> I'll look into this if I ever get the Mac from Bug 664011.  Been waiting a
> month now ...

I believe you fixed the issue for Mac on your push this weekend.
Do you need the Mac to run Windows on it? (I might be getting things mixed up).

Now I understand why we have xulrunner builds on the 13th since a fix got landed and backed out.


On another note, I believe xulrunner nightly builds should show up on Firefox's tbpl trees (m-c, m-a & m-b). Anyone disagrees? IMHO it would help catch issues earlier.

Comment 36

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

(In reply to comment #35)
> (In reply to comment #34)
> > The fix for Windows broke Mac, so we backed it out.
> > 
> > I'll look into this if I ever get the Mac from Bug 664011.  Been waiting a
> > month now ...
> 
> I believe you fixed the issue for Mac on your push this weekend.
> Do you need the Mac to run Windows on it? (I might be getting things mixed
> up).

Mac should have been fixed on June 13th or whenever the backout happened.  I need a Mac build environment to see why this change broke Mac.

> Now I understand why we have xulrunner builds on the 13th since a fix got
> landed and backed out.

Right.

> On another note, I believe xulrunner nightly builds should show up on
> Firefox's tbpl trees (m-c, m-a & m-b). Anyone disagrees? IMHO it would help
> catch issues earlier.

I don't disagree, but this is something that should be raised more widely.  I'll bring it up at the engineering meeting tomorrow.

Comment 37

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Sigh, apparently the Mac XR builds randomly turn red ... and it looks like landing this patch coincided with that.

Comment 38

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Trying this again ...

http://hg.mozilla.org/mozilla-central/rev/8753de11b181

Comment 39

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

And this was random XR mac bustage.

Comment 40

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Comment on attachment 537710 [details] [diff] [review]
Iterate over all component manifests when commenting out binary components during omnijar packing

Drivers, we will need to take this on Aurora and Beta so that third parties can build binary addons.

Comment 41

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Filed Bug 670951 for the intermittent Mac failures.

Comment 42

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Alex, can you verify that the SDKs you need are all on present on FTP for mozilla-central?

Comment 43

[Alex Vincent [:WeirdAl]]

Confirmed: ftp://ftp.mozilla.org/pub/xulrunner/nightly/latest-mozilla-central/xulrunner-8.0a1.en-US.win32.zip is there for me to pull, if that were the one I wanted.  (We're sticking to Aurora builds.  :-) )

Also, tinderbox shows XULRunner on mozilla-central building that .sdk.zip file.  That would be the one I need from Aurora, and now Beta.

Comment 44

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Yeah, I'll get it on the branches asap.

Comment 45

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

http://hg.mozilla.org/releases/mozilla-beta/rev/53cb50eb7dab
http://hg.mozilla.org/releases/mozilla-beta/rev/53cb50eb7dab

Comment 46

[Alex Vincent [:WeirdAl]]

:-D I now have SDK's for FF6 Beta and FF7 Aurora from ftp.mozilla.org - thanks!

Comment 47

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

Yay!

Comment 48

[:Ehsan Akhgari (Out of office 5/10-5/21)]

This seems to have landed on Aurora, so we didn't transplant it to beta during migration:

http://hg.mozilla.org/releases/mozilla-aurora/rev/e0a0e1800c6a

Comment 49

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

(In reply to Ehsan Akhgari [:ehsan] from comment #48)
> This seems to have landed on Aurora, so we didn't transplant it to beta
> during migration:
> 
> http://hg.mozilla.org/releases/mozilla-aurora/rev/e0a0e1800c6a

What does that mean?  It came along with the merge, no?

Comment 50

[:Ehsan Akhgari (Out of office 5/10-5/21)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #49)
> (In reply to Ehsan Akhgari [:ehsan] from comment #48)
> > This seems to have landed on Aurora, so we didn't transplant it to beta
> > during migration:
> > 
> > http://hg.mozilla.org/releases/mozilla-aurora/rev/e0a0e1800c6a
> 
> What does that mean?  It came along with the merge, no?

No, it was landed separately by you: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?changeset=e0a0e1800c6a

I just wanted to mention that this change did not make it into Beta.  If you think it should, please nominate it.  Thanks!

Comment 51

[Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)]

(In reply to Ehsan Akhgari [:ehsan] from comment #50)
> (In reply to Kyle Huey [:khuey] ([e-mail]) from comment #49)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #48)
> > > This seems to have landed on Aurora, so we didn't transplant it to beta
> > > during migration:
> > > 
> > > http://hg.mozilla.org/releases/mozilla-aurora/rev/e0a0e1800c6a
> > 
> > What does that mean?  It came along with the merge, no?
> 
> No, it was landed separately by you:
> http://hg.mozilla.org/releases/mozilla-aurora/
> pushloghtml?changeset=e0a0e1800c6a
> 
> I just wanted to mention that this change did not make it into Beta.  If you
> think it should, please nominate it.  Thanks!

http://hg.mozilla.org/releases/mozilla-beta/rev/e0a0e1800c6a

Sure looks like it's in beta to me ...

Comment 52

[:Ehsan Akhgari (Out of office 5/10-5/21)]

Hmm, then I don't remember why I commented here during the migration, but there _was_ a reason... :/