c-c "make install" fails to package omni.ja when building enigmail at the end of the build

RESOLVED FIXED

Status

SeaMonkey
Build Config
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: gaston, Assigned: gaston)

Tracking

SeaMonkey 2.19 Branch
x86
OpenBSD

SeaMonkey Tracking Flags

(seamonkey2.19 affected)

Details

Attachments

(1 attachment)

(Assignee)

Description

4 years ago
2.19b1 fails to package omni.ja because the packager complains about removing files that are supposed to be in omni.ja and listed in package-manifest.in. Iirc that was also a problem with 2.18b4 last i tried.
It looks that it lists all the js files under components...
(full error list at http://pastebin.mozilla.org/2554529)

I will try a seamonkey build from c-c.

 /usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/_virtualenv/bin/python /usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/toolkit/
mozapps/installer/packager.py -DMOZ_GLUE_IN_PROGRAM -DMOZ_SUITE=1 -DOSTYPE=\"OpenBSD5\" -DOSARCH=OpenBSD -DNO_NSPR_10_SUPPORT -
DAB_CD=en-US -DMOZ_APP_NAME=seamonkey -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_ENABLE_GNOME_COMPONENT=1 -DMOZ_GTK2=1 -DMOZ_URL_
CLASSIFIER=1 -DMOZ_NATIVE_NSPR=1 -DMOZ_NATIVE_NSS=1 -DMOZ_MOVEMAIL=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DDLL_PREFIX=lib
 -DDLL_SUFFIX=.so.30.0 -DBIN_SUFFIX= -DBINPATH=bin \
        --format omni \
        --removals /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in \
         \
         \
         \
        --optimizejars \
         \
        package-manifest ../../mozilla/dist ../../mozilla/dist/seamonkey \
        --non-resource defaults/messenger/mailViews.dat defaults/profile/localstore.rdf defaults/profile/panels.rdf 
Error: /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in:196: Removal of packaged file(s): components/nsFilePicker.js
Error: /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in:619: Removal of packaged file(s): components/addonManager.js

...
...

Error: /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in:716: Removal of packaged file(s): components/Webapps.js
Error: /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in:717: Removal of packaged file(s): components/WebContentConverter.js
Traceback (most recent call last):
  File "/usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/toolkit/mozapps/installer/packager.py", line 374, in <module>
    main()
  File "/usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/toolkit/mozapps/installer/packager.py", line 327, in main
    copier.add(mozpack.path.join(binpath, 'removed-files'), removals)
  File "/usr/local/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/python/mozbuild/mozpack/errors.py", line 129, in accumulate
    raise AccumulatedErrors()
mozpack.errors.AccumulatedErrors
gmake[1]: *** [stage-package] Error 1
gmake[1]: Leaving directory `/usr/obj/seamonkey-2.19beta1/build-amd64/suite/installer'
gmake: *** [install] Error 2
can you provide more information about your build environment. .. your mozconfig... etc.  it worked fine for our official builds.  but if there is an issue buried here I'd love to get it fixed
(Assignee)

Comment 2

4 years ago
No .mozconfig, this is building from the 2.19b1 source tarball within our ports infrastructure, and fails during make install to a temporary staging dir.

(lots of the configure arguments are now pointless ofc)
$make show=CONFIGURE_ARGS
--enable-calendar --enable-official-branding --disable-gconf --enable-gio --with-system-libevent=/usr/ --with-system-bz2=/usr/local --enable-system-sqlite --with-system-zlib=/usr       --with-system-nspr              --with-system-nss               --with-pthreads                         --disable-optimize              --disable-tests                         --disable-pedantic              --disable-installer             --disable-updater               --disable-gnomeui               --disable-gnomevfs              --disable-dbus                  --enable-default-toolkit=cairo-gtk2  --enable-xinerama          --enable-svg                    --enable-svg-renderer=cairo     --enable-canvas --enable-system-cairo --disable-freetypetest            --disable-mochitest             --disable-libIDLtest            --disable-glibtest              --disable-necko-wifi            --disable-crashreporter                 --disable-libnotify             --enable-xft                    --disable-ipc --enable-application=suite --prefix='/usr/local' --sysconfdir='/etc' --mandir='/usr/local/man' --infodir='/usr/local/info' --localstatedir='/var' --disable-silent-rules
I do know that anything outside the scope of the default such as including calendar will cause the packager to error out. Not sure of the specific reason but it DOES happen.

I have had nothing but trouble with the new packaging system in core for quite some time. Largely with multi-jar though. Omnija usually works.
(Assignee)

Comment 4

4 years ago
Fwiw regular make package from c-c tip is fine with sm 2.22a1 & --enable-application=suite. Will retry with --enable-calendar.

Comment 5

4 years ago
--enable-calendar isn't really well-supported and probably will fail way harder with make install.

That said, this bug might "just" mean that "make install" is broken - we are not using that for our official builds, we use "make package".

Comment 6

4 years ago
I thought make package just generates the ZIP file and make installer makes the .exe installer?

Also on windows I always do |make package && make installer| after every build to make sure that packaging is still working.
(Assignee)

Comment 7

4 years ago
We're rather talking about make install to a dir here ...

Other data points: i can do make package on c-c with --enable-calendar.

It also seems i can do DESTDIR=/tmp gmake -C /usr/obj/c-c install without issues on trunk, the packager doesnt complain about removed files.

Trying to build sm 2.19b1 without enigmail/lightning to rule out those parameters.
(Assignee)

Updated

4 years ago
Duplicate of this bug: 866494
(Assignee)

Comment 9

4 years ago
(In reply to Landry Breuil (:gaston) from comment #7)
> Trying to build sm 2.19b1 without enigmail/lightning to rule out those
> parameters.

Not building enigmail and disabling calendar helps, make install goes past the failing spot. puzzling.
This regression in the packaging was introduced when SM came into alignment with firefox on omnija, packaging, and pymake. Really there are not enough fallbacks and conditionals if you are doing anything outside the default shipping configuration that SM proper uses.

Comment 11

4 years ago
(In reply to Philip Chee from comment #6)
> I thought make package just generates the ZIP file and make installer makes
> the .exe installer?

"Make installer" is not "make install". The latter is a quasi-standard for "install needed files into the correct places on the system", the former a Mozilla-specific target to build the (Windows) installer packages.
Summary: 2.19b1 fails to package omni.ja → 2.19b1 "make install" fails to package omni.ja
(Assignee)

Comment 12

4 years ago
More data point: i can build and make install is fine when building with --enable-calendar on sm 2.19b1.

If i build enigmail within tb tree (after having done the regular TB build) with the recommended upstream instructions, make install fails.
(Assignee)

Comment 13

4 years ago
I compared the package-manifest file generated at the beginning of make install, and they're exactly the same, whether building enigmail or not.

Excerpt of a working log of make install without enigmail :

===>  Faking installation for seamonkey-2.19beta1
gmake[1]: Entering directory `/usr/obj/seamonkey-2.19beta1/build-amd64/suite/installer'
/usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/_virtualenv/bin/python /usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/config/Preprocessor.
py -DMOZ_GLUE_IN_PROGRAM -DMOZ_SUITE=1 -DOSTYPE=\"OpenBSD5\" -DOSARCH=OpenBSD -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=seamonkey -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_ENABLE_GNOME_COMPONENT=1 -DMOZ_GTK2=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_NATIVE_NSPR=1 -DMOZ_NATIVE_NSS=1
 -DMOZ_MOVEMAIL=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DDLL_PREFIX=lib -DDLL_SUFFIX=.so.30.0 -DBIN_SUFFIX= -DBINPATH=bin -DHAVE_64BIT_OS=1 -DMOZILLA_VERSION=\"22.0\" -DMOZILLA_VERSION_U=22.0 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_LIBM=1 -DHAVE_DLADDR=1 -DFUNCPROTO=15 -D_REENTRANT=1 -D_THREAD_SAFE=1 -DMMAP_MISSES_WRITES=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_LANGINFO_CODESET=1 -DVA_COPY=va_copy -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DMALLOC_H=\<sys/malloc.h\> -DHAVE_STRNDUP=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_VALLOC=1 -DHAVE_I18N_LC_MESSAGES=1 -DHAVE_LOCALECONV=1 -DNS_ALWAYS_INLINE= -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DNS_NORETURN=__attribute__\(\(noreturn\)\) -DMOZ_SUITE=1 -DMOZ_BUILD_APP=suite -DMOZ_X11=1 -DMOZ_WIDGET_GTK2=1 -DMOZ_OFFICIAL_BRANDING=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DMOZ_WEBSPEECH=1 -DMOZ_XTF=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_FEEDS=1 -DMOZ_STORAGE=1 -DMOZ_NATIVE_SQLITE=1 -DMOZ_HELP_VIEWER=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DHAVE___CXA_DEMANGLE=1 -DHAVE__UNWIND_BACKTRACE=1 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DBUILD_CTYPES=1 -DMOZ_MORK=1 -DMOZ_MORKREADER=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_COMMON=1 -DMOZ_SERVICES_CRYPTO=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_UA_BUILDID=\"20100101\" -DMOZ_DLL_SUFFIX=\".so.30.0\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1 -DMOZ_ACCESSIBILITY_ATK=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=8 -DATK_REV_VERSION=0 /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/package-manifest.in > package-manifest
/usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/_virtualenv/bin/python /usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/toolkit/mozapps/installer/packager.py -DMOZ_GLUE_IN_PROGRAM -DMOZ_SUITE=1 -DOSTYPE=\"OpenBSD5\" -DOSARCH=OpenBSD -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=seamonkey -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_ENABLE_GNOME_COMPONENT=1 -DMOZ_GTK2=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_NATIVE_NSPR=1 -DMOZ_NATIVE_NSS=1 -DMOZ_MOVEMAIL=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DDLL_PREFIX=lib -DDLL_SUFFIX=.so.30.0 -DBIN_SUFFIX= -DBINPATH=bin \
        --format omni \
        --removals /usr/obj/seamonkey-2.19beta1/comm-beta/suite/installer/removed-files.in \
         \
         \
         \
        --optimizejars \
         \
        package-manifest ../../mozilla/dist ../../mozilla/dist/seamonkey \
        --non-resource defaults/messenger/mailViews.dat defaults/profile/localstore.rdf defaults/profile/panels.rdf 
Executing /usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/dist/bin/xpcshell -g /usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/dist/bin/ 
-a /usr/obj/seamonkey-2.19beta1/build-amd64/mozilla/dist/bin/ -f /usr/obj/seamonkey-2.19beta1/comm-beta/mozilla/toolkit/mozapps/installer/precompile_cache.js -e precompile_startupcache("resource://gre/");
resource://gre/components/nsINIProcessor.js
resource://gre/components/AppProtocolHandler.js


The argument list passed to installer/packager.py seems to be the same in the working & non-working case... of course in the 'broken' case mozilla/dist contains more stuff (ie the enigmail files were installed there)
(Assignee)

Comment 14

4 years ago
I compared the content of dist/ in both cases, and the broken make install complains about 'wrongly removing' 83 js files under components/, while there are 107 of them under dist/components in the working case. So it's not *all* js files under components.

Note that i switched to use a zip from enigmail's git master to see if it changes anything.
(Assignee)

Comment 15

4 years ago
-components/ActivityOptions.js
-components/ActivityProxy.js
-components/ActivityRequestHandler.js
-components/ActivityWrapper.js
-components/AppProtocolHandler.js
-components/AppsService.js
-components/PageThumbsProtocol.js
-components/PeerConnection.js
-components/PermissionPromptService.js
-components/PermissionSettings.js
-components/Push.js
-components/PushService.js
-components/SettingsService.js
-components/SystemMessageInternal.js
-components/SystemMessageManager.js
-components/TCPSocketParentIntermediary.js
-components/TelemetryPing.js
-components/recording-cmdline.js

is the list of files that omni.ja packager doesnt complain about in the broken case, if that can matter..
(Assignee)

Comment 16

4 years ago
And it seems that tb 22.0b1 fails the same way - to be reconfirmed.
(Assignee)

Comment 17

4 years ago
Note that the issue is now in sm 2.19 release
status-seamonkey2.19: --- → affected
(Assignee)

Comment 18

4 years ago
cc'ing the moz.build/mach folks, hoping they might have an idea..
Summary: 2.19b1 "make install" fails to package omni.ja → c-c "make install" fails to package omni.ja when building enigmail at the end of the build
Obviously, suite/installer/removed-files.in is just wrong.
That being said, there's something fishy... components/nsFilePicker.js et al. should be in omni.ja and thus shouldn't match during removal.

Can you try this and report what it prints?

diff --git a/toolkit/mozapps/installer/packager.py b/toolkit/mozapps/installer/packager.py
--- a/toolkit/mozapps/installer/packager.py
+++ b/toolkit/mozapps/installer/packager.py
@@ -323,6 +323,7 @@ def main():
             lines = [l.lstrip() for l in open(args.removals).readlines()]
             removals_in = StringIO(''.join(lines))
             removals_in.name = args.removals
+            print "\n".join(copier._files.keys())
             removals = RemovedFiles(copier)
             preprocess(removals_in, removals, defines)
             copier.add(mozpack.path.join(binpath, 'removed-files'), removals)
(Assignee)

Comment 21

4 years ago
(In reply to Mike Hommey [:glandium] from comment #19)
> Obviously, suite/installer/removed-files.in is just wrong.

But then, why/how does it work when _not_ building enigmail ?

(In reply to Mike Hommey [:glandium] from comment #20)
> That being said, there's something fishy... components/nsFilePicker.js et
> al. should be in omni.ja and thus shouldn't match during removal.
> 
> Can you try this and report what it prints?
> 
> diff --git a/toolkit/mozapps/installer/packager.py
> b/toolkit/mozapps/installer/packager.py
> --- a/toolkit/mozapps/installer/packager.py
> +++ b/toolkit/mozapps/installer/packager.py
> @@ -323,6 +323,7 @@ def main():
>              lines = [l.lstrip() for l in open(args.removals).readlines()]
>              removals_in = StringIO(''.join(lines))
>              removals_in.name = args.removals
> +            print "\n".join(copier._files.keys())
>              removals = RemovedFiles(copier)
>              preprocess(removals_in, removals, defines)
>              copier.add(mozpack.path.join(binpath, 'removed-files'),
> removals)


With that, it prints a huge list, which also contains (among all the other files) all the files it complains about a removal:

chrome/omni.ja
components/chrome.manifest
components/omni.ja
chrome/en-US/locale/en-US/necko/necko.properties
chrome/en-US/locale/en-US/global/css.properties
chrome/en-US/locale/en-US/global/xbl.properties
chrome/en-US/locale/en-US/global/xul.properties
...
...
chrome/gloda/content/glodacomplete.xml
components/glautocomp.js
components/jsmimeemitter.js
components/newMailNotificationService.js
distribution/extensions/{59c81df5-4b7a-477b-912d-4e0fdf64e5f2}.xpi
distribution/extensions/{f13b157f-b174-47e7-a34d-4815ddfdfeb8}.xpi
distribution/extensions/inspector@mozilla.org.xpi
Error: /usr/obj/seamonkey-2.19/comm-release/suite/installer/removed-files.in:196: Removal of packaged file(s): components/nsFilePicker.js
Error: /usr/obj/seamonkey-2.19/comm-release/suite/installer/removed-files.in:619: Removal of packaged file(s): components/addonManager.js


I can attach the complete list if needed....
Ouch, something in dist/bin confuses the omnijar formatter... Could you put an archive of your dist/bin (dereferencing symlinks) up somewhere? (you can replace binaries with empty files, what matters is the other files)
(Assignee)

Comment 23

4 years ago
Interesting other data point. If, after building TB, i only do the two first steps of enigmail building (ie cd mailnews/extensions/enigmail && makemake -r -o $objdir + cd objdir && make) then the packager doesnt barf. It seems to only barf if i had done the "make xpi" step. I will save the two dist/bin somewhere.
(Assignee)

Comment 24

4 years ago
So yeah, it seems cd ${WRKBUILD}/${ENIGMAIL_DIR} && ${SETENV} ${MAKE_ENV} ${MAKE_PROGRAM} xpi in my ports' post-build target is the thing disturbing the packager. If i dont run 'make xpi' after 'make', the packager does omni.ja fine.

I've put two archives of mozilla/dist/bin generated with gtar, the first one in the working case (ie full seamonkey build + makemake & make for enigmail), and the second one in the broken case (same as above + make xpi for enigmail)

http://rhaalovely.net/~landry/shared/dist-bin-ok.tgz
http://rhaalovely.net/~landry/shared/dist-bin-nok.tgz
(Assignee)

Comment 25

4 years ago
Looking at the diff between the two tarballs.. the "broken" one doesnt have mozilla/dist/bin/chrome.manifest. Oops ? I see genxpi in enigmail's removing a chrome.manifest, but it should be the one within enigmail's objdir afaict...
(In reply to Landry Breuil (:gaston) from comment #25)
> Looking at the diff between the two tarballs.. the "broken" one doesnt have
> mozilla/dist/bin/chrome.manifest. Oops ?

Yeah, oops. That's obviously why the packager is not happy.
(Assignee)

Comment 27

4 years ago
That's the core of the issue here. enigmail's genxpi works in mozilla/dist/bin :

genxpi: Generating enigmail-1.5-openbsd-i386.xpi in ../../../mozilla/dist/bin

And it overwrites TB's chrome.manifest with the one from enigmail. If, before overwriting it, i make genxpi mv TB's chrome.manifest away, do its work on enigmail's chrome.manifest, then put back TB's file after the xpi has been generated (and comment out the removal of any chrome.manifest at the bottom of genxpi) then the packager is perfectly happy, as i am.

philipp, what do you think of this issue analysis & resolution ? can we fix it in enigmail with a patch to genxpi i'll wrap up ?
(Assignee)

Comment 28

4 years ago
Created attachment 773916 [details] [diff] [review]
Dont overwrite TB's chrome.manifest

Here's the genxpi patch i came up with, but i suppose a much better fix could be done - or something in my build infra is wrong and badly reuses mozilla/dist/bin instead of working in a separate dir to build enigmail's xpi ?

With that, i can package seamonkey 2.19 + enigmail 1.5.2 + lightning 2.4b1, and they all run fine.
Assignee: nobody → landry
Attachment #773916 - Flags: review?(philipp)
(Assignee)

Comment 29

4 years ago
Comment on attachment 773916 [details] [diff] [review]
Dont overwrite TB's chrome.manifest

Darn, sorry philipp, mixed you again with patrick...
Attachment #773916 - Flags: review?(philipp) → review?(patrick)

Updated

4 years ago
Attachment #773916 - Flags: review?(patrick) → review+

Comment 30

4 years ago
I applied the patch to the Enigmail repository.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Duplicate of this bug: 861690
You need to log in before you can comment on or make changes to this bug.