83200 - Incompatibility with libsafe xpidl malfunction

Status: RESOLVED WONTFIX

(Core :: XPCOM, defect)

Description

[(Don't use, old account) - Kai Engert]

I had a problem for quite a while on my system (RedHat Linux 7.0 / 7.1),
and now found the solution.

The symptoms were:
- the Netscape 6 installer always crashes on my system
- the Mozilla installer always crashes on my system
- when I try to compile Mozilla, the xpidl program produces interface files with
damaged numeric values. (This did not happen when I manually link xpidl
statically.)

I found out that there was a package installed on my system called "libsafe".
It's purpose is (I cite):

"The libsafe library protects a process against the exploitation of
buffer overflow vulnerabilities in process stacks. Libsafe works with
any existing pre-compiled executable and can be used transparently,
even on a system-wide basis. The method intercepts all calls to
library functions that are known to be vulnerable. A substitute
version of the corresponding function implements the original
functionality, but in a manner that ensures that any buffer overflows
are contained within the current stack frame. Libsafe has been shown
to detect several known attacks and can potentially prevent yet
unknown attacks. Experiments indicate that the performance overhead of
libsafe is negligible."

You can read more about this package at:
  http://www.avayalabs.com/project/libsafe/index.html

After I removed this package from my system, my problems immediately went away.
The Mozilla installer now works and xpidl works, too. Haven't tried with the
Netscape 6 installer yet, but assume this was the same problem.

My assumption is:
This library seems to be a plugin into the system shared library loading
process, which is in some way incompatible with the shared library loading
mechanisms of Mozilla (this should be XPCom from my understanding). Maybe it's
just the parts that use the "long long" type, as it seemed only those numbers
were affected in xpidl, and besides from that xpidl worked.

Asa Dotzler suggested, it might be a good idea to include this warning in the
browser's release notes. I agree.

I don't know how many people have this package installed on their systems. It is
NOT contained on the main Red Hat Linux installation CDs, so no problem even if
people do a full install.

However, Red Hat is delivered with an extra CD called "Powertools", which
contains the libsafe. Therefore, curious souls like me might have this package
installed.

The version of libsafe which showed the incompatibility with Mozilla was 1.3,
which is included with Red Hat 7.1.

Comment 1

[(Don't use, old account) - Kai Engert]

One of the consequences of this problem is: xpidl warns about "initialized with
negative constant" for some of the constants in the idl files.

Comment 2

[Doug Turner (:dougt)]

reassign all kandrot xpcom bug.

Comment 3

[Daniel Wang]

this is a very old bug. reporter, is this bug still valid?

Comment 4

[Kai Engert (:KaiE:)]

Since nobody else ever complained about this, I suspect only very few developers
have libsafe installed. After I had uninstalled it two years ago, I never
installed it again.

I don't see the bug is invalid, if at all, I suggest it to be WONTFIX.

However, if we resolve it, people might not find it, so it might make sense to
leave the bug open.

Comment 5

[Matthew Cline]

I haven't had Mozilla crash due to libsafe in a long time, so that part of the
bug is probably fixed.  However, I haven't compiled Mozilla in a long time, so
I can't speak about the xpidl part of the bug.

Comment 6

[Asa Dotzler [:asa]]

updating summary to remove crashing comment per comment 5