39808 - xptinfo attempts to create xptitemp.dat in component directory

Status: RESOLVED WORKSFORME

(Core :: XPCOM, defect, P5)

Description

[tor]

On a 5/18 cvs pull compiled --disable-debug --enable-optimize, I'm seeing
mozilla fail to start when installed on a read-only filesytem.  Looking
at the output of "truss", the last thing mozilla appears to be doing is
attempting to create xptitemp.dat in the components directory.

open64("/cs/lib/mozilla/components/xptitemp.dat", O_WRONLY|O_CREAT|O_TRUNC,
0666) Err#30 EROFS
Could not obtain CmdLine processing service
write(2, " C o u l d   n o t   o b".., 44)      = 44
llseek(4, 0, SEEK_CUR)                          = 719
close(4)                                        = 0
llseek(0, 0, SEEK_CUR)                          = 136645
_exit(1)

Does mozilla really need this file and can't continue without it?
If so, it shouldn't be trying to create it in the components directory,
which is likely not writable by the average user (unless they installed
their own version).

Comment 1

[Mike Shaver (:shaver emeritus)]

This is pretty serious: it means that if a system administrator installs
Mozilla, regular users will not be able to use it.

(Our Linux installer defaults to a path that requires system administration
privileges to access, as well.)

mccabe: you might have to take this one before jband is back.

Comment 2

[Mike McCabe]

Assigning to me.  I'm assuming there's some handy place to put this...

Do we have an xp notion of tempfile/temp directory, or do I need to hunt down
some user directory?

Comment 3

[John Bandhauer]

...As I am currently not without net access and unable to avoid peeking now 
and then...

This is the file that gets copied (should be renamed, but nsIFile::moveTo is 
broken) to "xpti.dat". It is completely analogous to "component.reg" but in 
"bin/components" rather than "bin". Once it gets correctly created and no 
components change then it will not need rewriting. 

You could make failure to write the file a non-fatal error. NOTE: this will mask 
the fact that a full autoreg of the .xpt files is then required on each and 
every start of the app. You have the same issue with component.reg too; i.e. 
though failing to create component.reg may not be causing failure to start the 
app, it *must* require a full component auotreg on each start, no?

I don't think that moving xpti*.dat to a temporary dir is a reasonable solution. 
Nor, should it be moved to a user dir as long as it is an index of all the 
typelibs of the app (shared by all users).

Running the app once from an account that has write priv's to this dir ought to 
make it run fine from then on. One solution is to simply document that fact.

Whatever solution is chosen ought to be applied to both the xpti*.dat and 
component.reg files.

Comment 4

[tor]

I'm doing something similar to what you suggest as a solution but

am still running into the problem I described.  Perhaps this might

shed some light onto what is happening:



  * mozilla is compiled on a local disk

  * mozilla-bin is run so component.reg and xpti*.dat are created

  * rdist is used to push the dist/bin directory into a read-only

    installation directory

  * running mozilla after installation results in a "Could not obtain

    CmdLine processing service" message and a segfault.  trussing

    the run shows it failing to open xptitemp.dat



Neither component.reg nor xpti*.dat include directory names anymore,

so this installation should work.

Comment 5

[Mike Shaver (:shaver emeritus)]

Are you talking about having to run the browser as root once in the _release_
builds?  I don't think that's acceptable: no distribution is going to start up
Mozilla as part of the install, just to get us to create some file.

For component.reg, I think the installer creates it.  Tor, if you're barfing on
component.reg too, could you try running regxpcom before you rdist?  That might
solve problems: I presume we only need to create xptitemp.dat when we're
registering new components, in which case blizzard's fixes for 14923 are
required before users can meaningfully install things anyway.

--disable-debug --enable-optimize should turn off autoreg, right?  Maybe
_that_'s the bug: non-debug builds still try and do that autoreg thing, which is
causing suck.  Even if we don't turn autoreg off #ifndef DEBUG, we should make
it fail more gracefully in that case.

Comment 6

[Suresh Duddi (gone)]

I would reiterate the general principle that xpt*.dat creation should mimic or 
piggyback on component.reg creation. This was what jband and myself were trying 
to achieve.

That being said, I dont understand why you are seeing such a behaviour tor. 
Unless Autoreg() was triggered even after first run (aka install). Can you like 
put a printf in nsComponentManager.cpp::AutoRegister() and see if that is being 
called. If so, then the call itself is the bug not the fact that xpt* stuff 
happens.

Comment 7

[tor]

Looks like this might be partially rdist wierdness:

  * if you rdist the dist/bin directory directly, you get the reported error
  * if you first "cp -rp" dist/bin to a new directory (which makes actual
    copies of all the symlinked files), run mozilla-bin/regxpcom, then it
    no longer tries to create xptitemp.dat.  mozilla still doesn't run, but
    that's that should be handled by another bug.

Is mozilla-bin/regxpcom somehow recording the fact that the libraries are
symlinks in the registration files and noticing a mismatch when it's run
from a copied tree?

Comment 8

[tor]

That crash I mentioned in the last comment looks like it might be related
to xpcom registration, so I'll include the stack trace here for reference
(sorry, no debugging info in this build):

Program received signal SIGSEGV, Segmentation fault.
0xff213574 in nsNativeComponentLoader::AutoRegisterComponent ()
(gdb) where
#0  0xff213574 in nsNativeComponentLoader::AutoRegisterComponent ()
#1  0xff212930 in nsNativeComponentLoader::RegisterComponentsInDir ()
#2  0xff2127f0 in nsNativeComponentLoader::AutoRegisterComponents ()
#3  0xff210be8 in nsComponentManagerImpl::AutoRegister ()
#4  0xff216e2c in nsComponentManager::AutoRegister ()
#5  0x17730 in NS_AutoregisterComponents ()
#6  0x17958 in NS_SetupRegistry_1 ()
#7  0x16b10 in main1 ()
#8  0x176fc in main ()

Comment 9

[Peter Trudelle]

need info: how critical is this for PR2 commercial release.  Is it just 
Solaris?

Comment 10

[tor]

For the purposes of this bug, Solaris==Unix==Linux.

This problem might also show up on WinNT/2000 multi-user
systems or Win98/MacOS with locked volumes.

Comment 11

[Mike Shaver (:shaver emeritus)]

Yeah, any Unix system with a ``properly'' installed Mozilla for its users will
get bit.  I vote for nsbeta2.

Comment 12

[tor]

More fun quirks: if I do the following steps, mozilla runs fine:

  * make a "cp -rp" copy of dist/bin
  * run mozilla in that tree
  * rdist that to the read-only filesytem
  * remove component.reg from the read-only copy and symlink out to a
     writable location
  * run mozilla

component.reg is filled in and mozilla runs normally.

Comment 13

[Mike McCabe]

Any connection between this and 38603?

Comment 14

[rickg]

Failure to launch==critical issue. Promoting to nsbeta2+

Comment 15

[Ben Bucksch (:BenB)]

Bug 16600 is about (the same issue (?) with) component.reg.

See also meta bug 42184 "Mozilla-bin must not write to bin dir during
installation".

Comment 16

[Albert Yarusso]

As an end-user I just wanted to comment on this bug.  I just installed M16 in 
RedHat 6.2 as root.  When I went to run it for the first time I received the 
"Could not obtain CmdLine processing service" error.  I looked around on Usenet 
for some answers, but could not find any.  As a last resort I came here and 
fortunately found this bug.  I was able to get Mozilla running by starting it 
as root once and then running it after that with a normal login.  I'm still a 
relatively novice Linux user, but an experienced PC/Windows programmer, and 
this stopped me dead in my tracks until I came to Bugzilla.  So just a vote 
from me that this is a pretty serious bug, IMHO.

Comment 17

[Mike Shaver (:shaver emeritus)]

Did you use Chris Blizzard's RPMs, or the tarball?  I think the RPMs (and the
installer version, I suspect) will do the registry creation piece for you, but
I'm not 100% certain.

Comment 18

[Albert Yarusso]

I downloaded the mozilla-i686-pc-linux-gnu-installer-M16.tar.gz tarball, 
extracted it, ran the installer (as root) and then tried to run Mozilla as a 
normal user.  That's when I got the CmdLine error.

Comment 19

[Mike Shaver (:shaver emeritus)]

If the RPMs and installer do the right thing WRT creating the registry as root,
then this might not be as critical an issue.  (We should probably add something
to the release notes about how to solve the CmdLineService issue, too.)

Comment 20

[Christopher Blizzard (:blizzard)]

The RPMS do the registry magic crap at build time.  You should be able to
install the RPM and just type "mozilla" and that should be enough.

Comment 21

[John Bandhauer]

After mozilla has been installed and run once then the xpti*.dat and 
component.reg file should not be being written. If they *are* being written 
again, then we need to look into that. If they are not being written again then 
we are looking at a release note kind of issue for beta purposes and not a bug 
that needs fixing.

Comment 22

[Patrick C. Beard]

Not crucial for beta2, workaround should be to either install in a user writable 
directory (likely for beta testing), or run browser once as root to do the 
autoregistration. Shouldn't the installer be doing that already? Nominating for 
beta3.

Comment 23

[leger]

Moving to nsbeta2- per triage mtg today.

Timeless, please DO NOT change plus and minus indications. 

Comment 24

[Mike McCabe]

John has agreed to take this one.

Comment 25

[John Bandhauer]

This one is lingering with no plan for a fix. I again say:

1) There is no conceptual difference between what we do with xpti*.dat and 
component.reg. I don't even see why we are talking about one file and not the 
other.

2) We are not currently shipping either file - they are created on first 
startup.

3) There is no clear better place to put them. These are not just 'per user' or 
'per browser' they are 'per installation of xpcom'.

4) I checked in code to xpti to factor out the places where we talk about the 
disk location of this xpti*.dat manifest file. If the DirectoryService defines a 
better place to put these files then this is not hard to change. *But*, like I 
said above, this manifest is on-to-one with an xpcom installation and it is not 
appropriate (and a very likely crasher!) to share it with another installation 
of xpcom. So, putting this in a user dir that is not for one and only one 
installation of xpcom would be a huge mistake.

No one has suggested a solution better than the status quo. This bug is going to 
get "minus'd" unless someone builds a winning argument on why and how to fix it.

Comment 26

[David Krause]

jband, according to grandrose "we *do* ship a component.reg file - it saves on
startup time"

Would it be possible to also ship the xpti.dat and xptitemp.dat files also,
thereby working around this problem?

Comment 27

[David Krause]

Actually I think this bug can be marked FIXED now.  The latest nightly tarball
comes with component.reg, xpti.dat, and xptitemp.dat already due to granrose's
fix for bug 51164.  I removed my .mozilla, chowned the unpacked package
directory to root.root and ran as my user.  I didn't see any errors about
CmdLineService.  Mozilla started without any errors concerning xpti*.dat, though
there were problems due the rdf/install-chrome.txt files which is another bug.
I also did an strace and it shows that mozilla is no longer trying to write to
the xpti*.dat files.

david@hermes:~/package$ strace -e open -f ./mozilla 2>&1 | grep xpti
[pid 27827] open("/home/david/package/components/xpti.dat", O_RDONLY) = 5
[pid 27827] open("/home/david/package/components/xpti.dat", O_RDONLY) = 8

Can we mark this FIXED?  Anyone still having problems with the nightly build
from today?

Comment 28

[John Bandhauer]

If we decide to leave this as it is then it would make more sense to LATER it 
then to mark it FIXED. 

A few points:

- tarballs are not the primary delivery mechanism. I don't think we are 
delivering xpti.dat or component.reg with installer builds at present.

granrose, you were quoted, can you confirm or deny?

- I think we *should* eventually put these component info cache files somewhere 
else.

- windows matters too for this issue.

Comment 29

[Jon Granrose]

tarballs are mozilla's primary delivery mechanism for non-linux unix builds
(solaris, hpux, etc.). Netscape uses tarballs of netscape 6 to do comparisons
whenever we have installer-only problems but only shipped them with PR1
(pre-installer days). As part of the daily build automation we run regxpcom and
generate a component.reg file before packaging the tarball (don't know anything
about a xptitemp.dat file).  However, component.reg is not in
packages-unix|mac|win so it does not get packaged into the xpi files we deliver
for the installer builds.  Not much sense in doing it since the installer allows
people to choose what they want and will most likely have to generate a new
component.reg anyway.

Comment 30

[David Krause]

It makes sense that the these files wouldn't be useful to package for the
installer since as you said they would probably need to be regenerated depending
on the components picked.  But for the linux installer I would like to see a
dialog at the very end saying please run mozilla once with write permissions to
the install directory.  As someone said in another bug, the installer doesn't
really install since you still have to run once to finish the installation.
Most users think that the install is over after the installer gui finishes but
in reality it isn't.  The other option there would be to have the installer run
regxpcom and mozilla -CreateProfile automatically at the end of downloading the
packages.  This way the installer would really 'install.'

Comment 31

[Mike Shaver (:shaver emeritus)]

I thought the installer did the equivalent of running regxpcom after it was
done, as the same UID.  Is that not the case?

Comment 32

[David Krause]

Mike, nope.  I just downloaded the latest installer tarball.  I installed it
into /usr/local/ which is only writeable by root.  Tried to run as my user and
mozilla crashed while starting up.  No component.reg, no xpti*.dat, no
chrome/*.rdf and no chrome/overlayinfo/*.rdf files.

See also Additional Comments From wdormann 2000-09-06 08:14 in bug 51164.
"BUT, I go and try the Mozilla Installer.   (as root of course)    Which, I
would think should "Install Mozilla".     Not quite.    I immediately go to run
Mozilla as a regular user.   (being that running non-sysadmin tasks as root is a
big no-no in the *NIX world)       Sorry...   doesn't work.    Mozilla needs to
create component.reg in the mozilla directory and it can't do that."

Comment 33

[John Bandhauer]

I'm whacking the nsbeta3+ because I don't think any change is necessary at this 
point.

Comment 34

[John Bandhauer]

*** Bug 53802 has been marked as a duplicate of this bug. ***

Comment 35

[sairuh (rarely reading bugmail)]

any chance of fixing this for RTM?

Comment 36

[Jim Roskind]

I didn't understand the dogfood nomination.  Marking dogfood-minus.
Please renominate with an explanation if I'm missing something.
Is this really blocking users and developers in daily use?? ..or is this an RTM
deployment issuen (not dogfood)?

Comment 37

[Jim Roskind]

correcting my typo in status whiteboard.

Comment 38

[verah (gone)]

I'm having trouble putting this in the release notes. Is it related to bug 
46558? That one is already in the release notes ("Before using the PR3 Installer 
on Linux, you must have write permission for the target installation 
directory"). PLEASE ADVISE! Thanks.

Comment 39

[verah (gone)]

Copying myself.

Comment 40

[Gervase Markham [:gerv]]

Relnote-usering, assuming someone says what exactly needs saying (IYSWIM).

Gerv

Comment 41

[selmer (gone)]

If this bug cannot be reproduced using the stub installer, it should be rtm-.
The primary audience for N6 is not going to be using tarballs.

Comment 42

[selmer (gone)]

rtm-, no response, no patch.

Comment 43

[verah (gone)]

Removing myself from list of cc's.

Comment 44

[John Bandhauer]

cleaning up whiteboard and keywords - not dogfood

Comment 45

[John Bandhauer]

mass reassign of xpconnect bugs to [e-mail]

Comment 46

[Patrick C. Beard]

This bug is over 1 year old, with no solution imminent. Clearly a system
administrator will have to run either RegXPCOM or the browser once as part of
the installation procedure. Changing milestone to FUTURE.

Comment 47

[John Bandhauer]

Just an update...

The fix to bug 104191 included making the inability to write this typelib 
manifest file a non-fatal error. Of course, if it can not be written on first 
run then the app will have to load every xpt file on each and every run.

As discussed here before (and plenty elsewhere) we'd like to put this manifest 
elsewhere, but where? The location must be known before doing things like 
choosing prefs.

Comment 48

[timeless]

before prefs? that sucks.

hrm. [badidea#1] Can you stick the file in 
#TEMP#:hash(pathto(mozilla-bin)):xptitemp.dat

[badidea#2] i thought i saw brendan mention xpti merging, supposing that we 
could merge xpti.dat files, and that we could generate an xpti.dat file for 
each package (xpi) that we make, would generating an xpti.dat file for each 
.xpi and then packaging them into the xpi as #xpiname#.xpd work? i'm assuming 
(bad idea, i know) that merging (and enumerating) a few xpd files (one per xpi) 
would be faster than (enumerating and) registering all of the (xpt?) components 
(many per xpi), and this would give you the granularity for installer builds...

Comment 49

[John Bandhauer]

> #TEMP#:hash(pathto(mozilla-bin)):xptitemp.dat

xptitemp.dat is just the file the gets renamed to xpti.dat when we are safely
done writing it. I don't consider it to be a temp file and am not excited about
putting it in a temp directory. OTH, locating it in a directory whose name is
partially based on a hash of the full application path would fill some
requirements here. The same goes for components.reg.

There is not going to be any merging of multiple xpti.dat files. This file is
the manifest that captures the decisions we made about which type infos to trust
and use. Making such trust decisions based on what little we can glean for some
other xpti.dat file (rather than the actual details in the xpt files) would be a
mistake.

Comment 50

[Martin Mokrejs]

I don't aggree this bug is fixed as of 20031104. The xpti.dat file is being
opened in write mode. Mozilla doesn't care that it actually fails to open it in
write mode and happily continues. That's bad because it fails to start later on.

Second, I don't care on the first execution root user has to start mozilla so
that it can write xpti.dat file and possibly many other. We use AFS filesystem
here and root has exectly zero privileges.

The only proper fix is to either:

- avoid opening that file and maybe other in write mode as much as possible

- or mozilla should immediately die with message that it could not open
file for writing. It seems ti me that some forked subprocess really exits but
the parent process continues.

Comment 51

[timeless]

Tor, at least as of the time i'm writing this comment... this is a snipet from
xpti.dat:

[Directories,3]
0,/mnt/ibm/mozhack/obj-i686-pc-linux-gnu-qt/dist/bin/components
1,/usr/local/mozplugins
2,/mnt/ibm/mozhack/obj-i686-pc-linux-gnu-qt/dist/bin/plugins

I had someone test cp -rp -L and it worked (both xpcshell and mozilla) w/ a
readonly (this is a mozilla that was run before cp -rp -L ala comment 4)
components dir (and the source renamed so it wasn't reachable).

That said, i'd suggest we revisit my hashing suggestion.

A few modifications:
1. the hash is used for xpti.dat/compreg.dat instead of xptitemp.dat i'm sorry i
specified the filename earlier, that was a mistake.
2. the file(s) land near a home directory ($HOME/.mozilla/, %APPDATA%\Mozilla\,
$HOME/Library/Mozilla/) instead of near a temp directory. This means that each
user is penalized for each xpcom app they ever run instead of penalizing per
xpcom installation.

A variation would only do this if bindir/components/[xpti.dat|compreg.dat] don't
exist or aren't newer than the stuff they reference (to limit the penalty to
misconfigured xpcom installations).

I think it works like this:
$hashhost = "$ENV{APPDATA}/Mozilla/";
$hash = hash($pathtoxpcombin);
$parent = "$hashhost$hash";
if (!-d $parent)
mkdir $parent;
$xptidat = $parent."xpti.dat";
$compreg = $parent."compreg.dat";
$xpcomhost = $parent."xpcomapp.dat";

hash makes sure that the directory it returns is either unused or maps to the
same xpcombin as the one it's hashing (as indicated by the line in xpcomapp.dat).

Users who share a homedir but have different privs for the xpcom app are
unsupported. The cost is 16k for each dat file + 16k for the directory, so we're
talking about 64k or less (probably much less, but there are slack
considerations). At 2000 users and 10 xpcom bin apps this becomes a bit
expensive. The release notes instruct people to wipe out unused instances of
these directories.

I believe i can do this with a rather small amount of directoryservice hacking.
I'm certainly not going to do this until someone promises not to bite me.

lastly, martin: please don't comment in bugs, you clearly failed to read the
bug. Nowhere did the bug claim to be fixed, certainly the log
<http://bugzilla.mozilla.org/show_activity.cgi?id=39808> doesn't mention fixed
anywhere. further, the release notes are fairly clear about multiuser installs,
if you have complaints please don't use bugzilla for them, thank you.

Comment 52

[Benjamin Smedberg]

timeless: FYI, the *birds are planning on moving xpti/compreg into the profile,
something like ([profile.dir]/compreg-$buildID.dat  . This will allow extensions
installed into the profile to provide components without interfering with other
users/profiles.

We might want to consider the same sort of solution for seamonkey.