46320 - Reduce global files and resources to allow peaceful installations of gecko.

Status: RESOLVED INCOMPLETE

(Firefox Build System :: General, defect)

Description

[Doug Turner (:dougt)]

Reduce global files and resources to allow peaceful installations of gecko.


There are two files that stand out, mozvers.dat and mozreg.dat.

The first, mozvers.dat is used for version tracking of files that were installed
by xpinstall.  Although probably not an immediate need of embedders, we should
address this if nothing else to than to understand the issues.

Only a small fraction of this file needs to be system wide - specifically the
parts that reference globally installed files.  I suggest that we factor this
part out and move the rest into the application directory.

The second file, the global registry, needs more thought.  It is a dumping
ground for prefernces and other tidbits of information that "need" to be system
wide.

I would like to see this file in the application directory.  For Seamonkey, this
is probably not what they want to have happen.  We need a discussion about this.

Comment 1

[Doug Turner (:dougt)]

Attachment: Make app registry relative to app.

Comment 2

[Doug Turner (:dougt)]

now the only offender is xpinstall.  It directly calls through to the NS_ 
functions.  

Comment 3

[Daniel Veditz [:dveditz]]

Your patch for the "application registry" won't work, this registry is 
specifically supposed to NOT live in the application directory. Or at least, it 
needs to live somewhere the app will always have write access, unlike 
component.reg which only needs to be updated at install time when presumably we 
*do* have write access.

In the embedding case do you even need the application 
registry? An embedded Gecko isn't the mozilla application, so the Gecko bits 
probably shouldn't even be touching that one. Maybe you can set a pref or some 
other way simply turn it off in the embedding case.

(note that Mozilla has problems with mozregistry.dat being in the windows dir 
that already need to be dealt with, particularly on Win2K and to a lesser 
extent NT. But the file still needs to be writable by the application after 
install time which we're pretty guaranteed won't be true of the application 
directory.)

For mozver.dat -- yes we should split it into global and application sections, 
as we did in Communicator on Unix systems. But should embedded Gecko even ship 
with the xpinstall component? There are lots of tricky issues there. If it's a 
requirement we're going to have to get together and really think this out.

Comment 4

[Doug Turner (:dougt)]

Embedding gecko has a wide scope.  There are people that will not need 
profiles, and there are others that might want to use what mozilla does.  The 
bug also applies to mozilla and netscape 6 - we need to minimize global files.  
Also, with some work, xpinstall could be embedded.  I can see some customers 
wanting a way to update there product remotely, etc.


I do not see why we need mozregistry.  Please tell me why it is global.

What is in this file that needs to be global.  What I see is:

Profiles - arguably not global.
Scheduled file stuff - should be in the version registry (rename it the xpireg, 
or something).

Your first comment is confusing.  How can the mozregistry file "NOT live in the 
application directory" and "live somewhere the app will always have write 
access"  There really is not a place on any machine where this is true.  At 
least if we put it in the application directory we have reasonable assurance 
that other developers/application will know that it is owned by another 
application.... matter of course.


What doesn't work with the patch?

Comment 5

[Doug Turner (:dougt)]

spoke to dan.

Here is what we agreed on.  the Application registry should be put into the HOME 
directory on all platforms (or what ever is the closes on the mac).  The file 
name should be the product name.  This should get us out of most of the trouble

XPInstall's registry should be split just like it is on unix for all platforms.

Comment 6

[Judson Valeski]

+

Comment 7

[Doug Turner (:dougt)]

This will break mozilla/netscape6 profiles.  

Profile migration of 3-4x profiles will still work, but if you have a netscape6 
beta1 profile or a mozilla profile, it will not be seen after this fix.

If we do this, it should be before netscape ships beta2.  

Comment 8

[Daniel Veditz [:dveditz]]

Then you should nominate it for nsbeta2 or it won't get looked at. But I'd give 
you a snowball's chance getting approval at this point.

Comment 9

[leger]

Putting on nsbeta2 nominee radr for review, and adding relnote2 keyword in case 
that's the way to go here.

Comment 10

[Judson Valeski]

cc'ing conrad for mac's equivelent of the TEMP dir.

so if we rename the appregistry file to reflect the app name, netscape6 would 
have it's own file in the TEMP dir, and so would joesEmbeddingApp that is using 
it's own distribution of mozilla (an embedding dist). What if joesEmbeddingApp 
wants to use the netscape6 bits (and subsequently netscape6's resgistry file)? 
Are we still ok here?

Say we do ship w/ an embedded xpinstall component (not a requirement at this 
stage), could this ver.dat file be named to reflect the embedding app name as 
well?

If we do all of the above, it seems to me that you could have an arbitrary 
number of SeaMonkey "installations" on a system. If that's true, then the 
outstanding question would be the one of installation sharing.

Comment 11

[Judson Valeski]

if we're going to do this, doug is right, we need to do so now (for beta2) so we 
break fewer installations in the future.

Comment 12

[Doug Turner (:dougt)]

There is alot involved here. 

First, I am suggested that the mozregistry be moved into the users HOME 
directory.  It should also be renamed to <ProductName>.dat.  What this means is 
that every product based on gecko will have their own product registry.  There 
would be no explict globally shared files (save xpinstall which we will get to 
in a sec).  This will effect the profiles folks.  After this change the Profile 
Migrator will see Netscape 4 (3?) profiles and be able to convert them.  
However, I doubt that it will see a Beta1 profile or a mozilla profile.  This is 
because it will stop looking out in the global location.

XPInstall is a different animal.  It installs files both system wide and 
application specific.  I suggested to dan and he has had the same idea to split 
the xpinstall registry up just like it is on unix.  There would be a "system 
wide" registry as well as an application (gecko embedding) specific registry.



Now, if we don't want to deal with this now (snowballs chance), at the very 
minimun we should force the profile code to save to a different file - 
mozprofiles.dat or something and have the profile code pass the full path to 
this file to nsRegistry.  This would at least minimize the users of this file 
sugnificately.

Comment 13

[Conrad Carlen]

The mozregistry file on the Mac is stored in the system preferences folder. Thus, 
it is shared by seamonkey and any embedded app. I think this file should be 
located using the nsIDirectoryService. This would allow embedding clients to have 
their own private version of this file (likely case) or they could use ns6's 
registry if they did not provide their own file location. Right now, the location 
of this file is determined deep down within the reg lib and is not easily 
changeable. This, I think, is the problem.

Comment 14

[leger]

[nsbeta2-], please wait until after branch to check this into the trunk :-)

Comment 15

[Doug Turner (:dougt)]

does PDT understand that this will break profiles between beta2 and beta3?

Comment 16

[Doug Turner (:dougt)]

Need fix. The one attached does not do the "<productName>.dat and save in the 
home dir" stuff.

Comment 17

[racham]

FYI : Profile Registry (mozregistry.dat) today contains only profile
information. It used to contain components information. When component.reg is
introduced, mozregistry.dat is left with only profile related information in it.
We can rename the file to be profile specific (profile.dat or something) unless
we have plans to store other information in that file. Also, please correct me
if we any other component is using this registry file to store any other
information...?

As mentioned by doug placing the mozregistry.dat somewhere else will certainly
break the transition from earlier beta versions. I guess we can handle that in
profilemanager code with the help of versioning system. I can help in that
regard. Migration from 4x will not have any problems as we make fresh profile
entries into the registry in that case.

Comment 18

[Michael La Guardia]

cc'ing kevin murray.  PM for profiles.

Comment 19

[Jay Patel [:jay]]

Putting on [nsbeta2-] radar. Not critical to beta2. Yes, we 
(PDT) understand...chofmann says, "lets do the profile migration fix."

Comment 20

[selmer (gone)]

I disagree with Doug's claim that profile information is not global.  There is a
choice here and it has large implications.  We need to discuss this before you
check in the fix.

Comment 21

[Doug Turner (:dougt)]

I claim that profile information *should not* be global, and I also agree that
there needs to be a discussion.  Why do you think that profile's should be global?

Just to reiterate my point of view.  Mozilla is going to be embedded into many
different applications.  Some of them will want to use profiles.  This profile
information is specific to the application that created the profile - not
global.  If mozilla and netscape want to share profiles, that is great, but it
should be done explictly.  We have some customers that do not want to see
mozilla profiles.  I believe that the default case should be so that the
application registry is localized to the application.

Another sinister problem with a global registry is that there is not file
locking.  Two appplication which share this same registry file will stomp each
other.

Comment 22

[selmer (gone)]

The ability to share profiles would help define a "family" of applications that
cooperate closely.  If all profiles are required to be separate, this would be
precluded.  Some reasons to care are bookmarks, certificates and security,
cookies, wallet, possibly some prefs (such as proxy info), etc.  I would expect
sharing of this information as a significant indication that the "family" really
was one.

Allowing for this type of sharing means we don't have to build every single
interesting thing directly into Seamonkey.  Complementary products could provide
added value to the customer by sharing profiles.  (I'm not minimizing the
difficulty of the locking problem, just explaining the potential benefit.)

Even if we don't think we want sharing right now, allowing for the possibility
is prudent.  One way to do that is to allow the application to specify what kind
of profile space it wants to work in and then set the default to "private".
This leaves us the ability to define other types of profile sharing in the future.

There were also a lot of issues for profiles with the out-of-process PSM module.
 They also had reasons to share and reasons to be separate.  How did they react
to this proposed change?

Comment 23

[Doug Turner (:dougt)]

There is no reason why these family of components can not know about a global 
location.  I want to make sure that Navigator and Mailnews use my same 
perferences, however, I don't want my embedding customers to have to bother 
filtering profile and other information spammed into a general registry.  If you 
want to share files between two application built on gecko, go for it.  They 
just have to know where to look at compilation time.

Basically, my changes make xpcom not have a default global registry location.  
If you try to open up the default application registry, you will get just that: 
a registry that is specific to the application/product - not a global file.  Two 
application could say that they are the "same" product and share this registry 
location. 

Also, you have not address the concurrency problems.  How will you deal with two 
applications sharing the same files?  

How does this pertain to PSM?  They still will be passed a file location to the 
current profile.  They do not care where the files exist.

Comment 24

[racham]

Fixed.

Application registry is now moved to HOME. User's profile information will be 
transfered to this new registry and a flag is set inside the registry indicating 
that on subsequent launches no data transfer is needed. Each product that is 
goign to use Mozilla can have a it's own registry as the product name is used to 
derive the path for new registry.

Comment 25

[Decklin Foster]

I just tried out this change (linux CVS 081619). Two directories were created in
my home dir with names consisting of 3 chars random garbage and then a '@', both
containing an 'appreg' file like the one in ~/.mozilla. After deleting them I
did not see them again, but we should make sure people don't see this upon
upgrading to M18/beta3. Tenatively reopening, lets see if someone can reproduce
this. *braces himself for a bunch of dupes*

Comment 26

[racham]

clobber and build your xpfe folder. Looks like FileLocator is passing garbage as 
product name to the XPCOM init. 

Fresh builds should not see this problem.
We will wait until for QA to say to test this.

Comment 27

[kmurray1115]

Adding tpringle (new PM for Profiles) to cc.

Comment 28

[Mike Kaply [:mkaply]]

After this change, is modules/libreg still used?

Because:

mozilla/ modules/ libreg/ src/ vr_stubs.c 

Looks for the mozregistry.dat in different places than the new profile code.

Comment 29

[Doug Turner (:dougt)]

dveditz is going to have to fix that.  It is for the xpinstaller.  It still has
a global file.  Assigning to dveditz.

Comment 30

[Daniel Veditz [:dveditz]]

Libreg is still used heavily, it just shouldn't be supplying a default 
filename.

Comment 31

[Doug Turner (:dougt)]

right, libreg is used under the sheets.



Dan, can you have xpinstall pass the full path to the version registry?

Comment 32

[Daniel Veditz [:dveditz]]

From an architectural POV the "version registry" layer should be moved out of 
libreg and into the xpinstall component, but it's pretty intertwined at the 
moment and will have to wait for a future release.

A quick fix of passing in the filename I think I can deal with now.

Comment 33

[racham]

dan,

Are we moving mozver.dat also under HOME/<productName> (where the registry.dat
lives today...) ?

If you need to keep any of the scheduled tasks information alive, you can do the
transfer into registry.dat just like profiles...right ?

bhuvan

Comment 34

[John Bandhauer]

Someone please remind me exactly what is in registry.dat.

This is application data?

We also have xpcom specific application data in component.reg and xpti.dat. Both 
of these files are potentially written at runtime. I have bug 39808 because 
xpti.dat is written in the same directory where the component DLLs live and on 
some systems this can not be done when running under the lowly user account. I 
would expect that there is a simlar bug on component.reg (but I can't find it). 
Failing to write xpti.dat currrently is a fatal error. I believe that failure to 
write component.reg just makes the app re-autoreg on every startup (a subtle 
perf hit).

Anyway, help me out... should these two xpcom specific files get written to the 
same place as this registry.dat?

Comment 35

[Daniel Veditz [:dveditz]]

At one point at least, if not still, failing to write to component.reg when 
there wasn't one at all was a crash. After install time, though, XPCOM deals 
with a read-only component.reg just fine (after several bugs were fixed on the 
issue).

If all the writing you're ever going to want to do can be done at "install" 
time (when we presumably had permission to write out the new components) then 
you should be fine.

Comment 36

[Daniel Veditz [:dveditz]]

*** Bug 49464 has been marked as a duplicate of this bug. ***

Comment 37

[John Bandhauer]

racham,

The change to -r1.31 of nsSpecialSystemDirectory.cpp under line 587 is a little 
broken.

http://lxr.mozilla.org/seamonkey/source/xpcom/io/nsSpecialSystemDirectory.cpp#58
7

This appears to cause a UMR in Purify. GetEnvironmentVariable is documented to 
return 0 if the var was not found in the environment. This leaves you calling 
PL_strlen on an uninitialied buffer. It looks like you cloned or modified 
mcmullen's code.

I don't know why it is using ">= 0". It looks like '> 0' is appropriate since it 
looks like you only want it to go this way if "HOME" is set.

It is not clear to me if the case below that where it checks for "HOMEDRIVE" 
ought to be '>= 0' or '> 0'. 

If '>= 0' *is* appropriate, then you should do "path[0] = 0;" before the call to 
GetEnvironmentVariable to ensure that the buffer is safely zero terminated 
first.

Note that release builds will leave garbage in auto variables, while debug 
builds generally initialize them to 0.

Comment 38

[Doug Turner (:dougt)]

jband, file a seperate bug against me.  

Comment 39

[John Bandhauer]

I'm not sure why dougt wanted a separate bug for the problem with the fix to 
this bug. He did file one (bug 49866). The point of my entry was that the fix to 
this bug is broken. This bug has [nsbeta3+]. On a release build on a machine 
with no "HOME" setting this can CRASH. Or you may endup up with mozilla trying 
to read and write to a directory whose name is garbage. Let's just fix this. OK?

Comment 40

[Daniel Veditz [:dveditz]]

Reassigning to racham to fix jband's concerns. 

Also, wouldn't it be better to use Win_AppData rather than an environment 
variable anyway?

Comment 41

[racham]

I am fighting with some dogfood bugs. Back to dougt (bug 49866 should probably be 

closed, as this bug already has required and nsbeta3+ status).



Dan, we certainly need a small discussion on this location issue (for profile 

registry and user profile directories) as lot of people started suggesting other 

locations (mainly AppData...what about windows 95, 98 ..etc). If you have anyone 

else who need to involved in the discussion, we hsould setup a formal 

meeting..otherwise, I can come dows to your cube and we can talk abou;it. Bug 646 

is going to be resolved soon when I find out the favorite location for Users50 on 

windows platform.

Comment 42

[racham]

correction for the previous update...it is not bug 646 that talks about users50 
location issue, it is 6464. sorry.

Comment 43

[Doug Turner (:dougt)]

I fixed the bug jband pointed out. 

The only remaing global shared file I know about is the xpinstall version
registry.  Assigning to dveditz.

Comment 44

[Daniel Veditz [:dveditz]]

Per PDT rules P3 bugs are now nsbeta3-

Comment 45

[Daniel Veditz [:dveditz]]

Resetting missed milestones

Comment 46

[Daniel Veditz [:dveditz]]

nominating nsbeta1

Comment 47

[Brian King [:kinger]]

I have noticed that mozregistry.dat / mozvers.dat / registry are no longer used.
Have these been replaced or are they no longer needed?

Comment 48

[timeless]

afaik we still use mozregistry.dat

Comment 49

[Daniel Veditz [:dveditz]]

And mozver.dat, though only by XPInstall so you may not notice any change for 
long stretches of time.

Comment 50

[Brian King [:kinger]]

They used to be defined in profile/src/nsProfileAccess.cpp
Now it appears only in modules/libreg/src/vr_stubs.c
dveditz, did you re-factor this code? Basically I'm looking on what files to 
patch if I wanted to remane these files.

What are the long term plans for these files (if any)?

Comment 51

[Daniel Veditz [:dveditz]]

the default registry (mozregistry.dat/registry.dat/Mozilla Registry) is going 
away after dbragg's "post install replace" feature is checked in.

mozver.dat will still be used, but only by XPInstall. Currently the name is 
defined in the file you mentioned and could be changed there. Ultimately it 
should probably be passed in from the client code rather than in libreg itself.

Comment 52

[Judson Valeski]

can we get a target milestone on this?

Comment 53

[Doug Turner (:dougt)]

dan is on sabbatical.  

Comment 54

[Jeet Shahani]

I am using WebClient and Embedding API's in BrowserG! application.  Does bug 
46320 cause this profile bug- http://mozdev.org/bugs/show_bug.cgi?id=676 ?  
Please advise, what is going wrong?  Thanks.

Comment 55

[Doug Turner (:dougt)]

no one has touched this bug in 8 years.

Comment 56

[Doug Turner (:dougt)]

incomplete is probably a better resolution.