Closed Bug 359145 Opened 18 years ago Closed 11 years ago

Configurable path for urlclassifier2.sqlite

Categories

(Firefox :: Settings UI, enhancement)

x86
Windows XP
enhancement
Not set
normal

Tracking

()

RESOLVED WONTFIX

People

(Reporter: muehlfeld, Unassigned)

References

Details

(Whiteboard: wontfix?)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1) Gecko/20061010 Firefox/2.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1) Gecko/20061010 Firefox/2.0

I have the problem, that my Windows-User-Profile is limited to 30 MB. Now FireFox stores the urlclassifier2.sqlite there and brings me closer to that limit.

Would be nice to have a configuration option like "browser.bookmarks.file" to set a different path for the bookmarks file.

Reproducible: Always
Maybe we can store it in CSIDL_LOCAL_APPDATA (or the equivalent on other platforms), just like bug 74085 did for the cache. The disadvantage would be that the phishing list isn't stored in the roaming profile, so you might have to download the list over and over again, if you're roaming. But you already lost your cache anyway.
How does firefox get the new data for the sqlite file? Does he allways download the new file? Then it could be cached by proxies (if the user have one). But if only incremental updates are done, then it`s not a good idea to store it in CSIDL_LOCAL_APPDATA.

Isn`t it a better way to add a config option "browser.urlclassifier2.file"? Then the user can decide where to store. It must not be a option that could be configured through the gui. Maybe this can be a feature for later. But when the user can edit his preferences file or with about:config it would be a good start.
(In reply to comment #2)
> How does firefox get the new data for the sqlite file? Does he allways download
> the new file? Then it could be cached by proxies (if the user have one). But if
> only incremental updates are done, then it`s not a good idea to store it in
> CSIDL_LOCAL_APPDATA.
> 

Incremental changes. And it's not just a file, it's a database.


Then no change for proxy users to get the data out of the cache.

Is it much work to add a config option that is not shown in gui? Just replace the hardcoded path with a variable trough the complete source tree? Then I would try doing that.
Why is this Bug still listed as "Unconfirmed"? 

What is the problem with providing an about:config preference for re-locating urlclasifier2.sqlite, like can be done with bookmarks.html and the cache?
This database shouldn't be stored per-profile -- it causes profile bloat, especially with roaming Windows profiles.

Allowing it to have a configurable path would solve that problem as well, since network administrators could just release a pref change to all the machines, telling Firefox to store the DB on the local hard drive (or something).
If I understand correctly, that file has exactly the same role as the cache: it reduces the need to re-download data the browser has already downloaded before. So, shouldn't it be placed directly in the cache directory, instead of in the profile proper?
(In reply to comment #7)
> If I understand correctly, that file has exactly the same role as the cache: it
> reduces the need to re-download data the browser has already downloaded before.
> So, shouldn't it be placed directly in the cache directory, instead of in the
> profile proper?
> 

It already is. Firefox 3 files it under Local Settings (on Windows XP), next to the cache (and it's called urlclassifier3.sqlite). That's not included in your roaming profile, so it should be a lot faster.
Is it planed to port this feature to FireFox 2, too?
I agree.  Can we please have this ported to Firefox 2 under the next release of patches?
This seems to becoming more of a problem with FF3.  Personally mine is at 52MB right now and there are many threads out there with similar tales, asking about ways to reduce this file or to move it.  Users with limited-size profile folders or with profiles on USB keys are having more trouble with it than the average user who just has their profile on C:. 

So, can this be made an option on FF3 soon?  

Perhaps along with a way to shrink it, like in :
Bug 383031 – how to shrink the urlclassifier2.sqlite / urlclassifier3.sqlite file?
With urlclassifier.sqlite now sitting in "Local Settings", this problem is now less urgent (at least for me). However, "places.sqlite" is growing as well...
(In reply to comment #12)
> With urlclassifier.sqlite now sitting in "Local Settings", this problem is now
> less urgent (at least for me). However, "places.sqlite" is growing as well...
 
Perhaps I'm confused, do you mean in the nightlies?  Or am I just confused about what "Local settings" is, since I don't see any way to move the urlclassifier2/3 files and they're definitely being stored with the rest of my profile. 
On Windows, the urlclassifier3.sqlite file is stored in Local Settings\Application Data\Mozilla\Firefox (local hard disk), not in Application Data\Mozilla\Firefox (which might come over the network, when you're using a roaming profile). Both can be found in your profile (for instance: C:\Documents and Settings\username). But that's only import for large networks, it doesn't matter for home users.
No not the nightlies, just FF 3.0. 
The difference is that the "C:\Documents and Settings\username\Local Settings" folder is /not/ stored in your roaming profile, whereas "C:\Documents and Settings\username\Application Data" /is/.
Will want to configure paths on linux as well.  I still don't quite understand - is there any user specific information here or could this be in a central location and shared?
(In reply to comment #16)
> Will want to configure paths on linux as well.  I still don't quite understand
> - is there any user specific information here or could this be in a central
> location and shared?

UrlclassifierN.sqlite is Google database of malicious sites and should be stored in a central location. As far as I know there is no user specific information there. I think a good place to store it on Windows is \Documents and Settings\All Users\Application Data\Mozilla\Firefox.
(In reply to comment #17)
> (In reply to comment #16)
> > Will want to configure paths on linux as well.  I still don't quite understand
> > - is there any user specific information here or could this be in a central
> > location and shared?
> 
> UrlclassifierN.sqlite is Google database of malicious sites and should be
> stored in a central location. As far as I know there is no user specific
> information there. I think a good place to store it on Windows is \Documents
> and Settings\All Users\Application Data\Mozilla\Firefox.

But, user with no privileges, have not the rigth to write at this location, 
so its necessary to store it on the local profile
The Linux issue should be a separate bug.

> > UrlclassifierN.sqlite is Google database of malicious sites and should be
> > stored in a central location. As far as I know there is no user specific
> > information there. I think a good place to store it on Windows is \Documents
> > and Settings\All Users\Application Data\Mozilla\Firefox.
>
>But, user with no privileges, have not the rigth to write at this location, 
>so its necessary to store it on the local profile

There is an additional issue that the database is locked while Firefox is running. This would prevent multiple instances of Firefox using the same database. For example with fast user switching enabled on Windows a user leaving Firefox open would lock a centrally stored database preventing any other users of Firefox accessing the database.

With the moving of the database in Windows to a folder that is not stored on the network when roaming profiles are enabled this is likely a wontfix.
Whiteboard: wontfix?
(In reply to comment #18)
> (In reply to comment #17)
> > (In reply to comment #16)
> > > Will want to configure paths on linux as well.  I still don't quite understand
> > > - is there any user specific information here or could this be in a central
> > > location and shared?
> > 
> > UrlclassifierN.sqlite is Google database of malicious sites and should be
> > stored in a central location. As far as I know there is no user specific
> > information there. I think a good place to store it on Windows is \Documents
> > and Settings\All Users\Application Data\Mozilla\Firefox.
> 
> But, user with no privileges, have not the rigth to write at this location, 
> so its necessary to store it on the local profile

What about creating a subfolder in "\Program Files\Mozilla Firefox" with write access for the "Users" group for database storage?
(In reply to comment #19)
> The Linux issue should be a separate bug.
> 
> > > UrlclassifierN.sqlite is Google database of malicious sites and should be
> > > stored in a central location. As far as I know there is no user specific
> > > information there. I think a good place to store it on Windows is \Documents
> > > and Settings\All Users\Application Data\Mozilla\Firefox.
> >
> >But, user with no privileges, have not the rigth to write at this location, 
> >so its necessary to store it on the local profile
> 
> There is an additional issue that the database is locked while Firefox is
> running. This would prevent multiple instances of Firefox using the same
> database. For example with fast user switching enabled on Windows a user
> leaving Firefox open would lock a centrally stored database preventing any
> other users of Firefox accessing the database.
> 
> With the moving of the database in Windows to a folder that is not stored on
> the network when roaming profiles are enabled this is likely a wontfix.

The solution may be:

1. The first instance of Firefox has write access to the database, allows other programs reading it and is responsible for updating.
2. If Firefox determines during startup that another instance of it is run by different user then it has read access to it.
3. When database update time comes in the second version of Firefox then it checks again if it has write access to the database and if it does (which may come true if the first user closes his instance of Firefox) then it opens the database for writing, updates it and becomes responsible for future updates.
Blocks: 466173
I'm using Firefox in a college network I administrate.

The server is relatively small, and the student profiles are not allowed to grow.

By default, the safebrowsing was enabled but it makes the profiles explode. This file shouldn't be in user's profile as it as nothing to do with personnal data.

It should be a system wide file, on the computer or elsewhere.

The solution proposed by user01, with a lock, should be a good one.

Actually, i will be in a situation where I would recommend not to use firefox. It will be really ugly for me. Please, fix.....
Another suggestion would be to allow a symlink to the database file.

The symlink should be detected and instead of overwriting the link,
the linked file should be overwritten.

So instead of configuring the file path as a configuration option, it would be configured in the filesystem.  (Of course there are pros and cons to each approach.)

Unfortunately the status quo is, any such symlink is deleted and replaced by the whole database file each time the file is downloaded.
Putting this file in a shared place has security implications. Any user could add or remove data to it and hence cause false positives or negatives for any other user. A user getting infected can compromise the security of others users on the system.

Bug 673470 reduces this database from 40M to 6M. 

Based on this, this strongly looks like a WONTFIX to me.
Status: UNCONFIRMED → NEW
Ever confirmed: true
I agree with you having a share file is a security concern.

BUT.
File should be read only for normal users and read/write for admins.
In the collge I work, we have almost 1000 students. Each students has only 30 Mo of personnal storage (due too the very expensive SAS disks and teachers storage)
Having for each student even 6M out for just a single profile file is not acceptable.

Security concern is real, but should be managed. 

As of now, I had to suppress the use of the base. That is far less secure IMHO.
>File should be read only for normal users and read/write for admins.

This doesn't work because the database is updated very regularly (every 30-45min), and there is a hard requirement to update the database with the network lookups that happen when the user hits a site that is tagged in the database.
A commenter mentioned putting the file in a shared place, and somebody else squealed "security! -- WONTFIX!"

But that comment had nothing to do with the original description of the problem.

Nevermind sharing.  This is not about security, it is about a user's flexibility in managing their own file systems.

Please focus on the original report.
I too would would like to see a feature like the one being sugested.
Why don't change the place where this file to a simple about:config value that allows to put the file anywhere the user wants.

Then it can stay on it's origin place for the most users. But in networks with limited profile storage, the admin can put this file to any other (save) location (e. g. the users personal home drive).

I think it's much more a security problem, if there is no way to change the place where to store it, because many sites on the internet recommend to disable safe browsing when having problems with limited profiles (like the university of Koblenz - which was my first search result for "urlclassifier3.sqlite": http://www.uni-koblenz-landau.de/koblenz/GHRKO/faq/fileserver)./urlclassifier)

about:config is only for advanced users (as the message already says when you open it). And advanced users should of course always think about any changes they make.
Sorry. The correct URL in my previous post should be:
http://www.uni-koblenz-landau.de/koblenz/GHRKO/faq/fileserver/urlclassifier
Hi,

Change location to public and read/write have security problem. 

But, it's not that bug's reporters and watchers wants. We want to have possibility to change location to another.

Why ?

Because when we have 20000-30000 profiles, and propose a little network space to users to store profiles (Firefo, thunderbird, LibreOffice etc.) and documents we want to optimize profile size and have two choise :

- store profile in network space to permit users to retrieve profile when changing computers (university, schools etc.)

- store profile locally but users does not retrieve profile when changing computers.

This file is the perfect candidate to the second way. We have no problem with bandwidth and have space in local disk of computers.

Just permit to change location and we script dynamic location to set location to local profile space of the user. If you want to be my God, add option to store this file in a dynamic location with possibility to use system user's id in the given path :)

like /home/%user%/localspace or "C:\Application Data\%user%\path\I\Want\"

Thanks :)
Here only 5Mb for roaming profile for more than 500 users and it gets corrupted each time someone updates/reinstall firefox because of urlclassifier.

This is one of the most recurring support request to our helpdesk.

Why urlclassifier path can't be configured as for the cache directory?
I don't understand comment 28 to 31 at all. 

They're all talking about being able to move the file to a local disk because the Roaming user profiles are limited in space. 

But the file isn't stored in the Roaming profile any more, instead it is in the Local AppData. The Local AppData should *not* be stored on a remote server or in a quota-ed profile, but on the local disk of the PC.

This was already explained in comment 14.

I could imagine that on Unix environments there is still an issue (I agree with comment 23 there), but those comments seem to be talking about Windows. As far as I can tell, we already fixed this issue and those comments are asking for a workaround for a problem that no longer exists.
False.

App Data directory IS configurable through registry or GPO (all the same), and it can be on server side for various reasons.

That's the case in a college I manage.
>App Data directory IS configurable through registry or GPO (all the same), and it 
>can be on server side for various reasons.

Ok, thanks for the clarification. The distinction between Local and Roaming exists exactly to avoid the issues people are complaining about here, so this sounds a bit like shooting yourself in the foot to me, but I admit not knowing much about large-scale Windows administration so there might be reasons for doing this that I'm not aware of.

Bug 239254 is marked [Linux] but that is probably because it was believed that the problem no longer existed on Windows (explanation in comment 32 here). It should address the issue reported here if it works on Windows. Let's confirm that and if so make this bug depend on 239254.

From what I understand, making this path configurable with about:config doesn't work because those settings are actually in the Roaming profile, so you could end up loading a config that points to a directory that doesn't exist on your local system when roaming.
@Gian-Carlo Pascutto: You're right. It's moved meanwhile. And the local AppData, we don't include in the roaming profile here. For my network this OK, too, even if it's not configurable. 

@Swirly: Changing AppData completely to a different location is not possible here, because if it is on a server share, many programs, that store their data in AppData, can't be used on multiple computers at the same time by the same user.
@Gian-Carlo Pascutto: Sorry for the noise, you're right. They are actually in local appdata and we were keeping that on roaming profile for other reasons (other software that uses local appdata to store configurations that should be in appdata). In my case changing local appdata to stay on local drive as it should and finding workarounds for the other software seems to be the best solution.
Bug 673470 replaced SQLite usage for this file with an optimized store.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.