1263209 - Site specific permissions are not persisted across browser sessions

Status: RESOLVED INVALID

(Core :: Permission Manager, defect)

Description

[Michael Kraft [:morac]]

I've noticed as of Firefox 45 (and reproducible in the nightly 48 loads), that any time I exit and re-run Firefox, it will ask me for permission for things on sites where I've already granted permission to do that action.  I first noticed this with the Java plugin, but it also affects other items like granting popup permissions and the like.  I checked the permissions.sql file and the site is in the file and permission is granted to java.

The permission seems to get written to the permissions.sql file (if that file exists, see below), but when Firefox is quit and started again, it won't be used.

It looks like permissions are not read from disk when Firefox starts.  If I give permissions they will apply until the browser is exited and started again. There is an exception to this.  If the browser version changes (up or down), then permissions are read back in.  

Here's the steps to recreate:

1. Go to https://www.java.com/en/download/installed8.jsp and click "Verify Java Version".
2. When asked for permission, choose allow and remember.
3. Quit Firefox.
4. Load Firefox again and repeat step #1.

Expected Result:
1. Java Applet should load.

Actual Result: 
1. Need to give permission again.


There's a few unusual things I've found when researching this:

1. If the permissions.sql file is deleted, it won't be recreated until the browser version changes.
2. After permission is given, if the "i" is clicked, it will say there is no site specific permission, though if the "page info" window is opened it will show that Java is set to allow.  After the browser is exiting and started again, it will revert back to "ask".
3. If switching between Firefox versions the permissions actually are used.  This can be seen easily by repeating the steps to reproduce above, but on step 4 run a different version of Firefox.

So basically the problems are:
1. permissions.sql file won't be created unless browser version changes
2. permissions not read from permissions.sql file when browser starts up unless browser version changes, so permissions are lost.

I reverted back to Firefox 44.0.2 and the problem does not exist there, so it's a change that was added in Firefox 45.

Comment 1

[Michael Kraft [:morac]]

Just to confirm my suspicions.  I installed Process Monitor (https://technet.microsoft.com/en-us/sysinternals/processmonitor) and added a filter for any disk access where the path contained "permissions.sql).

I ran Firefox 44.0.2 to make sure the file was created.  I then ran Firefox 45.  The first time I ran Firefox 45 I saw that the file was read in.  I then quit and ran Firefox 45 again and again it was read in.  I then made a change in permissions and saw that it wrote out the the permissions.sql file.   The next time I ran Firefox 45, it did not read or write from the permissions.sql file.

I repeated the above without making any changed to permissions and discovered that after switching to any version of Firefox 45 or up, the permissions.sql file will only be read or written to the first and second time that Firefox is launched after updating.  Any subsequent times Firefox is run, the permissions.sql file is never read or written to.  That means existing permissions aren't read and new ones aren't stored.

So basically:
1. Run Firefox 44.0.2 or lower - permissions.sql file used.
2. Run Firefox 45 or later - permissions.sql file used.
3. Run Firefox 45 or later - permissions.sql file used.
4. Run Firefox 45 or later - permissions.sql file NOT used.
5. Run Firefox 45 or later - permissions.sql file NOT used.
6. Run different version of Firefox 45 or later - permissions.sql file used.
7. Run same version of Firefox 45 or later as #6 - permissions.sql file used.
8. Run same version of Firefox 45 or later as #6 - permissions.sql file NOT used.
....

Basically when running Firefox 45 or higher, persisted permissions stop working after Firefox is run twice, until the version changes again.

Comment 2

[Michael Kraft [:morac]]

Another simpler step to reproduce that doesn't require the Java Plugin.

1. Go to a site that requests the ability to display desktop notifications (for example GMail or https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API)
2. Allow site to always send notifications.
3. Close Firefox and re-open.
4. Go back to site from #1. 

Expected result:
1. Site should still have permission to display notifications.

Actual result:
1. Site does not have permission to display notifications.


Additional information:
1. permissions.sqlite file was never updated with new site info after closing Firefox.

Comment 3

[Paul Pasca[Away. Please needinfo? Paul Oiegas [:pauloiegasSV]]]

Hi Michel,

I have tested your issue on latest FF release (45.0.2) and latest Nightly build and could not reproduce it. I have opened both links you provided in comment 0 and comment 2, but after I select allow and remember option for permissions and closed the browser, I wasn't asked to allow permissions again on reopening Firefox. This issue occurs of you select allow now option, but that's the expected behavior. I've also tested on Firefox 45.0 but I was not able to reproduce it.

Is this still reproducible on your end ? If yes, can you please retest this using latest FF release and latest Nightly build (https://nightly.mozilla.org/) and report back the results ? When doing this, please use a new clean Firefox profile, maybe even safe mode, to eliminate custom settings as a possible cause (https://goo.gl/PNe90E).

Thanks,
Paul.

Comment 4

[Michael Kraft [:morac]]

I've tested both in safe mode with a new profile in Firefox 45.0.2 and with the latest nightly build (32 bit and 64 bit) and I've reproduced the problem in both running of the 64 bit version of Windows 7.  Except for the first time after switching Firefox versions, the permissions.sql file isn't read or written to when running Firefox, so existing permissions aren't used and new permissions set aren't stored. On a newly created profile, the permissions.sql file isn't even created, nor is it even attempted to be created (confirmed with process manager). 

When I revert back to Firefox 44, the problem goes away.

Comment 5

[therube]

I cannot duplicate the non-creation of permissions.sqlite.

Otherwise, & not that I know how things are supposed to work, appears that there are some issues here?

(Just quoting one of my comments from, http://forums.mozillazine.org/viewtopic.php?f=9&t=2999261)

If we start fresh, & visit jsfiddle...

Load http://jsfiddle.net/yoshi6jp/Umc9A/
Click the 'Request Permissions' button
Select 'Always Block Notifications' from the dropdown dialog

At that point, note that the affected address is, http://fiddle.jshell.net (& is so stated in the dropdown that you just selected).

Click the (whatever you want to call it) to the left of the URL.
It states (correctly, presumably) "You have not granted this site any special permissions".

And that is correct.
Because "this site" is "jsfiddle.net" & not "jshell.net" - which is where the permission that you did grant, from the dropdown, dealt with.

Likewise, when you go to Page Info, you are again dealing with "this site", which is again jsfiddle.net & not "jshell.net".

So the "request" (if you will) comes from one address, & in this case, once it is written, there is no way to alter it.

When you use Page Info, you are "on the page" & so from the GUI, you can affect Permission changes for "this page", but "this page" is now different from the prompt you responded to initially.

Comment 6

[Paul Pasca[Away. Please needinfo? Paul Oiegas [:pauloiegasSV]]]

Hi Michael,

Given the fact that the issue seems to be related to something that is specific to your setup, could you please try to find a regression range using Mozregression tool?
Information on the tool is available at http://mozilla.github.io/mozregression/. Please don't hesitate to contact us if you encounter any problems.

Thanks,
Paul.

Comment 7

[Michael Kraft [:morac]]

Okay using that tool (which wasn't as simple as it seems since I need to run Firefox 3 times before the the permission.sql file stops being read/written to when cloning a profile), I narrowed the range down to:

Good: 20160210052223
Bad:  20160210024420

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2dfb45d74f42d2a0010696f5fd47c7a7da94cedb&tochange=7042e8a19f94d6e075ec149567aea74dfd06c392


None of the bugs looked like they had anything to do with permissions or persistance, though I think the bug is reading/writing the permissions.sql file so it may not be labeled as such.


20160210052223 seems to be a Firefox 47 build, but I'm seeing the problem with Firefox 45.0.2, so it would have to be a bug fix in that list that also landed in Firefox 45.

I went through the list and came up with the following bug fixes that landed in Firefox 45 based on the status-firefox45 flag (which may be wrong):

Bug 1227344
Bug 1233238
Bug 1237160 
Bug 1241558
Bug 1243750
Bug 1245212
Bug 1245745 
Bug 1246209
Bug 1246778 
Bug 1246850 

These bugs were access restricted though so they may have went into Firefox 45:

bug 1173229
bug 1228103
bug 1238558
bug 1243555
bug 1244248
bug 1245724
bug 1246054

Comment 8

[Paul Pasca[Away. Please needinfo? Paul Oiegas [:pauloiegasSV]]]

Hi Michael,

I will remove the "regressionwindow-wanted" keyword since you provided a push log and the good and bad builds.

Thanks,
Paul.

Comment 9

[Michael Kraft [:morac]]

Now that there is a known range for the regression, is it possible to start looking into why this is happening?  This is biting me on a daily basis as I have to keep re-allowing a lot of permissions that I set on a daily basis since I block Flash by default.

Comment 10

[Michael Kraft [:morac]]

More recent versions of the browser still do this, but not all the time.  I'd say about half the time it remembers permissions and half the time it doesn't.

Comment 11

[Cédric Dufour]

I experience exactly the same from with FF 45.2.0 ESR (Debian/Jessie).

I strace-d (-e trace=file) the main process and can see plently of *.sqlite files accesses but NONE to permissions.sqlite.

I added some LogToConsole statements in nsPermissionManager.cpp, including in the OpenDatabase function. The only see it allows me to see is a "profile-before-change" triggered event and the corresponding "CloseDB" function call when Firefox starts. NOTHING ELSE afterwards. It is as if the Permission Manager ended up orphaned. No amount of (cookie, popup, etc.) exceptions configuration triggers anything (except the exceptions remain in memory until FF is closed).

This is very annoying as it prevent popup exceptions to be configured, and breaks all Cookie Management add-ons that rely on the permission manager (e.g. Self-Destructing Cookies).

I can reproduce it systematically, even with a clean profile.

Comment 12

[Cédric Dufour]

ADDENDUM:

When I entirely cleanup the ~/.mozilla/firefox directory (and a new profile is therefor created automatically), I get (thanks to the added 'LogToConsole' statements; hope they are self-explanatory enough):

DEBUG: nsPermissionManager::Init
DEBUG: nsPermissionManager::InitDB
DEBUG: nsPermissionManager::InitDB; NS_GetSpecialDirectory(NS_APP_PERMISSION_PARENT_DIR) failed
[JavaScript Error: "NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIProperties.get]" {file: "resource://gre/components/XULStore.js" line: 66}]
DEBUG: nsPermissionManager::Observe
DEBUG: nsPermissionManager::Observe; Doing profile-do-change
DEBUG: nsPermissionManager::InitDB
DEBUG: nsPermissionManager::InitDB; NS_GetSpecialDirectory(NS_APP_PERMISSION_PARENT_DIR) failed
DEBUG: nsPermissionManager::OpenDatabase
DEBUG: nsPermissionManager::OpenDatabase; Opening file-based database...
[JavaScript Error: "uncaught exception: 2147500036"]
DEBUG: nsPermissionManager::Observe
DEBUG: nsPermissionManager::Observe; profile-before-change
DEBUG: nsPermissionManager::CloseDB

(a 'permissions.sqlite' file is created <-> nsPermissionManager::OpenDatabase; Opening file-based database)


When I do nothing else that close and re-open FF, using that freshly created profile, I get:

DEBUG: nsPermissionManager::Init
DEBUG: nsPermissionManager::InitDB
DEBUG: nsPermissionManager::InitDB; NS_GetSpecialDirectory(NS_APP_PERMISSION_PARENT_DIR) failed
DEBUG: nsPermissionManager::OpenDatabase
DEBUG: nsPermissionManager::OpenDatabase; do_GetService(MOZ_STORAGE_SERVICE_CONTRACTID) failed
[JavaScript Error: "uncaught exception: 2147500036"]
[JavaScript Error: "NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIIdleService.removeIdleObserver]" {file: "resource://app/components/nsBrowserGlue.js" line: 1717}]
DEBUG: nsPermissionManager::Observe
DEBUG: nsPermissionManager::Observe; profile-before-change
DEBUG: nsPermissionManager::CloseDB

Please note the "nsPermissionManager::OpenDatabase; do_GetService(MOZ_STORAGE_SERVICE_CONTRACTID) failed", which makes nsPermissionManager::OpenDatabase exit prematurely.

I tried:
 - closing FF; removing all *.sqlite files; open FF -> same behavior
 - closing FF; removing prefs.js file; open FF -> same behavior
 - closing FF; removing all files in the profile; open FF -> same behavior

The only operation that allows to get out of that behavior is wiping the entire profile (directory and 'profiles.ini' file)

Strangely, it is not the only bug that has to do with strange behavior of the Storage Service; see https://bugzilla.mozilla.org/show_bug.cgi?id=1277295 (which proposed patch I had to apply to allow installation of add-ons that require restart)

Any idea where to look from there ?

Comment 13

[Cédric Dufour]

Just a hunch: CVE-2016-2829 <-> (restricted) bug 1248329 ?

Comment 14

[Michael Kraft [:morac]]

(In reply to Cédric Dufour from comment #13)
> Just a hunch: CVE-2016-2829 <-> (restricted) bug 1248329 ?

Was that fixed in Firefox 45?  It doesn't appear in the list I found in comment 7.

Comment 15

[Cédric Dufour]

I have a feeling that there is a race condition somewhere during initialization.
I'm not familiar at all with FF code base, so I have no idea where to look at.
No one from dev can share a clue on where something might be wrong ?

Comment 16

[Michael Kraft [:morac]]

(In reply to Cédric Dufour from comment #15)
> I have a feeling that there is a race condition somewhere during
> initialization.
> I'm not familiar at all with FF code base, so I have no idea where to look
> at.
> No one from dev can share a clue on where something might be wrong ?

A race condition might explain why I don't see this problem after upgrading since there's more processing being done at start up.  It also might explain why others don't see it.

Currently I get hit by this problem on a near daily basis.  It really is annoying having to re-approve permissions every day for sites I've given permission to.

Comment 17

[Michael Kraft [:morac]]

Any movement on this?  It's actually gotten worse in that it's now happening even after updates.

For example in the nightly 64-bit windows build (20160727030230), if I open 
https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API it asks me to allow notifications.  If I choose always remember, then close Firefox and open it again and go back to the same web site, it asks me if I want to allow notifications again.  It does that every time I close and reopen Firefox.

Comment 18

[Michael Kraft [:morac]]

Any progress on this.  It's still an issue in Firefox 50.0.  Actually seems worse there.

I'm pretty sure it's a race condition as the permissions.sqlite simply isn't being read or written to at all when the problem occurs.

Comment 19

[Carsten Sommer]

I have a problem with cookie exceptions not being written to/load from permissions.sqlite, which seems to have the same root cause. I reported it in https://bugzilla.mozilla.org/show_bug.cgi?id=1319379 .

Btw., I'm the same person who reported the bug you mentioned in comment 12. I guess, from reading Cédric's findings in comment 12, that this was also due to the same root cause, i.e. the failure when trying to access the storage service. I'm not familiar with the code base, but still might try to find out what is going wrong with the storage service, or at least gather more information so another developer could provide a fix.

Comment 20

[Carsten Sommer]

I have put some debugging code into FF and it seems this happens because in storage/mozStorageService.cpp in Service::initialize() the call 

rc = ::sqlite3_config(SQLITE_CONFIG_MALLOC, &memMethods) 

fails with error code 21 (SQLITE_MISUSE). I'm not sure why this is the case. The SQLite documentation suggest this might happen, for example, because sqlite3_config() gets called after sqlite3_initialize(), but the only sqlite3_initialize() call I have found is in Service::initialize() and it only gets executed after the sqlite3_config() calls. 

Another interesting thing: After the do_GetService(MOZ_STORAGE_SERVICE_CONTRACTID) call fails, apparently Service::initialize() gets called a second time from somewhere else, and this time it is successful and does not generate an error when sqlite3_config(SQLITE_CONFIG_MALLOC, &memMethods) gets called.

Maybe there are other sqlite3_* calls elsewhere, that implicitly call sqlite3_initialize() before Service::initialize() gets called?

Comment 21

[Carsten Sommer]

After adding more debugging output, it turns out that sqlite3_initialize() is called implicitly about 100 times before sqlite3_config(SQLITE_CONFIG_MALLOC, &memMethods) gets called in Service::initialize(). This seems to be because we use NSS_DEFAULT_DB_TYPE=sql and apparently this triggers all those sqlite calls when initializing the NSS DB.

The code needs to make sure that this cannot happen. There is a (internal?) bug report for that, that sadly hasn't received much attention lately: https://bugzilla.mozilla.org/show_bug.cgi?id=730495

Here is a bug reports because of failing extensions installations which has the same root cause:
https://bugzilla.mozilla.org/show_bug.cgi?id=1277295  (reported by me)
https://bugzilla.mozilla.org/show_bug.cgi?id=1236865#c2

Comment 22

[Cédric Dufour]

(In reply to Carsten Sommer from comment #21)
> This seems to be because we use NSS_DEFAULT_DB_TYPE=sql

We also use NSS_DEFAULT_DB_TYPE=sql here at my shop (both for Firefox and Thunderbird). Good catch!

Comment 23

[MDNion]

The good news is bug 730495 has been marked as RESOLVED and seems to be slated for the Firefox 58 release.

Comment 24

[Andrei Purice]

Hey Michael,
Can you still reproduce this issue or should we close it?

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:timhuang, since the bug has high severity, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 26

[Tim Huang[:timhuang]]

The bug is old and I cannot reproduce it again. I will close this bug. Feel free to reopen if needed.