814554 - Firefox 17 silently stops processing permissions.sqlite when rejecting rules valid under Firefox 16

Status: RESOLVED FIXED

(Core :: Permission Manager, defect)

Description

[_ck_]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/17.0 Firefox/17.0
Build ID: 20121121075611

Steps to reproduce:

Firefox 17 (and 18) stops loading rules from permissions.sqlite and does not save new rules when it encounters rules from Firefox 16 that it no longer likes.

Please see for more details http://forums.mozillazine.org/viewtopic.php?f=38&t=2621009




Actual results:

One clear example is that scheme:file will cause all rules to stop processing and no new rules can be added as long as it exists in permissions.sqlite

I believe there is also a problem with single letter domains and domains that start with a dot. There may be other rejected rule formats.


Expected results:

All Firefox 16 rules should be accepted in Firefox 17 or bad rules skipped while processing continues and an error message is generated until rule is fixed or removed.

A better permissions editor, or at least a way to examine all rules like exexception https://addons.mozilla.org/firefox/addon/exexceptions/ would also be nice.

Comment 1

[Matt Grimes [:Matt_G]]

This is getting a lot of attention on SUMO as well. Here is the main thread: https://support.mozilla.org/en-US/questions/942274

Wondering if QA can reproduce?

Comment 2

[Alex Keybl [:akeybl]]

Anthony - can you try reproducing?

Comment 3

[Alex Keybl [:akeybl]]

Josh - can you please find an assignee to investigate in engineering?

Comment 4

[u279076]

It's not clear to me what specific behaviour is broken. I need more detailed steps to reproduce this. I've tested the basic behaviour of disabling cookies and adding exceptions and the behaviour is exactly the same in Firefox 16.0.2, 17.0, 18.0b1, 19.0a2, and 20.0a1.

I tried parsing the SUMO and Mozillazine threads but I was unable to find any information to better guide my testing. Someone who is able to see this, please provide more concrete examples of websites and steps to reproduce.

Thanks

Comment 5

[u279076]

I've also asked Softvision to help debug this issue since they come online overnight Sunday.

Comment 6

[u279076]

I tried this again with the settings in bug 814189 comment 0.

With Firefox 16.0.2, preferences/permissions are remembered across restart.
After updating to Firefox 17.0 via the About dialog, preferences/permissions are remembered and respected across restart. I've tried with a few different websites (Google, Youtube, Facebook, Twitter, Amazon, CNN, and Reddit). 

_ck_, since you reported this issue, is there any more information you can provide? I'm just not seeing any issues on my end.

Comment 7

[GreenRep]

I think I have the same problem. I am on Linux with this Firefox (FF) profile since ages. Using the binaries (local: de) from the mozilla website. Updated from 16.0.2 to 17.0 by manually downloading a new binary, deleting the old fox, extracting the new one (since the FF updater said I already have the newest version - which was wrong). 

I use the userdefined privacy setting with cookies expiration set to "ask me every time". I always use the "remember this" checkbox, so I am only prompted on first visit of some page.

This is the experience (which I successfully reproduced by restoring the backupped profile from 16.0.2):
Right after the first start with 17 I visited two websites which I frequently visit (so cookie acceptance rules are stored). On both visits FF asked me if I want to accept cookies. I accepted with the remember checkbox checked. Closed FF. Started FF again, visited the pages again: FF asked me again. 

If I "mv permissions.sqlite permissions.sqlite.old" and start FF afterwards, visit some website, accept. Exit FF. Then a new permission.sqlite file is created and the settings are remembered (if I start FF again and visit the websites I am not prompted). 

So, as mentioned elsewhere, it seems that my FF17 is really not happy about some rules in my permissions file which was accepted by FF16 without any problems. I never edited the permissions file by hand. The workaround which involves deleting the permissions file to get a new one is not really a workaround since that means I would loose several thousands of permissions entries.   I am afraid, I cannot post the apparently broken permissions file here since it contains (obviously) personal information - all the sites I visited. Hope you can get FF17 a bit more fault tolerant (like the versions before were).

Comment 8

[GreenRep]

To extend my previous comment (maybe it helps): I tried with Aurora 18.0a2 (2012-11-19) on Win XP. Using a fresh profile which is set up to ask if cookies should be accepted. On first website visit the popup asks, cookies are denied by me (remember checkbox is set). Firefox closed, opened website again - no question (since the "block" decision is saved). 

Now I replaced this permissions.sqlite by the permissions.sqlite from my old linux profile (mentioned above). Behaviour now is: FF asks on website visit, cookies are denied by me (remember checkbox is set). I restart FF, visit again, FF asks again, which is wrong.  So it seems FF18.0a2 behaves exactly like FF17 and it seems that the problem even occurs with a clean profile - just with an old permission.sqlite file.

Comment 9

[_ck_]

(In reply to GreenRep from comment #8)

GreenRep, see my mozillazine post to export your old Firefox 16 permissions.sqlite to a flat text file (hostperm.1) using the exexceptions extension so you can look for "bad" rules that Firefox 16 accepts but Firefox 17 rejects.

http://forums.mozillazine.org/viewtopic.php?p=12499889#p12499889

Comment 10

[GreenRep]

(In reply to _ck_ from comment #9)
> GreenRep, see my mozillazine post to export your old Firefox 16 ...

Yes, I saw this before.. but, eh, I cannot "look" for "bad" rules (however they will look like) - it is way too much lines. And: that is no solution for all the others (presumed) that have the same problem (well, maybe they either have killed their old profile or switched to another browser). Thank you anyway. :)

Comment 11

[_ck_]

(In reply to GreenRep from comment #10)

Well the other trick is that the exexceptions extension will export hostperm.1 alphabetically. Then when you remove permissions.sqlite and leave hostperm.1 in place, Firefox 17 will attempt to import hostperm.1 and recreate permissions.sqlite

Then when you look in your cookie exceptions list in Firefox 17, you will see alphabetically where it cuts off. Then by looking around that last entry in hostperm.1 you can find the bad rule pretty easily, it will stick out in some different way.

Comment 12

[GreenRep]

(In reply to _ck_ from comment #11)
Thanks for clarifying the workaround procedure. My "bad" entries were:

host	cookie	8	scheme:file
host	popup	1	scheme:file

I have removed them from an hostperm.1 export which I imported afterwards in FF17. FF17 works as expected. Also working: remove the entries with FF's own tools (from the cookies exceptions menu and the popup exceptions menu) in FF16.0.2. Then switch to FF17.

Comment 13

[GreenRep]

Attachment: permissions.sqlite file demonstrating the problem

Attached is a permissions.sqlite created in FF16 (by removing all other entries, exporting to hostperm.1 and importing hostperm.1 again - all in FF16) with the three cookie entries: www.openstreetmap.org scheme:file and schafmail.de

Procedure: Start FF17 with no profile (a new one is created), set cookie prefs to ask every time, close FF. Replace permissions.sqlite with the attached one.

(error) result in FF17: the only listed cookie exception (in FF settings) is for schafmail.de. Visit http://www.openstreetmap.org, accept the cookie and set the remember checkbox. This exception is now listed in the cookie exceptions and cookies are set. Close FF. Restart. The only cookie exception is schafmail.de. Visit openstreetmap again. It is asked again if cookies should be allowed.

Expected result (and how it is in FF16.0.2): three cookie exception entries. openstreetmap does not ask for permission (the exception is already existing) it just sets the cookies. Same after restart.

Comment 14

[George Billios]

I've noticed that if you have cookies from IPv6 IPs then it flags them also as invalid.

Comment 15

[_ck_]

Since people elsewhere are also reporting problem with IPv6 addresses in addition to `scheme:file`, might I suggest there is a bug when processing colons `:` in permissions.sqlite key and/or value data.

Someone might have unintentionally changed a regex or something similar in 17 that makes it far less flexible than 16.

Comment 16

[Alice0775 White]

Regression window(m-c)
Good:
http://hg.mozilla.org/mozilla-central/rev/e08a67884b9b
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826150859
Bad:
http://hg.mozilla.org/mozilla-central/rev/0a9e931cdcf3
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826192258
Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e08a67884b9b&tochange=0a9e931cdcf3

Regression window(m-i)
Good:
http://hg.mozilla.org/integration/mozilla-inbound/rev/ad09eafd9bd4
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826095258
Bad:
http://hg.mozilla.org/integration/mozilla-inbound/rev/a5d691072fd6
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826123658
Pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=ad09eafd9bd4&tochange=a5d691072fd6

Suspected:  Bug 777072

Comment 17

[Virgil Dicu [:virgil] [QA]]

FTR, couldn't reproduce previously while simply setting the preferences/upgrading from F16 to 17. Copying the permissions.sqlite from comment 13 did the trick, however.

Comment 18

[Paul Silaghi, QA [:pauly]]

(In reply to Virgil Dicu [:virgil] [QA] from comment #17)
> FTR, couldn't reproduce previously while simply setting the
> preferences/upgrading from F16 to 17. Copying the permissions.sqlite from
> comment 13 did the trick, however.
I confirm.

Comment 19

[Jason Duell]

Assigning to mounir based on regression window.

Comment 20

[Mounir Lamouri (:mounir)]

I can reproduce the bug. Will work on that.

Comment 21

[Mounir Lamouri (:mounir)]

This is not related to cookies but to the permission manager. In a nutshell, the changes we did for the permission manager this summer broke the hack that was allowing files to have permissions. At load time, the permission manager is trying to find the principal for "http://scheme:file" which, hum, isn't really working.

Comment 22

[Mounir Lamouri (:mounir)]

Attachment: Don't stop when an entry isn't readable

The permission manager has a quite un-healty behaviour right now: as soon as a permission entry isn't readable, it will stop loading the database and return an error. It happens that this error is ignored (except in one situation).

I think in the long term, we should not return an error and simply assume that the load will do the best thing and assert any error. This patch is a change in that direction but trying to be safe so it still return an error code but does that only after we have loaded *all* permissions.

With that simple patch, the most important part of the bug here is fixed: you can load your Firefox 16 permission file in Firefox 17+. However, the permissions with "scheme:file" will be ignored.

I don't know how much we want to support "scheme:file" though...

Comment 23

[Mounir Lamouri (:mounir)]

Comment on attachment 685626 [details] [diff] [review]
Don't stop when an entry isn't readable

Regarding "file://" handling, I've open bug 815640. I think the "right way" to do that is way more risky than this patch. I believe this patch might be enough for an emergency fix: all permissions will be loaded but "file://" ones are going to be ignored.

Comment 24

[Mounir Lamouri (:mounir)]

(In reply to _ck_ from comment #15)
> Since people elsewhere are also reporting problem with IPv6 addresses in
> addition to `scheme:file`, might I suggest there is a bug when processing
> colons `:` in permissions.sqlite key and/or value data.

Do you have a test case? ie. an IPv6 address I could use to try.

Comment 25

[Mounir Lamouri (:mounir)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/af301a7b9ecf

Comment 26

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/af301a7b9ecf

Comment 27

[Mounir Lamouri (:mounir)]

Comment on attachment 685626 [details] [diff] [review]
Don't stop when an entry isn't readable

[Approval Request Comment]
Regression caused by (bug #): bug 777072
User impact if declined: if a user has an invalid permission entry in his/her database, all permissions after that entry will be ignored.
Risk to taking this patch (and alternatives if risky): Very low, the only change is to not stop loading the file if an entry fail to be read. The permission manager already assumes that a broken database can happen and works fine with that so loading a partial database just works.

Given the risk/benefit, I think we should land this patch in aurora, beta, release and esr.

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 685626 [details] [diff] [review]
Don't stop when an entry isn't readable

Please go ahead with landing (priority on release/esr17 for our builds to go today) and on esr17 make sure to land to both default and GECKO170_2012111914_RELBRANCH relbranch, thank you.

Comment 29

[Mounir Lamouri (:mounir)]

Aurora:
https://hg.mozilla.org/releases/mozilla-aurora/rev/609d111f9b9c

Beta:
https://hg.mozilla.org/releases/mozilla-beta/rev/d4646b5033da

Release:
https://hg.mozilla.org/releases/mozilla-release/rev/909cd366fc7d

ESR:
https://hg.mozilla.org/releases/mozilla-esr17/rev/076b31b38565

ESR-GECKO170_2012111914_RELBRANCH:
https://hg.mozilla.org/releases/mozilla-esr17/rev/039489f19693

Comment 30

[GreenRep]

Mounir: thank you very much for fixing! A question and I hope that is not too much dumbness on my side: The root problem is not fixed, is it? I mean: is it not possible to have permissions for files (and maybe ipv6 addresses) now (with the patch)? I guess that should be another bug then?

Comment 31

[Michael Lefevre]

(In reply to GreenRep from comment #30)
> Mounir: thank you very much for fixing! A question and I hope that is not
> too much dumbness on my side: The root problem is not fixed, is it? I mean:
> is it not possible to have permissions for files (and maybe ipv6 addresses)
> now (with the patch)? I guess that should be another bug then?

Mounir filed bug 815640 for file permissions. If there's an issue with IPv6 addresses, that should be another bug too.

Comment 32

[Simona Badau, Desktop QA]

When trying to verify this on Firefox 17.0.1 I noticed that after replacing the permissions.sqlite file in my profile folder I had 2 cookies listed under the Excetions category in Options->Privacy-> History. One of them is schafmail.de and the other one is www.openstreetmap.org, even though I didn't accept the cookie and set the remember checkbox. 

Is that expected in any way?

Moreover on Firefox 17 after replacing the permissions.sqlite file the only listed cookie is schafmail.de, on Firefox 17.0.1 I see 2 cookies (schafmail.de and www.openstreetmap.org) and on Firefox 16.02 I see 3 cookies listed under Exceptions (www.openstreetmap.org scheme:file and schafmail.de). 

Is there any reason for all the  3 different results when replacing the same permission.sqlite file?

Comment 33

[Mounir Lamouri (:mounir)]

(In reply to Simona B [QA] from comment #32)
> When trying to verify this on Firefox 17.0.1 I noticed that after replacing
> the permissions.sqlite file in my profile folder I had 2 cookies listed
> under the Excetions category in Options->Privacy-> History. One of them is
> schafmail.de and the other one is www.openstreetmap.org, even though I
> didn't accept the cookie and set the remember checkbox. 
> 
> Is that expected in any way?

Yes. The file contains a cookie permission for openstreetmap.org.

> Moreover on Firefox 17 after replacing the permissions.sqlite file the only
> listed cookie is schafmail.de, on Firefox 17.0.1 I see 2 cookies
> (schafmail.de and www.openstreetmap.org) and on Firefox 16.02 I see 3
> cookies listed under Exceptions (www.openstreetmap.org scheme:file and
> schafmail.de). 
> 
> Is there any reason for all the  3 different results when replacing the same
> permission.sqlite file?

Yes. Since Firefox 17, we consider "scheme:file" as an invalid entry. Basically, the cookie permissions in the file are the following:
schafmail.de
scheme:file
openstreetmap.org

Firefox 16 reads, accepts and loads all of them (thus, three of them listed).
Firefox 17.0.0 accepts "schafmail.de" but refuses "scheme:file" so stop reading the permissions, thus do not load "openstreetmap.org"
Firefox 17.0.1 also refuses "scheme:file" but continue to read the file so "openstreetmap.org" is loaded.

The behaviour you see seem fine to me.

Comment 34

[Simona Badau, Desktop QA]

Verified as fixed on Firefox 17.0.1 (on Ubuntu 12.10, Mac OS X 10.8 and Windows 7) - cookies permissions persist across Firefox after restart - used the STR from Comment 13, both cookies (schafmail.de and www.openstreetmap.org) are listed under Cookies Exceptions in Options > Privacy > History. 

Based on this and on Comment 33 setting tracking flag for Firefox 17 to verified.

Comment 35

[GreenRep]

(In reply to Michael Lefevre from comment #31)
> Mounir filed bug 815640 for file permissions. If there's an issue with IPv6
> addresses, that should be another bug too.
Files: oops, sorry, had overlooked this.
IPv6: I have read about problems, maybe just guesses, but see e.g. _ck_ in comment #15 .

(In reply to Simona B [QA] from comment #32)
Yes, that is all expected that way. Furthermore, in FF17.0.1 (and FF<17), you should be able to add new cookie exceptions for other sites which also should be remembered by Firefox after a restart.

Comment 36

[GreenRep]

Verified with 17.0.1 build1 (local: de) on my old Linux system and on Win XP SP3. And with 19.0a2.en-US (20121129042015) on Win XP SP3. An existing scheme:file entry is ignored as expected.

Comment 37

[u279076]

This has now been verified fixed in 17.0.1esr builds as well. While we continue to focus on releasing these builds, I would appreciate it if someone could test the latest Nightly, Aurora, and 18.0b2 builds. These should now be fixed as well.

Thank you.

Comment 38

[Simona Badau, Desktop QA]

Verified as fixed on the latest Aurora and on the latest Nightly on Windows 7, Mac OS X 10.7 and on Ubuntu 12.10:

Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20121202 Firefox/19.0 Build ID:20121202042013
Mozilla/5.0 (Windows NT 6.1; rv:20.0) Gecko/20121202 Firefox/20.0 Build ID:20121202030723	

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20121203 Firefox/19.0 Build ID:20121203042014
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20121203 Firefox/20.0 Build ID:20121203030801

Mozilla/5.0 (X11; Linux i686; rv:19.0) Gecko/20121203 Firefox/19.0 Build ID:20121203042014
Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20121203 Firefox/20.0 Build ID:20121203030801

The fix didn't make it for Firefox 18 beta 2 (the patch landed only on 2012-11-28). I will verify that this is fixed on Firefox 18 beta 3.

Comment 39

[Simona Badau, Desktop QA]

Verified as fixed on Firefox 18 beta 3 on Windows 7, Ubuntu 12.10 and Mac OS X 10.7:

Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0
Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:18.0) Gecko/20100101 Firefox/18.0