Closed Bug 814554 Opened 7 years ago Closed 7 years ago
Firefox 17 silently stops processing permissions
.sqlite when rejecting rules valid under Firefox 16
64.00 KB, application/octet-stream
2.15 KB, patch
|Details | Diff | Splinter Review|
User Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/17.0 Firefox/17.0 Build ID: 20121121075611 Steps to reproduce: Firefox 17 (and 18) stops loading rules from permissions.sqlite and does not save new rules when it encounters rules from Firefox 16 that it no longer likes. Please see for more details http://forums.mozillazine.org/viewtopic.php?f=38&t=2621009 Actual results: One clear example is that scheme:file will cause all rules to stop processing and no new rules can be added as long as it exists in permissions.sqlite I believe there is also a problem with single letter domains and domains that start with a dot. There may be other rejected rule formats. Expected results: All Firefox 16 rules should be accepted in Firefox 17 or bad rules skipped while processing continues and an error message is generated until rule is fixed or removed. A better permissions editor, or at least a way to examine all rules like exexception https://addons.mozilla.org/firefox/addon/exexceptions/ would also be nice.
Component: Untriaged → Networking: Cookies
Product: Firefox → Core
This is getting a lot of attention on SUMO as well. Here is the main thread: https://support.mozilla.org/en-US/questions/942274 Wondering if QA can reproduce?
Anthony - can you try reproducing?
QA Contact: anthony.s.hughes
Josh - can you please find an assignee to investigate in engineering?
Assignee: nobody → joshmoz
It's not clear to me what specific behaviour is broken. I need more detailed steps to reproduce this. I've tested the basic behaviour of disabling cookies and adding exceptions and the behaviour is exactly the same in Firefox 16.0.2, 17.0, 18.0b1, 19.0a2, and 20.0a1. I tried parsing the SUMO and Mozillazine threads but I was unable to find any information to better guide my testing. Someone who is able to see this, please provide more concrete examples of websites and steps to reproduce. Thanks
I've also asked Softvision to help debug this issue since they come online overnight Sunday.
I tried this again with the settings in bug 814189 comment 0. With Firefox 16.0.2, preferences/permissions are remembered across restart. After updating to Firefox 17.0 via the About dialog, preferences/permissions are remembered and respected across restart. I've tried with a few different websites (Google, Youtube, Facebook, Twitter, Amazon, CNN, and Reddit). _ck_, since you reported this issue, is there any more information you can provide? I'm just not seeing any issues on my end.
I think I have the same problem. I am on Linux with this Firefox (FF) profile since ages. Using the binaries (local: de) from the mozilla website. Updated from 16.0.2 to 17.0 by manually downloading a new binary, deleting the old fox, extracting the new one (since the FF updater said I already have the newest version - which was wrong). I use the userdefined privacy setting with cookies expiration set to "ask me every time". I always use the "remember this" checkbox, so I am only prompted on first visit of some page. This is the experience (which I successfully reproduced by restoring the backupped profile from 16.0.2): Right after the first start with 17 I visited two websites which I frequently visit (so cookie acceptance rules are stored). On both visits FF asked me if I want to accept cookies. I accepted with the remember checkbox checked. Closed FF. Started FF again, visited the pages again: FF asked me again. If I "mv permissions.sqlite permissions.sqlite.old" and start FF afterwards, visit some website, accept. Exit FF. Then a new permission.sqlite file is created and the settings are remembered (if I start FF again and visit the websites I am not prompted). So, as mentioned elsewhere, it seems that my FF17 is really not happy about some rules in my permissions file which was accepted by FF16 without any problems. I never edited the permissions file by hand. The workaround which involves deleting the permissions file to get a new one is not really a workaround since that means I would loose several thousands of permissions entries. I am afraid, I cannot post the apparently broken permissions file here since it contains (obviously) personal information - all the sites I visited. Hope you can get FF17 a bit more fault tolerant (like the versions before were).
To extend my previous comment (maybe it helps): I tried with Aurora 18.0a2 (2012-11-19) on Win XP. Using a fresh profile which is set up to ask if cookies should be accepted. On first website visit the popup asks, cookies are denied by me (remember checkbox is set). Firefox closed, opened website again - no question (since the "block" decision is saved). Now I replaced this permissions.sqlite by the permissions.sqlite from my old linux profile (mentioned above). Behaviour now is: FF asks on website visit, cookies are denied by me (remember checkbox is set). I restart FF, visit again, FF asks again, which is wrong. So it seems FF18.0a2 behaves exactly like FF17 and it seems that the problem even occurs with a clean profile - just with an old permission.sqlite file.
(In reply to GreenRep from comment #8) GreenRep, see my mozillazine post to export your old Firefox 16 permissions.sqlite to a flat text file (hostperm.1) using the exexceptions extension so you can look for "bad" rules that Firefox 16 accepts but Firefox 17 rejects. http://forums.mozillazine.org/viewtopic.php?p=12499889#p12499889
(In reply to _ck_ from comment #9) > GreenRep, see my mozillazine post to export your old Firefox 16 ... Yes, I saw this before.. but, eh, I cannot "look" for "bad" rules (however they will look like) - it is way too much lines. And: that is no solution for all the others (presumed) that have the same problem (well, maybe they either have killed their old profile or switched to another browser). Thank you anyway. :)
(In reply to GreenRep from comment #10) Well the other trick is that the exexceptions extension will export hostperm.1 alphabetically. Then when you remove permissions.sqlite and leave hostperm.1 in place, Firefox 17 will attempt to import hostperm.1 and recreate permissions.sqlite Then when you look in your cookie exceptions list in Firefox 17, you will see alphabetically where it cuts off. Then by looking around that last entry in hostperm.1 you can find the bad rule pretty easily, it will stick out in some different way.
(In reply to _ck_ from comment #11) Thanks for clarifying the workaround procedure. My "bad" entries were: host cookie 8 scheme:file host popup 1 scheme:file I have removed them from an hostperm.1 export which I imported afterwards in FF17. FF17 works as expected. Also working: remove the entries with FF's own tools (from the cookies exceptions menu and the popup exceptions menu) in FF16.0.2. Then switch to FF17.
Attached is a permissions.sqlite created in FF16 (by removing all other entries, exporting to hostperm.1 and importing hostperm.1 again - all in FF16) with the three cookie entries: www.openstreetmap.org scheme:file and schafmail.de Procedure: Start FF17 with no profile (a new one is created), set cookie prefs to ask every time, close FF. Replace permissions.sqlite with the attached one. (error) result in FF17: the only listed cookie exception (in FF settings) is for schafmail.de. Visit http://www.openstreetmap.org, accept the cookie and set the remember checkbox. This exception is now listed in the cookie exceptions and cookies are set. Close FF. Restart. The only cookie exception is schafmail.de. Visit openstreetmap again. It is asked again if cookies should be allowed. Expected result (and how it is in FF16.0.2): three cookie exception entries. openstreetmap does not ask for permission (the exception is already existing) it just sets the cookies. Same after restart.
I've noticed that if you have cookies from IPv6 IPs then it flags them also as invalid.
Since people elsewhere are also reporting problem with IPv6 addresses in addition to `scheme:file`, might I suggest there is a bug when processing colons `:` in permissions.sqlite key and/or value data. Someone might have unintentionally changed a regex or something similar in 17 that makes it far less flexible than 16.
Regression window(m-c) Good: http://hg.mozilla.org/mozilla-central/rev/e08a67884b9b Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826150859 Bad: http://hg.mozilla.org/mozilla-central/rev/0a9e931cdcf3 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826192258 Pushlog: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e08a67884b9b&tochange=0a9e931cdcf3 Regression window(m-i) Good: http://hg.mozilla.org/integration/mozilla-inbound/rev/ad09eafd9bd4 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826095258 Bad: http://hg.mozilla.org/integration/mozilla-inbound/rev/a5d691072fd6 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 ID:20120826123658 Pushlog: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=ad09eafd9bd4&tochange=a5d691072fd6 Suspected: Bug 777072
Attachment #684953 - Attachment mime type: text/plain → application/octet-stream
FTR, couldn't reproduce previously while simply setting the preferences/upgrading from F16 to 17. Copying the permissions.sqlite from comment 13 did the trick, however.
(In reply to Virgil Dicu [:virgil] [QA] from comment #17) > FTR, couldn't reproduce previously while simply setting the > preferences/upgrading from F16 to 17. Copying the permissions.sqlite from > comment 13 did the trick, however. I confirm.
Assigning to mounir based on regression window.
Assignee: joshmoz → mounir
I can reproduce the bug. Will work on that.
This is not related to cookies but to the permission manager. In a nutshell, the changes we did for the permission manager this summer broke the hack that was allowing files to have permissions. At load time, the permission manager is trying to find the principal for "http://scheme:file" which, hum, isn't really working.
Component: Networking: Cookies → General
The permission manager has a quite un-healty behaviour right now: as soon as a permission entry isn't readable, it will stop loading the database and return an error. It happens that this error is ignored (except in one situation). I think in the long term, we should not return an error and simply assume that the load will do the best thing and assert any error. This patch is a change in that direction but trying to be safe so it still return an error code but does that only after we have loaded *all* permissions. With that simple patch, the most important part of the bug here is fixed: you can load your Firefox 16 permission file in Firefox 17+. However, the permissions with "scheme:file" will be ignored. I don't know how much we want to support "scheme:file" though...
Attachment #685626 - Flags: review?(jonas)
Comment on attachment 685626 [details] [diff] [review] Don't stop when an entry isn't readable Regarding "file://" handling, I've open bug 815640. I think the "right way" to do that is way more risky than this patch. I believe this patch might be enough for an emergency fix: all permissions will be loaded but "file://" ones are going to be ignored.
Attachment #685626 - Attachment description: Part 1 - Don't stop when an entry isn't readable → Don't stop when an entry isn't readable
(In reply to _ck_ from comment #15) > Since people elsewhere are also reporting problem with IPv6 addresses in > addition to `scheme:file`, might I suggest there is a bug when processing > colons `:` in permissions.sqlite key and/or value data. Do you have a test case? ie. an IPv6 address I could use to try.
Attachment #685626 - Flags: review?(jonas) → review+
Target Milestone: --- → mozilla20
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Comment on attachment 685626 [details] [diff] [review] Don't stop when an entry isn't readable [Approval Request Comment] Regression caused by (bug #): bug 777072 User impact if declined: if a user has an invalid permission entry in his/her database, all permissions after that entry will be ignored. Risk to taking this patch (and alternatives if risky): Very low, the only change is to not stop loading the file if an entry fail to be read. The permission manager already assumes that a broken database can happen and works fine with that so loading a partial database just works. Given the risk/benefit, I think we should land this patch in aurora, beta, release and esr.
Comment on attachment 685626 [details] [diff] [review] Don't stop when an entry isn't readable Please go ahead with landing (priority on release/esr17 for our builds to go today) and on esr17 make sure to land to both default and GECKO170_2012111914_RELBRANCH relbranch, thank you.
Attachment #685626 - Flags: approval-mozilla-release?
Attachment #685626 - Flags: approval-mozilla-release+
Attachment #685626 - Flags: approval-mozilla-esr17?
Attachment #685626 - Flags: approval-mozilla-esr17+
Attachment #685626 - Flags: approval-mozilla-beta?
Attachment #685626 - Flags: approval-mozilla-beta+
Attachment #685626 - Flags: approval-mozilla-aurora?
Attachment #685626 - Flags: approval-mozilla-aurora+
Aurora: https://hg.mozilla.org/releases/mozilla-aurora/rev/609d111f9b9c Beta: https://hg.mozilla.org/releases/mozilla-beta/rev/d4646b5033da Release: https://hg.mozilla.org/releases/mozilla-release/rev/909cd366fc7d ESR: https://hg.mozilla.org/releases/mozilla-esr17/rev/076b31b38565 ESR-GECKO170_2012111914_RELBRANCH: https://hg.mozilla.org/releases/mozilla-esr17/rev/039489f19693
Mounir: thank you very much for fixing! A question and I hope that is not too much dumbness on my side: The root problem is not fixed, is it? I mean: is it not possible to have permissions for files (and maybe ipv6 addresses) now (with the patch)? I guess that should be another bug then?
(In reply to GreenRep from comment #30) > Mounir: thank you very much for fixing! A question and I hope that is not > too much dumbness on my side: The root problem is not fixed, is it? I mean: > is it not possible to have permissions for files (and maybe ipv6 addresses) > now (with the patch)? I guess that should be another bug then? Mounir filed bug 815640 for file permissions. If there's an issue with IPv6 addresses, that should be another bug too.
When trying to verify this on Firefox 17.0.1 I noticed that after replacing the permissions.sqlite file in my profile folder I had 2 cookies listed under the Excetions category in Options->Privacy-> History. One of them is schafmail.de and the other one is www.openstreetmap.org, even though I didn't accept the cookie and set the remember checkbox. Is that expected in any way? Moreover on Firefox 17 after replacing the permissions.sqlite file the only listed cookie is schafmail.de, on Firefox 17.0.1 I see 2 cookies (schafmail.de and www.openstreetmap.org) and on Firefox 16.02 I see 3 cookies listed under Exceptions (www.openstreetmap.org scheme:file and schafmail.de). Is there any reason for all the 3 different results when replacing the same permission.sqlite file?
(In reply to Simona B [QA] from comment #32) > When trying to verify this on Firefox 17.0.1 I noticed that after replacing > the permissions.sqlite file in my profile folder I had 2 cookies listed > under the Excetions category in Options->Privacy-> History. One of them is > schafmail.de and the other one is www.openstreetmap.org, even though I > didn't accept the cookie and set the remember checkbox. > > Is that expected in any way? Yes. The file contains a cookie permission for openstreetmap.org. > Moreover on Firefox 17 after replacing the permissions.sqlite file the only > listed cookie is schafmail.de, on Firefox 17.0.1 I see 2 cookies > (schafmail.de and www.openstreetmap.org) and on Firefox 16.02 I see 3 > cookies listed under Exceptions (www.openstreetmap.org scheme:file and > schafmail.de). > > Is there any reason for all the 3 different results when replacing the same > permission.sqlite file? Yes. Since Firefox 17, we consider "scheme:file" as an invalid entry. Basically, the cookie permissions in the file are the following: schafmail.de scheme:file openstreetmap.org Firefox 16 reads, accepts and loads all of them (thus, three of them listed). Firefox 17.0.0 accepts "schafmail.de" but refuses "scheme:file" so stop reading the permissions, thus do not load "openstreetmap.org" Firefox 17.0.1 also refuses "scheme:file" but continue to read the file so "openstreetmap.org" is loaded. The behaviour you see seem fine to me.
Verified as fixed on Firefox 17.0.1 (on Ubuntu 12.10, Mac OS X 10.8 and Windows 7) - cookies permissions persist across Firefox after restart - used the STR from Comment 13, both cookies (schafmail.de and www.openstreetmap.org) are listed under Cookies Exceptions in Options > Privacy > History. Based on this and on Comment 33 setting tracking flag for Firefox 17 to verified.
(In reply to Michael Lefevre from comment #31) > Mounir filed bug 815640 for file permissions. If there's an issue with IPv6 > addresses, that should be another bug too. Files: oops, sorry, had overlooked this. IPv6: I have read about problems, maybe just guesses, but see e.g. _ck_ in comment #15 . (In reply to Simona B [QA] from comment #32) Yes, that is all expected that way. Furthermore, in FF17.0.1 (and FF<17), you should be able to add new cookie exceptions for other sites which also should be remembered by Firefox after a restart.
Verified with 17.0.1 build1 (local: de) on my old Linux system and on Win XP SP3. And with 19.0a2.en-US (20121129042015) on Win XP SP3. An existing scheme:file entry is ignored as expected.
This has now been verified fixed in 17.0.1esr builds as well. While we continue to focus on releasing these builds, I would appreciate it if someone could test the latest Nightly, Aurora, and 18.0b2 builds. These should now be fixed as well. Thank you.
Verified as fixed on the latest Aurora and on the latest Nightly on Windows 7, Mac OS X 10.7 and on Ubuntu 12.10: Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20121202 Firefox/19.0 Build ID:20121202042013 Mozilla/5.0 (Windows NT 6.1; rv:20.0) Gecko/20121202 Firefox/20.0 Build ID:20121202030723 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20121203 Firefox/19.0 Build ID:20121203042014 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20121203 Firefox/20.0 Build ID:20121203030801 Mozilla/5.0 (X11; Linux i686; rv:19.0) Gecko/20121203 Firefox/19.0 Build ID:20121203042014 Mozilla/5.0 (X11; Linux i686; rv:20.0) Gecko/20121203 Firefox/20.0 Build ID:20121203030801 The fix didn't make it for Firefox 18 beta 2 (the patch landed only on 2012-11-28). I will verify that this is fixed on Firefox 18 beta 3.
Verified as fixed on Firefox 18 beta 3 on Windows 7, Ubuntu 12.10 and Mac OS X 10.7: Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0 Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:18.0) Gecko/20100101 Firefox/18.0
No longer blocks: 814189
Duplicate of this bug: 814189
No longer depends on: 813865
Duplicate of this bug: 813865
You need to log in before you can comment on or make changes to this bug.