Closed
Bug 434624
Opened 16 years ago
Closed 16 years ago
safebrowsing file not updated on first launch
Categories
(Toolkit :: Safe Browsing, defect)
Tracking
()
VERIFIED
FIXED
People
(Reporter: jbecerra, Assigned: dcamp)
References
Details
(Keywords: verified1.9.0.1, Whiteboard: [MU+])
Attachments
(2 files)
16.94 KB,
text/plain
|
Details | |
929 bytes,
patch
|
tony
:
review+
beltzner
:
approval1.9.0.1+
|
Details | Diff | Splinter Review |
While testing Fx3rc1, the urlclassifier3.sqlite file does not get updated at all after the first start. Steps: 1. Install Fx3rc1, or nightyl, using a fresh profile. 2. Locate the urlclassifier3.sqlite and see if gets updated from it's original 32kb size Expected: This file should be updated within the first 5 minutes of launch. Actual: It doesn't get updated, until you quit and start the browser after the very first launch.
Reporter | ||
Updated•16 years ago
|
Severity: normal → critical
Version: Trunk → 3.0 Branch
Reporter | ||
Comment 1•16 years ago
|
||
Actually, it looks like the urlclassifier3.sqlite file does get updated, however the OS does not report correctly the size of the file in the file explorer, in XP for example. So while the explorer lists the file initially as being 32k, if you get the properties from the context menu after a few minutes it shows a larger size, and it keeps getting larger. All the while, safebrowser warnings do show up for me on new installations on first run.
Reporter | ||
Comment 2•16 years ago
|
||
In Linux (ubuntu) the urlclassifier3.sqlite doesn't seem to be updated on first launch, and checking the file properties doesn't indicate that it is being updated.
Reporter | ||
Comment 3•16 years ago
|
||
If you are on linux and try out Fx3rc1 for the first time you might not be malware/phishing protected until you start a new session, so I'm nominating this for rc2.
Flags: blocking-firefox3?
Whiteboard: [RC2?]
Comment 4•16 years ago
|
||
Dave's not around this week and he's the guy who will be able to say for sure, but the following information would be helpful:
1) Is this being seen after, say, 10 minutes running a full profile, or 2 hours? The update scheduling has moved from a fixed interval to a random one, so it's possible that a pull wouldn't have happened after 10 minutes, that maybe would have happened after 12. If no update is happening after a few hours though, that's clearly not the explanation here.
2) If we're sure that updates aren't being fetched even after significant run time, then we need to know whether the fetches are actually happening or not. I recommend something like the LiveHTTPHeaders add-on to watch for update pings to google's safebrowsing servers.
It would also be helpful to get Dave's about:safebrowsing addon installed (attachment 320610 [details]) and see if there are chunks reported. The output is pretty raw, but if you look at the Local/Server ratios, you should see local chunk counts climbing towards 100% over time (you will likely have to reload the page though, it doesn't update dynamically).
Comment 5•16 years ago
|
||
Urr - stupid me - obviously you can't install addons on a first run, since they require a restart - so external tools may be necessary to watch for that traffic. #1 above is still relevant.
Updated•16 years ago
|
Flags: wanted1.9.0.x?
Flags: blocking-firefox3?
Flags: blocking-firefox3-
Comment 6•16 years ago
|
||
Incidentally, for me on a fresh profile, OS/X 4 minutes after start: [johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite Tue 27 May 2008 14:21:15 EDT -rw-r--r--@ 1 johnath johnath 32768 27 May 14:19 urlclassifier3.sqlite [johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite Tue 27 May 2008 14:22:55 EDT -rw-r--r--@ 1 johnath johnath 4005888 27 May 14:22 urlclassifier3.sqlite [johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite Tue 27 May 2008 14:24:02 EDT -rw-r--r--@ 1 johnath johnath 8814592 27 May 14:24 urlclassifier3.sqlite ... and so forth. It levels off at around 10M for that update, with another to come during the next refresh cycle.
Reporter | ||
Comment 7•16 years ago
|
||
Ubuntu 8.04 vm. I installed Fx3rc1 afresh yesterday, and I left it running for maybe a half hour. The urlclassifier file did not get updated. Then I installed livehttpheaders (knowing this might void the bug report), and I left the browser running for several hours. Livehttpheaders shows a POST (with some random string) and a GET /safebrowsing/newkey?... and a few more GETs for the default rss feed in my bookmarks toolbar. The size of the urlclassifier file has not changed, after at least 2 hours. I'll try again, this time installing livehttpeheaders and the about:safebrowsing extension at the same time, and I will leave it running for a few hours.
Comment 8•16 years ago
|
||
i can confirm this Problem on Fedora F8, new Firefox Profile and running over an hour and the file size does not change ...urlclassifier3.sqlite is still 32.0 KB (32768 bytes) big/small goog-malware-shavar Server: 2768 add chunks Local: 0 add chunks (0 percent) Server adds: 8,10,29,38,40-42,63-86,88-115,118-126,128-134,136-140,142,144-148,150-159,161-168,170-177,179-180,184-186,188-190,192-195,198-211,213-244,246-251,253,262,269-299,301-308,310-313,329-331,333-346,348-366,368-413,415-416,418-419,421-424,426-474,476-532,534-535,537-539,541-583,585-589,591-608,610-613,615-617,619-623,625,627-631,634-636,639,645,647-648,650,652,663-664,666-667,669,674,680,684-685,687-688,692,699-700,705,708,713,716,723-724,726,730-731,741,749,751,754-755,760,764,767,771,777-780,782-784,786,789-790,793,796-799,801-804,806,808,812-814,816-817,820-821,824-826,829-831,833,836-839,841,843-845,847,849-850,852,854-857,861-864,869,871-873,875-878,884-885,888-889,891-893,895-898,900-903,909-914,919,921,923-932,934-941,944-946,949,953-954,958,962,964-967,970-971,974-977,979-981,985-987,989-991,993,1000-1001,1003,1005,1008-1009,1013,1019,1022-1023,1026-1027,1029-1031,1035-1038,1040,1049,1051,1053-1055,1063-1067,1069,1071-1072,1075-1077,1083,1087,1089,1093-1094,1096,1099-1101,1103,1106-1107,1110-1111,1115,1118-1119,1122,1126,1128-1129,1134,1139-1140,1146-1148,1150-1152,1154-1155,1157-1158,1162-1164,1170,1172,1175-1177,1179-1180,1182-1183,1185,1188,1192-1193,1197-1198,1200-1205,1207,1209,1212-1213,1216,1219,1222-1224,1226-1227,1229-1230,1233-1235,1238,1243-1244,1246,1250,1255-1256,1263,1265-1267,1270,1272-1273,1275-1276,1279,1281,1284-1285,1289-1290,1292-1293,1296,1299-1300,1303-1308,1310,1313-1314,1316-1324,1329,1333,1336,1343,1346,1348-1349,1351,1353,1355-1356,1359-1360,1362,1368,1374,1378-1387,1390-1391,1394-1395,1398-1399,1402-1403,1407,1411,1413,1416-1417,1419,1423,1425-1427,1431-1432,1434-1436,1438-1445,1448-1450,1452-1453,1455-1456,1458-1464,1466-1467,1469,1473,1477-1478,1482,1484,1486-1487,1490,1493,1497-1532,1535-1540,1542-1571,1573-1575,1577-1579,1581-1583,1585,1588-1592,1594,1597,1600,1607-1611,1618,1620-1627,1629-1631,1633,1693,1696,1726,1729,1731-1732,1738-1739,1742,1746-1749,1752,1755,1758,1773,1820,1823,1851,1879,1888,1911-1912,1915,1919,1932,1936,1950,1953-1954,1956,1961-1962,1964-1965,1968-1972,1976-1978,1981,1989,1992,2000,2002-2007,2010-2011,2017,2025,2027-2028,2031-2036,2038-2040,2042,2044-2047,2065-2067,2071-2072,2074,2085,2091,2093-2096,2098-2102,2104,2107-2115,2117-2133,2135-2136,2138-2139,2141,2143-2169,2183-2192,2194-2198,2200-2208,2210-2218,2221,2224,2227,2260,2330-2331,2372,2378,2384-2387,2390,2392,2395,2397-2398,2400,2428-2431,2433-2436,2438-2441,2445-2446,2448-2449,2451-2455,2457,2462,2466-2469,2471-2473,2475,2479-2486,2488-2496,2498-2502,2505,2508,2510,2512,2516,2518,2520-2535,2543-2546,2548-2551,2557,2561,2563-2565,2570-2578,2583,2585-2587,2589,2592-2593,2596-2607,2610,2615,2619-2622,2625-2627,2631-2632,2640-2649,2651,2653-2656,2664-2665,2668-2672,2674-2675,2679,2681,2683,2691-2692,2695-2700,2702-2721,2723-2726,2728-2736,2738,2742-2743,2745-2776,2778-2779,2781,2783,2785-2786,2790-2792,2794,2797-2978,2980-3186,3188-3190,3193-3197,3199-3205,3207-3208,3210-3211,3215,3218-3219,3221,3223,3227-3229,3231-3232,3234-3237,3240,3242-3243,3245-3246,3249-3250,3252-3256,3258,3260,3263-3264,3266,3268-3277,3279-3280,3284,3286,3289,3291-3295,3297,3299-3302,3305,3307-3311,3313-3326,3328-3331,3336-3338,3342,3345-3349,3351-3377,3379,3381-3383,3385-3391,3394-3399,3405-3420,3423-3424,3430,3432,3434,3436,3439-3742,3747,3749,3751,3753-3786,3788-3789,3791-3792,3794-3803,3805-3809,3813-3918,3920,3922,3924,3926-3927,3929-3933,3935-3939,3941,3943-3945,3947-3956,3960-3961,3963-3966,3968-3970,3972-3973,3976,3978-3981,3983-3984,3986-4003,4005,4007-4008,4011-4023,4025,4027-4028,4030-4032,4034-4045,4047-4062,4065,4068-4070,4073,4075-4080,4082-4083,4086-4095,4097-4100,4104-4106,4108-4114,4116-4122,4124-4127,4129-4141,4143-4149,4151,4154-4159,4161-4162,4164-4170,4172-4183,4185-4186,4188-4198,4200-4201,4203-4204,4206,4208-4219,4221-4236,4240-4246,4248-4256 Local adds: undefined Server subs: 1-4035 Local subs: undefined goog-phish-shavar Server: 4183 add chunks Local: 0 add chunks (0 percent) Server adds: 27,60,79,90,105,3853,3997,4041-4043,4111,4152,4173-4174,4243,4258,4260,4275,4279,4305,4323,4330,4386,4415,4520,4556,4562,4660-4661,4669,4679-4680,4684,4814,4816,4822,4933,4938,4940,4961,4964,5068,5079,5081,5084,5110-5111,5116,5140,5193,5222,5236,5250,5313,5328,5351,5353,5358,5394-5395,5412,5444,5461-5463,5466-5467,5470,5475,5518-5519,5574,5587,5623,5714-5715,5730-5731,5734,5772,5810,5877,5939,5957,6039,6084,6089,6120,6122,6147,6165,6233,6280,6286,6364,6391,6407-6408,6429,6479,6508-6509,6537,6541,6551,6671,6680,6728,6741,6746,6754,6769,6775,6793,6863,6886,6902,6904,6907,6914,6916,6918,6943,7002-7003,7008,7010-7012,7033,7035,7039,7041-7042,7044,7046-7048,7057-7062,7065-7077,7080-7084,7087-7088,7090-7093,7095,7097-7106,7108-7110,7112-7116,7118-7121,7123,7126-7134,7136-7142,7144-7150,7152-7159,7161-7173,7175-7188,7191-7192,7194-7200,7202-7207,7209,7211-7221,7223-7226,7228-7230,7232-7234,7236-7246,7248-7252,7255-7259,7261-7262,7264-7276,7278-7280,7282-7284,7286-7291,7293-7300,7302-7328,7330-7336,7338-7341,7343-7347,7349-7369,7371-7375,7377-7379,7381-7390,7396-7401,7403-7404,7406,7409-7410,7412-7414,7417-7427,7429-7430,7433-7435,7437-7443,7445-7447,7449-7451,7453-7456,7458,7460,7462-7477,7480-7489,7491,7494-7495,7497-7498,7500-7501,7503,7505,7507-7510,7512-7513,7515-7520,7522-7531,7533-7537,7540-7549,7551-7552,7554,7556,7558-7561,7563-7585,7587-7590,7593-7595,7597,7599-7608,7610-7612,7614-7616,7618-7624,7626,7628-7635,7637-7638,7640,7642-7648,7650-7653,7655-7660,7662-7667,7669-7682,7684,7686-7708,7710-7723,7726-7750,7752-7757,7759-7764,7768-7783,7786-7788,7791-7805,7807-7809,7811-7813,7815-7821,7824-7841,7843-7857,7859-7870,7872,7874-7876,7878-7879,7881,7883-7891,7893-7897,7899-7902,7904-7905,7908,7910-7918,7920-7926,7929-7946,7948-7957,7959,7961-7976,7978-8008,8010-8016,8018-8019,8021-8024,8026-8029,8031,8033-8045,8048-8049,8051-8083,8085-8087,8089-8114,8116-8130,8132-8134,8136-8146,8148-8152,8155-8158,8160-8163,8165-8177,8179-8191,8193-8195,8197-8199,8201-8205,8207-8215,8217-8219,8221-8222,8224-8259,8262-8263,8266-8282,8284-8287,8289-8292,8294,8296-8297,8300-8302,8304-8307,8310-8314,8316,8318-8335,8337-8339,8341-8356,8358-8359,8361-8391,8393-8399,8402,8404-8409,8412-8419,8421-8425,8427,8431-8433,8435-8439,8441-8443,8446,8448-8464,8466,8468-8469,8471-8478,8480-8497,8499-8500,8502-8504,8506-8507,8510-8511,8513-8517,8519-8522,8524-8525,8527-8531,8533,8538-8539,8541-8546,8548-8552,8554-8570,8572,8574-8576,8578,8580-8597,8599-8619,8621-8638,8640-8661,8663-8684,8686-8689,8691-8706,8708-8731,8733-8736,8738-8762,8764-8777,8779-8782,8785-8788,8790-8797,8799-8801,8803-8813,8815-8833,8835-8837,8839-8848,8851-8856,8858-8867,8869-8890,8892-8893,8895-8899,8901-8915,8919-8925,8927,8932,8936-8940,8942-8943,8946,8948-8949,8951-8952,8955,8957,8959-8967,8970-8973,8975-8978,8980-8990,8992-8997,8999-9015,9017-9018,9020-9023,9025-9028,9030-9032,9034-9038,9040-9041,9044-9053,9055,9057-9058,9060-9127,9129-9138,9140-9142,9144-9150,9152,9154-9169,9171,9173-9178,9181-9183,9185-9197,9199-9202,9205-9209,9211-9212,9214-9215,9218-9219,9221-9232,9234-9238,9240-9278,9280-9287,9289,9291,9293-9303,9305-9306,9308-9310,9313-9315,9317-9321,9323-9326,9328-9331,9333,9335,9337,9339-9341,9343-9349,9352-9353,9355,9357-9362,9364-9367,9370-9382,9384-9401,9404-9406,9408-9411,9413-9424,9426-9444,9446-9448,9450-9465,9468-9471,9473-9475,9477,9479-9483,9486-9489,9491-9500,9502-9530,9532-9533,9535-9553,9555-9595,9597-9601,9603-9604,9606-9623,9625-9646,9648-9675,9677-9681,9683-9690,9692-9693,9695-9699,9701-9737,9739-9803,9805-9825,9827-9850,9852-9868,9870-9900,9902-9938,9940-9964,9966-9981,9983-10012,10014-10080,10082-10092,10094-10097,10100-10109,10111-10123,10125-10140,10142-10158,10160-10174,10176-10212,10214-10217,10219-10226,10228-10234,10236-10238,10241-10251,10253-10267,10271-10278,10280-10284,10286-10300,10302-10303,10305-10314,10316-10318,10320-10321,10323-10327,10329-10337,10339-10344,10346-10359,10361-10366,10368-10393,10395-10424,10426-10433,10435-10463,10465-10467,10470-10480,10484-10498,10501-10548,10550-10575,10578-10589,10591-10705,10707-10722,10724-10731,10733,10735-10739,10741-10758,10760-10781,10783-10844,10846-10851,10853-10873,10875-10912,10914-10929,10931-10955,10957-10981,10983-10994,10996-11004,11006-11007,11009-11029,11031,11033-11039,11041-11054,11057-11058,11060-11079,11081-11086,11088-11089,11091-11098,11100-11101,11103-11104,11108-11110,11112-11117,11119-11121,11124-11125,11127-11130,11132-11135,11137,11140-11145,11147-11172,11174-11182,11184,11186-11187,11189-11226,11228-11230,11232-11236,11239-11331,11333-11341,11343-11360,11362-11366,11368-11492,11494-11504,11506-11547,11549-11582 Local adds: undefined Server subs: 1-9654 Local subs: undefined
Comment 9•16 years ago
|
||
That's helpful information, since it's clear that we're talking to google, but not storing things. Can anyone reproduce this on windows? I can't on mac, so I'm wondering if it's linux specific? I also wonder if we have a regression range, or if it's always been this way?
Updated•16 years ago
|
Flags: wanted1.9.0.x?
Flags: wanted1.9.0.x+
Flags: blocking1.9.0.1?
Whiteboard: [RC2?] → [RC2-]
Reporter | ||
Comment 10•16 years ago
|
||
To summarize, if you run Fx3rc1 on Linux for the first time on a new profile, the urlclassifier3.sqlite file doesn't seem to be updated in that session, even after several hours. You will get a warning for the testing malware/phishing pages, but you will not get a warning for other urls that are known to be phishy and which do trigger a warning on those systems where the safebrowsing data is 100% downloaded. If you restart your browser then urlclassifier3.sqlite file starts getting larger. Still, it takes a while for the system to detect those phishy urls. It took my linux system about 2 hours before it got to 30% of the data to be downloaded, and at this point realoading those urls triggered the warning. The file size not getting larger on first launch seems like a regression in Linux, something Fx3b4 and Fx3b5. After double checking I can see that in Windows and Mac this doesn't happen, so it looks like a problem in Linux only.
Assignee | ||
Comment 11•16 years ago
|
||
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/url-classifier/content/url-crypto-key-manager.js#270 replaceKey_() is using atob() to decode the client key sent by Google. The client keys aren't actually base64, they're "urlsafe base64", which is base64 with two of the characters replaced. So when google sends a key that includes one of these two characters, it gets confused and treats it as "no key" (which disables updates rather than fetching potentially-malicious updates). Once we get a "valid" key, we save it for 30 days (unless cookies are cleared). After that we have to fetch another. This is actually a problem in the Firefox 2 code too - people using "Advanced Mode" might be running in to this problem, disabling remote checks. Fix is simple, we just need to un-urlsafe the key before atob, patch coming up.
Assignee | ||
Comment 12•16 years ago
|
||
Updated•16 years ago
|
Attachment #323466 -
Flags: review?(tony) → review+
Reporter | ||
Comment 13•16 years ago
|
||
Tested fix on my linux vm using tryserver build: https://build.mozilla.org/tryserver-builds/2008-06-02_17:54-cbook@mozilla.com-Tomcat-Testing-Bug-434624/ With this build I consistently get a urlclassifier3key.txt file created in the profile and subsequent updates of the urlclassifier3.sqlite file on the first run.
Comment 14•16 years ago
|
||
(In reply to comment #11) > (...) > > Once we get a "valid" key, we save it for 30 days (unless cookies are cleared). > After that we have to fetch another. > (...) Why is this? Why are cookies and key related at all? And BTW: how about changing status of bug 368255 to WONTFIX (again)? It is obvious you are not going to fix that bug. (Hmm, I guess I should make some comment there...).
Updated•16 years ago
|
Flags: blocking1.9.0.1? → blocking1.9.0.1+
Updated•16 years ago
|
Flags: wanted1.8.1.x?
Comment 15•16 years ago
|
||
I did some ad-hoc testing and found that out of 4 test profiles I created, 2 of them failed to get a safebrowsing key on first run. I think the problem is well-understood, I just wanted to point out that it seems to happen fairly frequently (and will happen with the same probability on all platforms). If by some chance there's an opportunity to get this fix in for 3.0, that would really improve the new user experience with respect to phishing/malware protection.
Comment 16•16 years ago
|
||
Dave: what's the chance of regression here? Should we bake on mozilla-central first, or just land on both trees at once? Do we need this on 1.8.x?
Whiteboard: [RC2-] → [MU+]
Comment 17•16 years ago
|
||
Comment on attachment 323466 [details] [diff] [review] fix a=beltzner, please watch these branch builds for regressions
Attachment #323466 -
Flags: approval1.9.0.1+
Assignee | ||
Comment 18•16 years ago
|
||
Landed on mozilla-central as revision 39d0829ed9e2.
Assignee | ||
Comment 19•16 years ago
|
||
Landed on 1.9.0: Checking in url-crypto-key-manager.js; /cvsroot/mozilla/toolkit/components/url-classifier/content/url-crypto-key-manager.js,v <-- url-crypto-key-manager.js new revision: 1.14; previous revision: 1.13
Comment 20•16 years ago
|
||
juanb, please verify the fix when you get a chance. thanks.
Reporter | ||
Comment 21•16 years ago
|
||
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 Verified on XP, Vista, Mac OS X (10.5.4), and Ubuntu 8.04 by launching Fx3.0.1 several times using new profiles and checking that I got a urlclassifier3key.txt file, and that the urlclassifier3.sqlite file was updated within minutes of first launch, every time I tried.
Keywords: fixed1.9.0.1 → verified1.9.0.1
Updated•16 years ago
|
Status: RESOLVED → VERIFIED
Updated•10 years ago
|
Product: Firefox → Toolkit
You need to log in
before you can comment on or make changes to this bug.
Description
•