Closed Bug 434624 Opened 16 years ago Closed 16 years ago

safebrowsing file not updated on first launch

Categories

(Toolkit :: Safe Browsing, defect)

3.0 Branch
defect
Not set
critical

Tracking

()

VERIFIED FIXED

People

(Reporter: jbecerra, Assigned: dcamp)

References

Details

(Keywords: verified1.9.0.1, Whiteboard: [MU+])

Attachments

(2 files)

While testing Fx3rc1, the urlclassifier3.sqlite file does not get updated at all after the first start.

Steps:
1. Install Fx3rc1, or nightyl, using a fresh profile.
2. Locate the urlclassifier3.sqlite and see if gets updated from it's original 32kb size

Expected: This file should be updated within the first 5 minutes of launch.

Actual: It doesn't get updated, until you quit and start the browser after the very first launch.
Severity: normal → critical
Version: Trunk → 3.0 Branch
Actually, it looks like the urlclassifier3.sqlite file does get updated, however the OS does not report correctly the size of the file in the file explorer, in XP for example. So while the explorer lists the file initially as being 32k, if you get the properties from the context menu after a few minutes it shows a larger size, and it keeps getting larger. All the while, safebrowser warnings do show up for me on new installations on first run.
In Linux (ubuntu) the urlclassifier3.sqlite doesn't seem to be updated on first launch, and checking the file properties doesn't indicate that it is being updated.
If you are on linux and try out Fx3rc1 for the first time you might not be malware/phishing protected until you start a new session, so I'm nominating this for rc2.
Flags: blocking-firefox3?
Whiteboard: [RC2?]
Depends on: 435366
Dave's not around this week and he's the guy who will be able to say for sure, but the following information would be helpful:

1) Is this being seen after, say, 10 minutes running a full profile, or 2 hours?  The update scheduling has moved from a fixed interval to a random one, so it's possible that a pull wouldn't have happened after 10 minutes, that maybe would have happened after 12.  If no update is happening after a few hours though, that's clearly not the explanation here.

2) If we're sure that updates aren't being fetched even after significant run time, then we need to know whether the fetches are actually happening or not.  I recommend something like the LiveHTTPHeaders add-on to watch for update pings to google's safebrowsing servers.  

It would also be helpful to get Dave's about:safebrowsing addon installed (attachment 320610 [details]) and see if there are chunks reported.  The output is pretty raw, but if you look at the Local/Server ratios, you should see local chunk counts climbing towards 100% over time (you will likely have to reload the page though, it doesn't update dynamically).

Urr - stupid me - obviously you can't install addons on a first run, since they require a restart - so external tools may be necessary to watch for that traffic.  #1 above is still relevant.
Flags: wanted1.9.0.x?
Flags: blocking-firefox3?
Flags: blocking-firefox3-
Incidentally, for me on a fresh profile, OS/X 4 minutes after start:

[johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite 
Tue 27 May 2008 14:21:15 EDT
-rw-r--r--@ 1 johnath  johnath  32768 27 May 14:19 urlclassifier3.sqlite
[johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite 
Tue 27 May 2008 14:22:55 EDT
-rw-r--r--@ 1 johnath  johnath  4005888 27 May 14:22 urlclassifier3.sqlite
[johnath@ca-228 ~/Profiles/pib62mtd.dispo]$ date ; ls -l urlclassifier3.sqlite 
Tue 27 May 2008 14:24:02 EDT
-rw-r--r--@ 1 johnath  johnath  8814592 27 May 14:24 urlclassifier3.sqlite

... and so forth. It levels off at around 10M for that update, with another to
come during the next refresh cycle.
Ubuntu 8.04 vm. I installed Fx3rc1 afresh yesterday, and I left it running for maybe a half hour. The urlclassifier file did not get updated. Then I installed livehttpheaders (knowing this might void the bug report), and I left the browser running for several hours. Livehttpheaders shows a POST (with some random string) and a GET /safebrowsing/newkey?... and a few more GETs for the default rss feed in my bookmarks toolbar. The size of the urlclassifier file has not changed, after at least 2 hours.

I'll try again, this time installing livehttpeheaders and the about:safebrowsing extension at the same time, and I will leave it running for a few hours.
Attached file live http headers
i can confirm this Problem on Fedora F8, new Firefox Profile and running over an hour and the file size does not change ...urlclassifier3.sqlite is still  32.0 KB (32768 bytes) big/small

goog-malware-shavar
Server: 2768 add chunks
Local: 0 add chunks (0 percent)

Server adds: 8,10,29,38,40-42,63-86,88-115,118-126,128-134,136-140,142,144-148,150-159,161-168,170-177,179-180,184-186,188-190,192-195,198-211,213-244,246-251,253,262,269-299,301-308,310-313,329-331,333-346,348-366,368-413,415-416,418-419,421-424,426-474,476-532,534-535,537-539,541-583,585-589,591-608,610-613,615-617,619-623,625,627-631,634-636,639,645,647-648,650,652,663-664,666-667,669,674,680,684-685,687-688,692,699-700,705,708,713,716,723-724,726,730-731,741,749,751,754-755,760,764,767,771,777-780,782-784,786,789-790,793,796-799,801-804,806,808,812-814,816-817,820-821,824-826,829-831,833,836-839,841,843-845,847,849-850,852,854-857,861-864,869,871-873,875-878,884-885,888-889,891-893,895-898,900-903,909-914,919,921,923-932,934-941,944-946,949,953-954,958,962,964-967,970-971,974-977,979-981,985-987,989-991,993,1000-1001,1003,1005,1008-1009,1013,1019,1022-1023,1026-1027,1029-1031,1035-1038,1040,1049,1051,1053-1055,1063-1067,1069,1071-1072,1075-1077,1083,1087,1089,1093-1094,1096,1099-1101,1103,1106-1107,1110-1111,1115,1118-1119,1122,1126,1128-1129,1134,1139-1140,1146-1148,1150-1152,1154-1155,1157-1158,1162-1164,1170,1172,1175-1177,1179-1180,1182-1183,1185,1188,1192-1193,1197-1198,1200-1205,1207,1209,1212-1213,1216,1219,1222-1224,1226-1227,1229-1230,1233-1235,1238,1243-1244,1246,1250,1255-1256,1263,1265-1267,1270,1272-1273,1275-1276,1279,1281,1284-1285,1289-1290,1292-1293,1296,1299-1300,1303-1308,1310,1313-1314,1316-1324,1329,1333,1336,1343,1346,1348-1349,1351,1353,1355-1356,1359-1360,1362,1368,1374,1378-1387,1390-1391,1394-1395,1398-1399,1402-1403,1407,1411,1413,1416-1417,1419,1423,1425-1427,1431-1432,1434-1436,1438-1445,1448-1450,1452-1453,1455-1456,1458-1464,1466-1467,1469,1473,1477-1478,1482,1484,1486-1487,1490,1493,1497-1532,1535-1540,1542-1571,1573-1575,1577-1579,1581-1583,1585,1588-1592,1594,1597,1600,1607-1611,1618,1620-1627,1629-1631,1633,1693,1696,1726,1729,1731-1732,1738-1739,1742,1746-1749,1752,1755,1758,1773,1820,1823,1851,1879,1888,1911-1912,1915,1919,1932,1936,1950,1953-1954,1956,1961-1962,1964-1965,1968-1972,1976-1978,1981,1989,1992,2000,2002-2007,2010-2011,2017,2025,2027-2028,2031-2036,2038-2040,2042,2044-2047,2065-2067,2071-2072,2074,2085,2091,2093-2096,2098-2102,2104,2107-2115,2117-2133,2135-2136,2138-2139,2141,2143-2169,2183-2192,2194-2198,2200-2208,2210-2218,2221,2224,2227,2260,2330-2331,2372,2378,2384-2387,2390,2392,2395,2397-2398,2400,2428-2431,2433-2436,2438-2441,2445-2446,2448-2449,2451-2455,2457,2462,2466-2469,2471-2473,2475,2479-2486,2488-2496,2498-2502,2505,2508,2510,2512,2516,2518,2520-2535,2543-2546,2548-2551,2557,2561,2563-2565,2570-2578,2583,2585-2587,2589,2592-2593,2596-2607,2610,2615,2619-2622,2625-2627,2631-2632,2640-2649,2651,2653-2656,2664-2665,2668-2672,2674-2675,2679,2681,2683,2691-2692,2695-2700,2702-2721,2723-2726,2728-2736,2738,2742-2743,2745-2776,2778-2779,2781,2783,2785-2786,2790-2792,2794,2797-2978,2980-3186,3188-3190,3193-3197,3199-3205,3207-3208,3210-3211,3215,3218-3219,3221,3223,3227-3229,3231-3232,3234-3237,3240,3242-3243,3245-3246,3249-3250,3252-3256,3258,3260,3263-3264,3266,3268-3277,3279-3280,3284,3286,3289,3291-3295,3297,3299-3302,3305,3307-3311,3313-3326,3328-3331,3336-3338,3342,3345-3349,3351-3377,3379,3381-3383,3385-3391,3394-3399,3405-3420,3423-3424,3430,3432,3434,3436,3439-3742,3747,3749,3751,3753-3786,3788-3789,3791-3792,3794-3803,3805-3809,3813-3918,3920,3922,3924,3926-3927,3929-3933,3935-3939,3941,3943-3945,3947-3956,3960-3961,3963-3966,3968-3970,3972-3973,3976,3978-3981,3983-3984,3986-4003,4005,4007-4008,4011-4023,4025,4027-4028,4030-4032,4034-4045,4047-4062,4065,4068-4070,4073,4075-4080,4082-4083,4086-4095,4097-4100,4104-4106,4108-4114,4116-4122,4124-4127,4129-4141,4143-4149,4151,4154-4159,4161-4162,4164-4170,4172-4183,4185-4186,4188-4198,4200-4201,4203-4204,4206,4208-4219,4221-4236,4240-4246,4248-4256
Local adds: undefined
Server subs: 1-4035
Local subs: undefined
goog-phish-shavar
Server: 4183 add chunks
Local: 0 add chunks (0 percent)

Server adds: 27,60,79,90,105,3853,3997,4041-4043,4111,4152,4173-4174,4243,4258,4260,4275,4279,4305,4323,4330,4386,4415,4520,4556,4562,4660-4661,4669,4679-4680,4684,4814,4816,4822,4933,4938,4940,4961,4964,5068,5079,5081,5084,5110-5111,5116,5140,5193,5222,5236,5250,5313,5328,5351,5353,5358,5394-5395,5412,5444,5461-5463,5466-5467,5470,5475,5518-5519,5574,5587,5623,5714-5715,5730-5731,5734,5772,5810,5877,5939,5957,6039,6084,6089,6120,6122,6147,6165,6233,6280,6286,6364,6391,6407-6408,6429,6479,6508-6509,6537,6541,6551,6671,6680,6728,6741,6746,6754,6769,6775,6793,6863,6886,6902,6904,6907,6914,6916,6918,6943,7002-7003,7008,7010-7012,7033,7035,7039,7041-7042,7044,7046-7048,7057-7062,7065-7077,7080-7084,7087-7088,7090-7093,7095,7097-7106,7108-7110,7112-7116,7118-7121,7123,7126-7134,7136-7142,7144-7150,7152-7159,7161-7173,7175-7188,7191-7192,7194-7200,7202-7207,7209,7211-7221,7223-7226,7228-7230,7232-7234,7236-7246,7248-7252,7255-7259,7261-7262,7264-7276,7278-7280,7282-7284,7286-7291,7293-7300,7302-7328,7330-7336,7338-7341,7343-7347,7349-7369,7371-7375,7377-7379,7381-7390,7396-7401,7403-7404,7406,7409-7410,7412-7414,7417-7427,7429-7430,7433-7435,7437-7443,7445-7447,7449-7451,7453-7456,7458,7460,7462-7477,7480-7489,7491,7494-7495,7497-7498,7500-7501,7503,7505,7507-7510,7512-7513,7515-7520,7522-7531,7533-7537,7540-7549,7551-7552,7554,7556,7558-7561,7563-7585,7587-7590,7593-7595,7597,7599-7608,7610-7612,7614-7616,7618-7624,7626,7628-7635,7637-7638,7640,7642-7648,7650-7653,7655-7660,7662-7667,7669-7682,7684,7686-7708,7710-7723,7726-7750,7752-7757,7759-7764,7768-7783,7786-7788,7791-7805,7807-7809,7811-7813,7815-7821,7824-7841,7843-7857,7859-7870,7872,7874-7876,7878-7879,7881,7883-7891,7893-7897,7899-7902,7904-7905,7908,7910-7918,7920-7926,7929-7946,7948-7957,7959,7961-7976,7978-8008,8010-8016,8018-8019,8021-8024,8026-8029,8031,8033-8045,8048-8049,8051-8083,8085-8087,8089-8114,8116-8130,8132-8134,8136-8146,8148-8152,8155-8158,8160-8163,8165-8177,8179-8191,8193-8195,8197-8199,8201-8205,8207-8215,8217-8219,8221-8222,8224-8259,8262-8263,8266-8282,8284-8287,8289-8292,8294,8296-8297,8300-8302,8304-8307,8310-8314,8316,8318-8335,8337-8339,8341-8356,8358-8359,8361-8391,8393-8399,8402,8404-8409,8412-8419,8421-8425,8427,8431-8433,8435-8439,8441-8443,8446,8448-8464,8466,8468-8469,8471-8478,8480-8497,8499-8500,8502-8504,8506-8507,8510-8511,8513-8517,8519-8522,8524-8525,8527-8531,8533,8538-8539,8541-8546,8548-8552,8554-8570,8572,8574-8576,8578,8580-8597,8599-8619,8621-8638,8640-8661,8663-8684,8686-8689,8691-8706,8708-8731,8733-8736,8738-8762,8764-8777,8779-8782,8785-8788,8790-8797,8799-8801,8803-8813,8815-8833,8835-8837,8839-8848,8851-8856,8858-8867,8869-8890,8892-8893,8895-8899,8901-8915,8919-8925,8927,8932,8936-8940,8942-8943,8946,8948-8949,8951-8952,8955,8957,8959-8967,8970-8973,8975-8978,8980-8990,8992-8997,8999-9015,9017-9018,9020-9023,9025-9028,9030-9032,9034-9038,9040-9041,9044-9053,9055,9057-9058,9060-9127,9129-9138,9140-9142,9144-9150,9152,9154-9169,9171,9173-9178,9181-9183,9185-9197,9199-9202,9205-9209,9211-9212,9214-9215,9218-9219,9221-9232,9234-9238,9240-9278,9280-9287,9289,9291,9293-9303,9305-9306,9308-9310,9313-9315,9317-9321,9323-9326,9328-9331,9333,9335,9337,9339-9341,9343-9349,9352-9353,9355,9357-9362,9364-9367,9370-9382,9384-9401,9404-9406,9408-9411,9413-9424,9426-9444,9446-9448,9450-9465,9468-9471,9473-9475,9477,9479-9483,9486-9489,9491-9500,9502-9530,9532-9533,9535-9553,9555-9595,9597-9601,9603-9604,9606-9623,9625-9646,9648-9675,9677-9681,9683-9690,9692-9693,9695-9699,9701-9737,9739-9803,9805-9825,9827-9850,9852-9868,9870-9900,9902-9938,9940-9964,9966-9981,9983-10012,10014-10080,10082-10092,10094-10097,10100-10109,10111-10123,10125-10140,10142-10158,10160-10174,10176-10212,10214-10217,10219-10226,10228-10234,10236-10238,10241-10251,10253-10267,10271-10278,10280-10284,10286-10300,10302-10303,10305-10314,10316-10318,10320-10321,10323-10327,10329-10337,10339-10344,10346-10359,10361-10366,10368-10393,10395-10424,10426-10433,10435-10463,10465-10467,10470-10480,10484-10498,10501-10548,10550-10575,10578-10589,10591-10705,10707-10722,10724-10731,10733,10735-10739,10741-10758,10760-10781,10783-10844,10846-10851,10853-10873,10875-10912,10914-10929,10931-10955,10957-10981,10983-10994,10996-11004,11006-11007,11009-11029,11031,11033-11039,11041-11054,11057-11058,11060-11079,11081-11086,11088-11089,11091-11098,11100-11101,11103-11104,11108-11110,11112-11117,11119-11121,11124-11125,11127-11130,11132-11135,11137,11140-11145,11147-11172,11174-11182,11184,11186-11187,11189-11226,11228-11230,11232-11236,11239-11331,11333-11341,11343-11360,11362-11366,11368-11492,11494-11504,11506-11547,11549-11582
Local adds: undefined
Server subs: 1-9654
Local subs: undefined
That's helpful information, since it's clear that we're talking to google, but not storing things.

Can anyone reproduce this on windows?  I can't on mac, so I'm wondering if it's linux specific?  I also wonder if we have a regression range, or if it's always been this way?
Flags: wanted1.9.0.x?
Flags: wanted1.9.0.x+
Flags: blocking1.9.0.1?
Whiteboard: [RC2?] → [RC2-]
To summarize, if you run Fx3rc1 on Linux for the first time on a new profile, the urlclassifier3.sqlite file doesn't seem to be updated in that session, even after several hours. You will get a warning for the testing malware/phishing pages, but you will not get a warning for other urls that are known to be phishy and which do trigger a warning on those systems where the safebrowsing data is 100% downloaded.

If you restart your browser then urlclassifier3.sqlite file starts getting larger. Still, it takes a while for the system to detect those phishy urls. It took my linux system about 2 hours before it got to 30% of the data to be downloaded, and at this point realoading those urls triggered the warning.

The file size not getting larger on first launch seems like a regression in Linux, something Fx3b4 and Fx3b5. After double checking I can see that in Windows and Mac this doesn't happen, so it looks like a problem in Linux only.
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/url-classifier/content/url-crypto-key-manager.js#270

replaceKey_() is using atob() to decode the client key sent by Google.  The client keys aren't actually base64, they're "urlsafe base64", which is base64 with two of the characters replaced.

So when google sends a key that includes one of these two characters, it gets confused and treats it as "no key" (which disables updates rather than fetching potentially-malicious updates).

Once we get a "valid" key, we save it for 30 days (unless cookies are cleared).  After that we have to fetch another.

This is actually a problem in the Firefox 2 code too - people using "Advanced Mode" might be running in to this problem, disabling remote checks.

Fix is simple, we just need to un-urlsafe the key before atob, patch coming up.

Attached patch fixSplinter Review
Assignee: nobody → dcamp
Status: NEW → ASSIGNED
Attachment #323466 - Flags: review?(tony)
Attachment #323466 - Flags: review?(tony) → review+
Tested fix on my linux vm using tryserver build: https://build.mozilla.org/tryserver-builds/2008-06-02_17:54-cbook@mozilla.com-Tomcat-Testing-Bug-434624/

With this build I consistently get a urlclassifier3key.txt file created in the profile and subsequent updates of the urlclassifier3.sqlite file on the first run.
(In reply to comment #11)
> (...)
> 
> Once we get a "valid" key, we save it for 30 days (unless cookies are cleared).
>  After that we have to fetch another.
> (...)

Why is this? Why are cookies and key related at all?
And BTW: how about changing status of bug 368255 to WONTFIX (again)? It is obvious you are not going to fix that bug. (Hmm, I guess I should make some comment there...).
Flags: blocking1.9.0.1? → blocking1.9.0.1+
Flags: wanted1.8.1.x?
I did some ad-hoc testing and found that out of 4 test profiles I created, 2 of them failed to get a safebrowsing key on first run.  I think the problem is well-understood, I just wanted to point out that it seems to happen fairly frequently (and will happen with the same probability on all platforms).  If by some chance there's an opportunity to get this fix in for 3.0, that would really improve the new user experience with respect to phishing/malware protection.
Dave: what's the chance of regression here? Should we bake on mozilla-central first, or just land on both trees at once?

Do we need this on 1.8.x?
Whiteboard: [RC2-] → [MU+]
Comment on attachment 323466 [details] [diff] [review]
fix

a=beltzner, please watch these branch builds for regressions
Attachment #323466 - Flags: approval1.9.0.1+
Landed on mozilla-central as revision 39d0829ed9e2.
Landed on 1.9.0:

Checking in url-crypto-key-manager.js;
/cvsroot/mozilla/toolkit/components/url-classifier/content/url-crypto-key-manager.js,v  <--  url-crypto-key-manager.js
new revision: 1.14; previous revision: 1.13
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Keywords: fixed1.9.0.1
Resolution: --- → FIXED
juanb, please verify the fix when you get a chance. thanks.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

Verified on XP, Vista, Mac OS X (10.5.4), and Ubuntu 8.04 by launching Fx3.0.1 several times using new profiles and checking that I got a urlclassifier3key.txt file, and that the urlclassifier3.sqlite file was updated within minutes of first launch, every time I tried.
Status: RESOLVED → VERIFIED
Product: Firefox → Toolkit
You need to log in before you can comment on or make changes to this bug.