Closed
Bug 1305801
Opened 9 years ago
Closed 9 years ago
Store V4 update data to disk
Categories
(Toolkit :: Safe Browsing, defect, P2)
Toolkit
Safe Browsing
Tracking
()
RESOLVED
FIXED
mozilla52
| Tracking | Status | |
|---|---|---|
| firefox52 | --- | fixed |
People
(Reporter: dimi, Assigned: dimi)
References
Details
(Whiteboard: #sbv4-m1)
Attachments
(6 files, 4 obsolete files)
|
58 bytes,
text/x-review-board-request
|
francois
:
review+
|
Details |
|
58 bytes,
text/x-review-board-request
|
francois
:
review+
gcp
:
review+
|
Details |
|
58 bytes,
text/x-review-board-request
|
francois
:
review+
gcp
:
review+
|
Details |
|
58 bytes,
text/x-review-board-request
|
francois
:
review+
gcp
:
review+
|
Details |
|
58 bytes,
text/x-review-board-request
|
gcp
:
review+
|
Details |
|
4.55 KB,
patch
|
Details | Diff | Splinter Review |
To store v4 update data, we have to
1. Support variable length PrefixSet (bug 1283007)
2. Support full update and write variable length prefixes to disk (bug 1283009)
3. Support partial update (bug 1287058)
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Assignee | ||
Comment 6•9 years ago
|
||
Comment on attachment 8795477 [details]
Bug 1305801 - P2. Add telemetry for variable-length prefix set.
This patch is already r+ in bug 1283007
Attachment #8795477 -
Flags: review?(gpascutto)
Attachment #8795477 -
Flags: review?(francois)
Attachment #8795477 -
Flags: review+
| Assignee | ||
Comment 7•9 years ago
|
||
Comment on attachment 8795478 [details]
Bug 1305801 - P2. Add telemetry for variable-length prefix set.
This patch is already r+ in bug 1283007
Attachment #8795478 -
Flags: review?(francois) → review+
| Assignee | ||
Comment 8•9 years ago
|
||
Comment on attachment 8795479 [details]
Bug 1305801 - P3. Testcase for variable length prefixSet.
This patch is already r+ in bug 1283007
Attachment #8795479 -
Flags: review?(gpascutto)
Attachment #8795479 -
Flags: review?(francois)
Attachment #8795479 -
Flags: review+
| Assignee | ||
Comment 9•9 years ago
|
||
Comment on attachment 8795480 [details]
Bug 1305801 - P4. Store variable-length prefix to disk.
This patch is already r+ in bug 1283009
Attachment #8795480 -
Flags: review?(gpascutto)
Attachment #8795480 -
Flags: review?(francois)
Attachment #8795480 -
Flags: review+
| Assignee | ||
Comment 10•9 years ago
|
||
Comment on attachment 8795481 [details]
Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
This patch is already r+ in bug 1287058
Attachment #8795481 -
Flags: review?(gpascutto) → review+
Comment 11•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795477 [details]
Bug 1305801 - P2. Add telemetry for variable-length prefix set.
https://reviewboard.mozilla.org/r/81510/#review80082
Carrying existing r+
Attachment #8795477 -
Flags: review+
Comment 12•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795478 [details]
Bug 1305801 - P2. Add telemetry for variable-length prefix set.
https://reviewboard.mozilla.org/r/81512/#review80084
::: toolkit/components/telemetry/Histograms.json:3769
(Diff revision 1)
> "n_buckets": 15,
> "description": "Time spent constructing PrefixSet from DB (ms)"
> },
> + "URLCLASSIFIER_VLPS_FILELOAD_TIME": {
> + "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> + "expires_in_version": "56",
Let's make these 58 since Nightly is now up to 52.
::: toolkit/components/telemetry/Histograms.json:3778
(Diff revision 1)
> + "bug_numbers": [1283007],
> + "description": "Time spent loading Variable-Length PrefixSet from file (ms)"
> + },
> + "URLCLASSIFIER_VLPS_FALLOCATE_TIME": {
> + "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> + "expires_in_version": "56",
ditto
Attachment #8795478 -
Flags: review?(francois)
Comment 13•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795479 [details]
Bug 1305801 - P3. Testcase for variable length prefixSet.
https://reviewboard.mozilla.org/r/81514/#review80086
Carrying previous r+
Attachment #8795479 -
Flags: review+
Comment 14•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795480 [details]
Bug 1305801 - P4. Store variable-length prefix to disk.
https://reviewboard.mozilla.org/r/81516/#review80092
Carrying prior r+
Attachment #8795480 -
Flags: review+
| Comment hidden (mozreview-request) |
| Assignee | ||
Updated•9 years ago
|
Attachment #8795478 -
Attachment is obsolete: true
Attachment #8795478 -
Flags: review?(francois)
| Assignee | ||
Updated•9 years ago
|
Attachment #8795479 -
Attachment is obsolete: true
| Assignee | ||
Updated•9 years ago
|
Attachment #8795480 -
Attachment is obsolete: true
| Assignee | ||
Updated•9 years ago
|
Attachment #8795481 -
Attachment is obsolete: true
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Assignee | ||
Updated•9 years ago
|
Attachment #8795492 -
Flags: review?(gpascutto)
| Assignee | ||
Updated•9 years ago
|
Attachment #8795493 -
Flags: review?(gpascutto)
| Assignee | ||
Updated•9 years ago
|
Attachment #8795494 -
Flags: review?(gpascutto)
Comment 20•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795493 [details]
Bug 1305801 - P3. Testcase for variable length prefixSet.
https://reviewboard.mozilla.org/r/81538/#review80096
Carrying prior r+
Attachment #8795493 -
Flags: review?(francois) → review+
Comment 21•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795494 [details]
Bug 1305801 - P4. Store variable-length prefix to disk.
https://reviewboard.mozilla.org/r/81540/#review80098
Carrying prior r+
Attachment #8795494 -
Flags: review?(francois) → review+
Comment 22•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795492 [details]
Bug 1305801 - P1. Implement variable length PrefixSet class for Safe Browsing v4.
https://reviewboard.mozilla.org/r/81536/#review80100
Carrying prior r+
Attachment #8795492 -
Flags: review?(francois) → review+
| Assignee | ||
Comment 23•9 years ago
|
||
Hi gcp,
Since we want to land bug 1283007, bug 1283009 and bug 1257058 together, so I move those patches to this new bug. Could you help review the last part again ? Attach the interdiff(compare with the patch you have reviewed before) because there are some modifications according to francois's suggestion in Bug 1287058 Comment 25.
Thanks for help!
Flags: needinfo?(gpascutto)
| Assignee | ||
Updated•9 years ago
|
Attachment #8795509 -
Attachment description: interdiff.patch → P5 interdiff
Comment 27•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795495 [details]
Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
https://reviewboard.mozilla.org/r/81542/#review80292
Francois asked:
- In the case of a prefix being in the add list and also in the original prefix set, we may want to issue a warning and to ignore it instead of returning an error. Perhaps with telemetry as well.
The Telemetry part wasn't addressed. I think it would be good to add a telemetry boolean (or enum for failing in the different update parts?) whenever we encounter a failure to update. If the algorithm fails somewhere in the field we won't otherwhise know.
::: toolkit/components/url-classifier/LookupCacheV4.cpp:159
(Diff revision 1)
> + nsDependentCSubstring smallestOldPrefix;
> + nsDependentCSubstring smallestAddPrefix;
> +
> + // This is used to avoid infinite loop for partial update algorithm.
> + uint32_t index = MAX_LOOPS;
> + for(;index > 0; index--) {
I know francois asked to make sure this loop doesn't get infinite, but these magic numbers here are way too ugly.
As I understand, every loop must process either an element from oldPSet or one from addPSet. That means that the maximum number of iterations here would be the sum of their sizes, plus or minus one due to loop boundary conditions.
So I suggest replacing the magic number here by the amount of times you expect the loop to run. If you break out of the loop due to hitting that maximum, there's been an error somewhere.
::: toolkit/components/url-classifier/tests/gtest/TestUrlClassifierTableUpdateV4.cpp:281
(Diff revision 1)
> + CreateRandomSortedPrefixArray(5000, 4, 4, pArray);
> + CreateRandomSortedPrefixArray(1000, 5, 32, pArray);
> + RemoveIntersection(fArray, pArray);
> + PrefixArrayToPrefixStringMap(pArray, pMap);
> +
> + // Remove 1/5 of elemets of original prefix set.
...elements...
This is misspelled throughout the file, so do a search & replace.
Attachment #8795495 -
Flags: review?(gpascutto) → review-
Updated•9 years ago
|
Flags: needinfo?(gpascutto)
| Assignee | ||
Comment 28•9 years ago
|
||
(In reply to Gian-Carlo Pascutto [:gcp] from comment #27)
>
> The Telemetry part wasn't addressed. I think it would be good to add a
> telemetry boolean (or enum for failing in the different update parts?)
> whenever we encounter a failure to update. If the algorithm fails somewhere
> in the field we won't otherwhise know.
>
We discussed after that and we think maybe a warning message is good enough for now, so that's why i didn't add telemetry.
But we all agree add telemetry will be useful so i will add it in this patch.
>
> So I suggest replacing the magic number here by the amount of times you
> expect the loop to run. If you break out of the loop due to hitting that
> maximum, there's been an error somewhere.
>
That's a really a good idea, thank you!
| Comment hidden (mozreview-request) |
Comment 30•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795495 [details]
Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
https://reviewboard.mozilla.org/r/81542/#review80458
::: toolkit/components/telemetry/Histograms.json:3773
(Diff revision 2)
> "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> "expires_in_version": "58",
> "kind": "exponential",
> "high": 1000,
> "n_buckets": 10,
> - "bug_numbers": [1283007],
> + "bug_numbers": [1305801],
I think it would be better to keep the old bug number because that's where we explained why we need to collect this data.
::: toolkit/components/telemetry/Histograms.json:3782
(Diff revision 2)
> "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> "expires_in_version": "58",
> "kind": "exponential",
> "high": 1000,
> "n_buckets": 10,
> - "bug_numbers": [1283007],
> + "bug_numbers": [1305801],
ditto
::: toolkit/components/telemetry/Histograms.json:3828
(Diff revision 2)
> },
> + "URLCLASSIFIER_UPDATE_ERROR_TYPE": {
> + "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> + "expires_in_version": "58",
> + "kind": "enumerated",
> + "n_values": 3,
It would be better to use a larger number here because we can never change the `n_values` once we add it. That way if we need to add other types of errors later, we have room to do it. Maybe use 10?
::: toolkit/components/telemetry/Histograms.json:3830
(Diff revision 2)
> + "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> + "expires_in_version": "58",
> + "kind": "enumerated",
> + "n_values": 3,
> + "bug_numbers": [1305801],
> + "description": " (0=DUPLICATE PREFIX, 1=INFINITE LOOP, 2=WRONG REMOVAL INDICES)"
"An error was encountered while parsing a partial update returned by a Safe Browsing V4 server (0 = addition of an already existing prefix, 1 = parser got into an infinite loop, 2 = removal index out of bounds)"
::: toolkit/components/url-classifier/LookupCacheV4.cpp:216
(Diff revision 2)
> + DUPLICATE_PREFIX);
> + return NS_ERROR_FAILURE;
> + }
> + }
> +
> + // We expecet index will be greater or equal to 0 because number of maximum runs will be
typo: "expect"
"number of maximum runs" => "max number of runs"
Attachment #8795495 -
Flags: review?(francois)
| Comment hidden (mozreview-request) |
| Assignee | ||
Comment 32•9 years ago
|
||
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
Comment 38•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795495 [details]
Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
https://reviewboard.mozilla.org/r/81542/#review81494
::: toolkit/components/url-classifier/LookupCacheV4.cpp:162
(Diff revisions 1 - 4)
> nsDependentCSubstring smallestOldPrefix;
> nsDependentCSubstring smallestAddPrefix;
>
> // This is used to avoid infinite loop for partial update algorithm.
> - uint32_t index = MAX_LOOPS;
> - for(;index > 0; index--) {
> + int32_t index = oldPSet.Count() + addPSet.Count();
> + for(;index >= -1; index--) {
Why not do the +1 on the previous line? (And maybe, comment where it comes from! Right now -1 is just a magic number)
It's even weirder because the error check below is also <= -1. So if the for loop check here is on the -1 condition, we're already in an error state? So why not break out immediately instead of continuing to -2?
::: toolkit/components/url-classifier/LookupCacheV4.cpp:218
(Diff revisions 1 - 4)
> }
> }
>
> - if (index <= 0) {
> - NS_WARNING("There are too many prefxies than expected.");
> + // We expect index will be greater or equal to 0 because max number of runs will be
> + // the number of original prefix + add prefix.
> + if (index <= -1) {
so it's if (index < 0)
::: toolkit/components/url-classifier/LookupCacheV4.cpp:241
(Diff revisions 1 - 4)
> + : mCount(0)
> {
> for (auto iter = aMap.ConstIter(); !iter.Done(); iter.Next()) {
> uint32_t size = iter.Key();
> mMap.Put(size, new PrefixString(*iter.Data(), size));
> + mCount += iter.Data()->Length() / size;
Is this not equivalent to mCount++?
::: toolkit/components/url-classifier/LookupCacheV4.cpp:251
(Diff revisions 1 - 4)
> + : mCount(0)
> {
> for (auto iter = aMap.ConstIter(); !iter.Done(); iter.Next()) {
> uint32_t size = iter.Key();
> mMap.Put(size, new PrefixString(iter.Data()->GetPrefixString(), size));
> + mCount += iter.Data()->GetPrefixString().Length() / size;
Same question.
::: toolkit/components/url-classifier/Classifier.cpp:979
(Diff revision 4)
> nsCString* prefix = new nsCString(iter.Data()->GetPrefixString());
> - prefixes.Put(iter.Key(), prefix);
> + output->Put(iter.Key(), prefix);
> }
> } else {
> - // TODO: Bug 1287058, partial update
> + PrefixStringMap* input = nullptr;
> + // If both prefix set is empty, this means we are doing a partial update
...if both prefix sets are...
Attachment #8795495 -
Flags: review?(gpascutto) → review+
Comment 39•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795492 [details]
Bug 1305801 - P1. Implement variable length PrefixSet class for Safe Browsing v4.
https://reviewboard.mozilla.org/r/81536/#review81486
Attachment #8795492 -
Flags: review?(gpascutto) → review+
Comment 40•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795493 [details]
Bug 1305801 - P3. Testcase for variable length prefixSet.
https://reviewboard.mozilla.org/r/81538/#review81490
Attachment #8795493 -
Flags: review?(gpascutto) → review+
Comment 41•9 years ago
|
||
| mozreview-review | ||
Comment on attachment 8795494 [details]
Bug 1305801 - P4. Store variable-length prefix to disk.
https://reviewboard.mozilla.org/r/81540/#review81492
Attachment #8795494 -
Flags: review?(gpascutto) → review+
| Assignee | ||
Comment 42•9 years ago
|
||
Thanks for review!
(In reply to Gian-Carlo Pascutto [:gcp] from comment #38)
> Comment on attachment 8795495 [details]
> Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
>
> https://reviewboard.mozilla.org/r/81542/#review81494
>
> ::: toolkit/components/url-classifier/LookupCacheV4.cpp:162
> (Diff revisions 1 - 4)
> > nsDependentCSubstring smallestOldPrefix;
> > nsDependentCSubstring smallestAddPrefix;
> >
> > // This is used to avoid infinite loop for partial update algorithm.
> > - uint32_t index = MAX_LOOPS;
> > - for(;index > 0; index--) {
> > + int32_t index = oldPSet.Count() + addPSet.Count();
> > + for(;index >= -1; index--) {
>
> Why not do the +1 on the previous line? (And maybe, comment where it comes
> from! Right now -1 is just a magic number)
>
I'll change to +1. I use -- in this patch is only because then i don't have to write
"oldPSet.Count() + addPSet.Count()" again in error check statement.
>
> ::: toolkit/components/url-classifier/LookupCacheV4.cpp:241
> (Diff revisions 1 - 4)
> > + : mCount(0)
> > {
> > for (auto iter = aMap.ConstIter(); !iter.Done(); iter.Next()) {
> > uint32_t size = iter.Key();
> > mMap.Put(size, new PrefixString(*iter.Data(), size));
> > + mCount += iter.Data()->Length() / size;
>
> Is this not equivalent to mCount++?
>
No, mCount calculates the total prefixes we have, and for prefixes with the same prefix size will be
stored in |iter.Data| as a large string. So string length/prefix size will be the number of prefixes for a given prefix size.
| Comment hidden (mozreview-request) |
| Assignee | ||
Comment 44•9 years ago
|
||
| Assignee | ||
Comment 45•9 years ago
|
||
(In reply to Dimi Lee[:dimi][:dlee] from comment #42)
> >
> > Why not do the +1 on the previous line? (And maybe, comment where it comes
> > from! Right now -1 is just a magic number)
> >
>
> I'll change to +1. I use -- in this patch is only because then i don't have
> to write
> "oldPSet.Count() + addPSet.Count()" again in error check statement.
>
ah, sorry i found i mis-understood what you are saying first time.
I will fix this...
| Comment hidden (mozreview-request) |
| Assignee | ||
Comment 47•9 years ago
|
||
try result is in Comment 44
| Assignee | ||
Updated•9 years ago
|
Keywords: checkin-needed
Comment 48•9 years ago
|
||
Please mark the unresolved issues with Part 5 in MozReview as resolved so that this can autoland.
Keywords: checkin-needed
| Assignee | ||
Comment 49•9 years ago
|
||
All issues are marked as fixed. Could you help land it, thanks.
Flags: needinfo?(ryanvm)
Updated•9 years ago
|
Flags: needinfo?(ryanvm)
Comment 50•9 years ago
|
||
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/14457cc4c325
Part 1: Implement variable length PrefixSet class for Safe Browsing v4. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/56ceae52d847
Part 2: Add telemetry for variable-length prefix set. r=francois
https://hg.mozilla.org/integration/autoland/rev/def8da367beb
Part 3: Testcase for variable length prefixSet. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/bca0e706dbc5
Part 4: Store variable-length prefix to disk. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/0c95d5dec6d9
Part 5: Support SafeBrowsing v4 partial update. r=gcp
Comment 51•9 years ago
|
||
Backed out in https://hg.mozilla.org/integration/autoland/rev/aa1675d1dcf6633e1fcc6921090ce253be2c23ff for ASan gtest heap-buffer-overflows in VariableLengthPrefixSet.VariableLengthSet a la https://treeherder.mozilla.org/logviewer.html#?job_id=4404877&repo=autoland#L7615
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Comment hidden (mozreview-request) |
| Assignee | ||
Comment 57•9 years ago
|
||
Fix the try error:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3432a293f98e&selectedJob=28504105
The root cause is memcmp in P1 may out of bound in certain case.
Keywords: checkin-needed
Comment 58•9 years ago
|
||
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/39a5fc72213d
Part 1: Implement variable length PrefixSet class for Safe Browsing v4. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/4b01da3f451a
Part 2: Add telemetry for variable-length prefix set. r=francois
https://hg.mozilla.org/integration/autoland/rev/94d8f8d4e3d5
Part 3: Testcase for variable length prefixSet. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/122db2c234f1
Part 4: Store variable-length prefix to disk. r=francois, r=gcp
https://hg.mozilla.org/integration/autoland/rev/2d4f6c10aacf
Part 5: Support SafeBrowsing v4 partial update. r=gcp
Keywords: checkin-needed
Comment 59•9 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/39a5fc72213d
https://hg.mozilla.org/mozilla-central/rev/4b01da3f451a
https://hg.mozilla.org/mozilla-central/rev/94d8f8d4e3d5
https://hg.mozilla.org/mozilla-central/rev/122db2c234f1
https://hg.mozilla.org/mozilla-central/rev/2d4f6c10aacf
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
status-firefox52:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Comment 60•9 years ago
|
||
(In reply to François Marier [:francois] from comment #30)
> Comment on attachment 8795495 [details]
> Bug 1305801 - P5. Supports SafeBrowsing v4 partial update.
>
> https://reviewboard.mozilla.org/r/81542/#review80458
>
> ::: toolkit/components/telemetry/Histograms.json:3773
> (Diff revision 2)
> > "alert_emails": ["safebrowsing-telemetry@mozilla.org"],
> > "expires_in_version": "58",
> > "kind": "exponential",
> > "high": 1000,
> > "n_buckets": 10,
> > - "bug_numbers": [1283007],
> > + "bug_numbers": [1305801],
>
> I think it would be better to keep the old bug number because that's where
> we explained why we need to collect this data.
Or just add both in these cases - it's an array to allow for adding multiple related bugs.
You need to log in
before you can comment on or make changes to this bug.
Description
•