Certain Canvas Telemetry is wrong
Categories
(Core :: Privacy: Anti-Tracking, defect, P3)
Tracking
()
People
(Reporter: tjr, Assigned: tjr)
References
Details
Attachments
(18 files)
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review |
We report telemetry here. The first if statement seems to say "If we didn't see any canvas fingerprinting, just log some canvas fingerprinting as 'no known text and unknown variant'.
However we will hit this case if we loop through all the content blocking logs and see no canvas fingerprinting logs. We should report telemetry if we saw nothing.
Conceptually, 'no known text and unknown variant' is a scenario we do nothing for so Unknown_none is an invalid value for telemetry.
I guess the thing I haven't quite figured out is why we don't report bajillions of these unknown_none events, because it seems like we should log one of these events everytime we go into ContentBlockingLog::ReportCanvasFingerprintingLog() which is every WindowGlobalPArent::ActorDestroy() (ref).
| Assignee | ||
Comment 1•2 years ago
•
|
||
I wonder if the reason we're seeing X million unknown_none events, compared to Y billion page loads, is because the numerator and denominator are out of sync. We would only ever report 1 entry for all the content blocking actions taken on a top-level document. So if a Document loaded 15 iframes, would that be 16 page loads but only one possible event?
That seems unlikely, because if we were truely seeing one unknown_none event per WindowGlobalParent::ActorDestroy(), that would imply there were ~227 page loads per WindowGlobalParent. AFAIK WindowGlobalParent is unique per top-level document, so that seems like too many...
Making MoCo only because I don't know if these stats are private. Make this comment private before unhiding if so.
Comment 2•2 years ago
|
||
Tom, would you take a look at this? Thanks
Comment 3•2 years ago
•
|
||
I wonder if the reason we're seeing XX million unknown_none events, compared to 5 billion page loads
Okay, XX million seems like a really low number, I agree.
So if a Document loaded 15 iframes, would that be 16 page loads but only one possible event?
Yes correct. This event is basically per-tab.
That seems unlikely, because if we were truely seeing one unknown_none event per WindowGlobalParent::ActorDestroy(), that would imply there were ~227 page loads per WindowGlobalParent
Do note we don't log anything for non-content principals, so if you e.g. count about:newtab, that could maybe partly explain the difference.
Have you tried comparing this to the STATE_BLOCKED_EMAILTRACKING_CONTENT telemetry that already existed before and I think works similarly?
| Assignee | ||
Comment 4•2 years ago
|
||
https://sql.telemetry.mozilla.org/queries/96829/source#238995 is the query I use to explore (all) the canvas telemetry
Comment 5•2 years ago
•
|
||
Just to share some thoughts here, in case they got lost in Slack:
- In the sql query, we're getting page loads from 100% of users and all window types/ETP settings (via use counters)
- However, the subquery
sampleto get the canvas data works on a 1% sample. So XX million are probably more like XX billion - Another difference is that the canvas telemetry probe is collected per tab, not per page load.
- Lastly, I'm wondering: are we collecting canvas data on every page, or does it need to have at least a canvas element present in order to count even as unknown_none? If the latter, only ~45% of pages even use a canvas element (according to chromestatus)
| Assignee | ||
Comment 6•1 year ago
•
|
||
I ran into this issue again and am revisiting it. I am using https://sql.telemetry.mozilla.org/queries/96083/source#237307 as the source of data, and we see the Unknown_none on 10-13% of page loads.
(In reply to Martin Balfanz [:mbalfanz] from comment #5)
Just to share some thoughts here, in case they got lost in Slack:
- In the sql query, we're getting page loads from 100% of users and all window types/ETP settings (via use counters)
In my query, yes this is what I do.
- However, the subquery
sampleto get the canvas data works on a 1% sample. So XX million are probably more like XX billion
I had fixed this, and I query from telemetry.main
- Another difference is that the canvas telemetry probe is collected per tab, not per page load.
Per tab meaning... if you use a tab to visit 10 pages on 3 origins and then close the tab - that's only one entry? That seems crazy....
- Lastly, I'm wondering: are we collecting canvas data on every page, or does it need to have at least a canvas element present in order to count even as unknown_none? If the latter, only ~45% of pages even use a canvas element (according to chromestatus)
This may be the culprit. We only record things if there is a canvas extraction.
| Assignee | ||
Comment 7•1 year ago
|
||
(In reply to Tom Ritter [:tjr] from comment #6)
I ran into this issue again and am revisiting it. I am using https://sql.telemetry.mozilla.org/queries/96083/source#237307 as the source of data, and we see the Unknown_none on 10-13% of page loads.
- Lastly, I'm wondering: are we collecting canvas data on every page, or does it need to have at least a canvas element present in order to count even as unknown_none? If the latter, only ~45% of pages even use a canvas element (according to chromestatus)
This may be the culprit. We only record things if there is a canvas extraction.
This is not the culprit. While this is true, we still call ReportCanvasFingerprintingLog on every ActorDestroy and that function will always log a telemetry item for any non-content principal. So we need to understand why the 10-13% number is not higher if we want to have any confidence that the other numbers mean anything.
- Another difference is that the canvas telemetry probe is collected per tab, not per page load.
Per tab meaning... if you use a tab to visit 10 pages on 3 origins and then close the tab - that's only one entry? That seems crazy....
This could still be it, unsure.
Comment 8•1 year ago
•
|
||
So I think we might be getting this issue wrong.
First, WindowGlobalParent::ActorDestroy seem to be called after a page is navigated away and not a tab is closed. I say this because 1) my local testing shows that after a while a page is navigated away, or on exiting firefox, WindowGlobalParent::ActorDestroy is called, and 2) mPageUseCountersWindow->FinishAccumulatingPageUseCounters(); is called in this function too. It is used to collect per page load (or per destroyed really) data.
So we need to understand why the 10-13% number is not higher if we want to have any confidence that the other numbers mean anything.
Second, I think this isn't 10-13% (which is now around 13-15%), but 1000-1300%. Looking at the query, I don't see any any multiplication by 100 to convert numbers from 0.xxxx to xx.xx. Similarly, if we ignore unknown_none entries, the remaining metrics are all less than 1, suggesting we have a over-reporting issue for unknown_none instead of a under-reporting issue for all the metrics.
When it comes to why we would have more than 100% for unknown_none and if we are over-reporting all the metrics, I'm still trying to figure it out.
Comment 9•1 year ago
|
||
(for easier reading I say page loads instead of top_level_content_documents_destroyed and canvas telemetry instead of canvas_fingerprinting_per_tab)
The more I look into it the less it makes sense. I added MOZ_LOGs to here and here. Then I counted the number of page loads and canvas telemetry. I visited ~50 pages on different websites 3 times, and in all 3 instances page loads were > canvas telemetry. Inspecting the code, I don't see anything weird either.
When I query the tables though, there's something strange. I copied and pasted parts of :tjr's query to create my own query here. What I noticed is, there are client ids that report 0 pages visited while reporting multiple unknown_<some-variant> occurrences.
I also wrote this query to check what percentage of users report more canvas telemetry than page loads, and it shows 30% of users report more canvas_unknown than pages loaded.
notes: I used 1pct table, and thought maybe it is because I'm not using the full table. I then asked on #telemetry on matrix, if it makes sense to join on two different 1pct tables, and was told yes, because "Both are using WHERE sample_id = 0 and sample_id is a stable hash of the client_id". On top of this, if you go over the 20 client ids in my query, you'll see that for any given day, there's always main_use_counter_pings rows of data.
Comment 10•1 year ago
|
||
I wonder if we would benefit from using Document::ShouldIncludeInTelemetry(). Our filters aren't 100% the same, we only accept http(s) and don't collect pbm data. We do our checks based on only principal though, while Document::ShouldIncludeInTelemetry() incorporates much more stuff.
Comment 11•1 year ago
|
||
I'm just placing this patch to see if it is a good idea. Maybe there's something that doesn't return true from Document::ShouldIncludeInTelemetry() but we include it. I personally couldn't find any such document type but maybe I'm missing something.
Updated•1 year ago
|
Updated•1 year ago
|
Updated•1 year ago
|
Comment 12•1 year ago
|
||
Comment 13•1 year ago
|
||
Comment 14•1 year ago
•
|
||
I updated https://sql.telemetry.mozilla.org/queries/106649/source with the beta data and we have much more (about 300K) un-matched canvas_unknowns than matched canvas_unknowns.
Even the matched canvas_unknown isn't always smaller than pages visited. I think this is because we just check if we called FinishAccumulatingPageUseCounters, and not check if the we actually incremented top_level_content_documents_destroyed.
According to beta data, for unknown canvas increments,
- ~6% of calls to
FinishAccumulatingPageUseCountersdon't incrementtop_level_content_documents_destroyed. - ~14% (20%-6%) of calls to
WindowGlobalParent::ActorDestroydon't callFinishAccumulatingPageUseCounters.
Despite us incrementing canvas counters only when BrowsingContext()->IsTopContent() there seems to be a huge difference between top_level_content_documents_destroyed and BrowsingContext()->IsTopContent().
Since we already filter for BrowsingContext()->IsTopContent(), we might pass a bool reference to FinishAccumulatingPageUseCounters and set it to true in this branch so that we have a 100% match with top_level_content_documents_destroyed, or we can keep it as is.
IMO the only reason for keeping it in the current state would be if we are seeing more plain-known values than matched-known values. i.e. we see fingerprinting in BrowsingContext()->IsTopContent() contexts where we don't increment top_level_content_documents_destroyed. I'm not sure what that would be.
The data suggests that we do see more fingerprinting in BrowsingContext()->IsTopContent(). 300K canvas_known instances, and 95K instances of canvas_known_matched instances.
So, matching with the top_level_content_documents_destroyed would mean we would miss out on some data.
Given that, we could also add out page counter similar to mixed content page loaded metric.
| Assignee | ||
Comment 15•1 year ago
|
||
there seems to be a huge difference between top_level_content_documents_destroyed and BrowsingContext()->IsTopContent().
I think this is what we should focus on; figure out the reason for this, and it will probably tell us what we should do...
Or, alternatively, with the data we have saying that fingerprinting is happening in situations we're not accumulating top_level_content_documents_destroyed - maybe we help other people find a bug in the logic of top_level_content_documents_destroyed.
Comment 16•1 year ago
|
||
Just so that it isn't lost, I asked what might cause the difference on Matrix DOM channel and Emilio suggested checking the following:
top_level_content_documents_destroyed is only for top level pages, WindowGlobalParent is also used for iframes
So it seems the metric you should be comparing to is content_documents_destroyed
Oh sorry, you are checking IsTopContent()...
Are you sure you're counting iframes right / that the BrowsingContext is still properly attached by the time you get there?
Also the use counters ping is different from your ping, so that could also be a difference, not sure how often it's reported vs yours
I'd try to reproduce that locally loading some frame-heavy page (e.g. cnn or something)
About "Are you sure...that the BrowsingContext is still properly attached by the time you get there?", we do call WindowContext::Discard() and then later call BrowsingContext() which calls WindowContext::GetBrowsingContext(). I don't think BrowsingContext is destroyed at that point, but it would be nice if we would move WindowContext::Discard() after our calls.
I'd try to reproduce that locally loading some frame-heavy page (e.g. cnn or something)
I tried comparing top_level_content_documents_destroyed and our metric recording, and we increment both at the same time always, i couldn't find a weird case.
Also the use counters ping is different from your ping, so that could also be a difference, not sure how often it's reported vs yours
I think this could be a huge part of the difference. One of them is stored in main_use_counters and the other one is in main.
| Assignee | ||
Comment 17•1 year ago
|
||
I think this could be a huge part of the difference. One of them is stored in main_use_counters and the other one is in main.
Can we move our metrics to the use counter ping? You'll have to flag a dom peer for this I believe.
Comment 18•1 year ago
•
|
||
I asked :chutten why some clients would have more pages visited than canvas fingerprinting and basically why our metrics are weird and he had these suggestions
- For the future, add some data collection for counts of the specific conditions that exclude and include a given top-level document dtor from being in one or the other of the two counts (
top_level_content_documents_destroyedand our branch) - For the future, consider reporting
top_level_content_documents_destroyedalso in the "metrics" ping (he said we can do it if we "simply add- metricssomewhere around here") - For now, it might be "good enough" for our purposes to join on client_info.client_id and DATE(submission_timestamp), but be aware that the schedule differences between these two pings will cause noise
Can we move our metrics to the use counter ping? You'll have to flag a dom peer for this I believe.
So for this point, i think we can just do to what :chutten suggested
Comment 19•1 year ago
|
||
Comment 20•1 year ago
|
||
Comment 21•1 year ago
|
||
Comment 22•1 year ago
|
||
Comment 23•1 year ago
|
||
Comment 24•1 year ago
|
||
https://hg.mozilla.org/mozilla-central/rev/24eddc0150f7
https://hg.mozilla.org/mozilla-central/rev/f00bbe8e6a34
Updated•1 year ago
|
Comment 25•1 year ago
|
||
i think we missed the leave-open keyword. reopening.
Updated•1 year ago
|
Updated•1 year ago
|
Comment 26•1 year ago
|
||
| Assignee | ||
Comment 27•9 months ago
|
||
So in my latest attempt to investigate this, I examined the font fingerprinting metric, because that is simpler.
At a high level, we want to measure the frequency of font fingerprinting on the web: {pages with font fingerprinting} / {all pages}
The numerator is font_fingerprinting_per_tab and as a labeled counter we record the count of all loads: with a label of false or true as to whether they had font fingerprinting.
The denominator should be top_level_content_documents_destroyed. Note that we added it to the metrics ping so we could compare the metrics apples to apples - comparing them across pings was believed to be inaccurate.
(We could just use font_fingerprinting_per_tab: {true} / {true + false} - that should work just fine. BUT it may not work for other fingerprinting metrics (e.g. canvas), and we want to understand this metric in relation to top_level_content_documents_destroyed which is an important metric, so we shouldn't just sweep inconsistencies under the rug!)
Okay, so coming back to these two metrics, we should have several comparisons that hold:
- top_level_content_documents_destroyed = font_fingerprinting_per_tab.false + font_fingerprinting_per_tab.true
- font_fingerprinting_per_tab >= font_fingerprinting_per_tab.false
- font_fingerprinting_per_tab >= font_fingerprinting_per_tab.true
And of course, in our data, none of these hold all the time. Specifically, for the past couple days of submissions on release on sample_id=0:
- 60.5% of submissions have more top_level_content_documents_destroyed than font_fingerprinting_per_tab.false + font_fingerprinting_per_tab.true
- 34.3% match
- 1.9% have more font_fingerprinting_per_tab records than top_level_content_documents_destroyed
- 3.2% don't meet any of those criteria (I believe these are NULL submissions)
So we want to dive in and try to understand the inconsistencies of course.
Both metrics get recorded in WindowGlobalParent::ActorDestroy. Let's look at the stuff in between them:
- top_level_content_documents_destroyed will record Parent Process page loads; while the font metric will not
- top_level_content_documents_destroyed will record PBM loads while the font metric will not
- If there is not a browserParent->GetLoadContext(), then the font metric will not be recorded
- If there is no documentURI, or the scheme of the URI is not HTTP or HTTPS, then the font metric will not be recorded
- If
mPageUseCountersWindowis null, then top_level_content_documents_destroyed will not be recorded, but this restriction is not the case for the font metric
This seems to be sane. The mPageUseCountersWindow is null scenario seems really odd, so it being 2% of submissions seems okay... Excluding parent process loads, PBM loads, and non-HTTP loads means the 60% majority seems reasonable.
I looked specifically at that 60% majority, and the difference between pages loaded and font metric count, and the breakdown on the percent difference is:
0.0 - .1 | ████████████████ 19.4
0.1 - .2 | ██████████████████████ 24.0
0.2 - .3 | ████████████████ 19.4
0.3 - .4 | ████████████ 15.1
0.4 - .5 | ████████ 10.9
0.5 - .6 | ████ 5.5
0.6 - .7 | ██ 2.7
0.7 - .8 | █ 1.3
0.8 - .9 | 0.8
0.9 - 1 | 0.9
I would have expected it to be smaller, but it's reassuring to see that large differences are rare.
In conclusion, I think the font fingerprinting metric is sane, and we can use it - and just it, ignoring top_level_content_documents_destroyed - to track font fingerprinting on the web.
Next I will look at the canvas metrics and see if we can migrate them to a similar design.
| Assignee | ||
Comment 28•9 months ago
|
||
Okay, so knowing the above, the desired course of action as I see it is to refactor the canvas fingerprinting metric in a similar fashion. However, unlike the font fingerprinting metric, the canvas fingerprinting metric is not a simple 'Yes' or 'No'.
Additionally, in June, we added another dimension to the metric: if the result of FinishAccumulatingPageUseCounters contains geUseCounterResultBits::DATA_RECEIVED - if that is true, we increment the 'Matched' metric, but whether or not it is true, we increment the 'Normal' metric. We added this to debug the metric and try to understand why the top level content document destroyed metric was not helping us. This dimensionality is probably not necessary, but it makes me wonder which of the two scenarios we should use exclusively. The font fingerprinting metric does not care about this; so that is good to keep in mind.
So what do we actually record in this metric? We loop through all the content blocking log entries. We pick the entry with the highest fingerprinting likelihood. Specifically, we prefer the first entry that has known fingerprinting text, and if no entry has known fingerprinting text, we prefer the first entry with an identified fingerprinter.
The dimensionality of the metric is:
unknown - 0 - no identified fingerprinting
1 - eFingerprintJS
2 - eAkamai
3 - eVariant1
4 - eVariant2
5 - eVariant3
6 - eVariant4
7 - eMaybe - all from https://searchfox.org/firefox-main/rev/7f33a0cc184f6372c66281499063bda0f42f9c83/toolkit/components/antitracking/ContentBlockingNotifier.h#43
knownText - 0 - no identified fingerprinting
1 - eFingerprintJS
2 - eAkamai
3 - eVariant1
4 - eVariant2
5 - eVariant3
6 - eVariant4
7 - eMaybe
For each page load, we will increment one of those 16 buckets. (With the following restrictions: it has to be a content principal, it can't be a parent process load, there has to be a load context, and a document URI, it has to be HTTP/HTTPS, and it can't be a PBM load.)
Additionally, if the result of FinishAccumulatingPageUseCounters contains geUseCounterResultBits::DATA_RECEIVED we will also increment one of those 16 buckets for unknownMatched and knownTextMatched
So the main driver of the refactoring is not actually needed, we should be recording something in this metric on every load. (But there are other reasons to refactor.) Let's dig into the dimentionality.
knownText is used when we detected a known fingerprinting text. So everything in that bucket is already extremely likely to be canvas fingerprinting. The individual numbers inside that bucket will be interesting: if 0 is very high it means there's a fingerprinter using known text but techniques we otherwise haven't attributed to them.
unknown is used when we did not detect known fingerprinting text. unknown 0 is used when we did not detect any fingerprinting at all. So that should be our large metric.
All the other unknown values are situations where we detected no known text, but some other behavior we've attributed to fingerprinters.
The fact that we give precedence to certain fingerprinters over others and that we take the first important one we saw mean that there is no single fingerprinter we can say is a 'maximum', it's possible we are undercounting any fingerprinter because we defer to the loading/occurrence order on the page. The fact that Maybe is pretty broad means there is a degree of uncertainty even when we sum all the fingerprinters. What we can say for certain is that the sum of all the knownText fingerprinters as compared to unknown 0 gives a good indicator of frequency. Looking at this now I can imagine the following changes:
- Pick either 'matched' (
if the result of FinishAccumulatingPageUseCounters contains geUseCounterResultBits::DATA_RECEIVED) or not and align it with font fingerprinting - Probably separate
unknown 0fromunknown != 0because it's weird thatunknown 0means no fingerprinting butunknown 1means fingerprinting - Design the metric so we can track the ratio of every type reliably
- Add in a dimentionality that tracks the canvas extraction method, because we are interested in that
The next step is to analyze the data we have and confirm our beliefs about what we have today is correct.
| Assignee | ||
Comment 29•9 months ago
|
||
| Assignee | ||
Comment 30•8 months ago
|
||
Presently we use CheckSuspiciousFingerprintingActivity to see
if the number of fingerprinting activities exceed a theshold
and if so, we indicate that an entry has suspicious fingerprinting.
That threshold is 1. And we don't have any plans to adjust it.
So we can just remove the check altogether: if that function is
called, it is called immediately after creating or finding a log
of the two relevant types: Canvas or Font fingerprinting. So it
will always return true.
| Assignee | ||
Comment 31•8 months ago
|
||
| Assignee | ||
Comment 32•8 months ago
|
||
| Assignee | ||
Comment 33•8 months ago
|
||
| Assignee | ||
Comment 34•8 months ago
|
||
| Assignee | ||
Comment 35•8 months ago
|
||
| Assignee | ||
Comment 36•8 months ago
|
||
| Assignee | ||
Comment 37•8 months ago
|
||
| Assignee | ||
Comment 38•8 months ago
|
||
| Assignee | ||
Comment 39•8 months ago
|
||
We encountered a fingerprinter that created an about:blank iframe
and did fingerprinting there. This iframe had no channel, so it
did not get a ContentBlockingEvent on it. Traverse the document
tree upwards to find a document we can log it on.
| Assignee | ||
Comment 40•8 months ago
|
||
This updates one test to use the new metrics.
It updates the other test to not require both font and
canvas fingerprinting to be present for 'suspicious fingerprinting'
to be triggered.
| Assignee | ||
Comment 41•8 months ago
|
||
I tried to figure out if we should use if the result of FinishAccumulatingPageUseCounters contains geUseCounterResultBits::DATA_RECEIVED as a limiter, or ignore it.
Firstly, no one else does this. So that's a good sign we shouldn't.
Numbers-wise, the values reported under the matched label (the label that is limited) are about 25%-30% of the values reported without the labels. That's not 'percentage of loads' that's raw 'number of times we saw a thing'. So that limiter is eliminating a lot of reporting.
Taken together, and without an actual deeper understanding of what we're actually testing, my conclusion is that we should not use this limiter.
| Assignee | ||
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
Updated•8 months ago
|
| Assignee | ||
Comment 42•8 months ago
|
||
Based on the fact that no other reporting mechanism cares about
the DATA_RECEIVED flag, and that it eliminates about 75% of the
reports we would have received, remove this argument and do not
change any behavior based on it
Updated•8 months ago
|
Updated•8 months ago
|
| Assignee | ||
Comment 43•8 months ago
|
||
Comment 44•8 months ago
|
||
Comment 45•8 months ago
|
||
Comment 46•8 months ago
|
||
Backed out for causing ThreadSanitizer failures.
Backout link: https://hg-edge.mozilla.org/integration/autoland/rev/29c21d76fcf24b26cd8b8383ce810412e2513d29
Failure log: https://treeherder.mozilla.org/logviewer?job_id=539070141&repo=autoland&task=DYut5tmMQ_-MVvT-6ie6xg.0&lineNumber=4786
Comment 47•8 months ago
|
||
Comment 48•8 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/b80938513b46
https://hg.mozilla.org/mozilla-central/rev/edfc7e7f214e
https://hg.mozilla.org/mozilla-central/rev/ef47c69b9a44
https://hg.mozilla.org/mozilla-central/rev/556d736dcaaf
https://hg.mozilla.org/mozilla-central/rev/8453eab54f4f
https://hg.mozilla.org/mozilla-central/rev/92096cc6e298
https://hg.mozilla.org/mozilla-central/rev/01f80c54bd68
https://hg.mozilla.org/mozilla-central/rev/c06b62db4b23
https://hg.mozilla.org/mozilla-central/rev/9a057151d9cb
https://hg.mozilla.org/mozilla-central/rev/5e55a1f34d89
https://hg.mozilla.org/mozilla-central/rev/2e9b483cf9d2
https://hg.mozilla.org/mozilla-central/rev/d295ca8f25a5
https://hg.mozilla.org/mozilla-central/rev/fa041ec0ef75
https://hg.mozilla.org/mozilla-central/rev/482c92d5640f
| Assignee | ||
Updated•8 months ago
|
Updated•8 months ago
|
| Assignee | ||
Updated•8 months ago
|
Updated•6 months ago
|
Description
•