1615269 - Perform AS Number lookup before throwing away IP information

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Mark Reid [:mreid]]

This is a follow-up for Bug 1505491.

AS Number could be useful for analysis, but there's a concern that it might be too specific or identifying.

Tim, as the proposed responsible party, could you fill out a data review request with the questions you're hoping to answer that are different from those answerable by ISP name?

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Data collection request

Comment 2

[Chris H-C :chutten]

Comment on attachment 9126506 [details]
Data collection request

Is the proposed collection the AS number itself, or these derived counted measurements based on it?

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

We are not currently mapping IP addresses to ASNs, and we are already sending events to count how many users trip the DoH canary domain, so the AS is the novel aspect.

Capturing the ASN that appears to be announcing an IP address is strictly less informative than the IP address, but can still be sensitive (e.g. my IP address maps to an AS for the Vancouver Mozilla office, so you can make a strong inference about my identity). To mitigate that privacy risk, I want to avoid tagging individual pings with IP addresses or ASNs, and instead want to compute some aggregates for ASNs where we have more than a critical threshold of users.

Comment 4

[Chris H-C :chutten]

Okay, so the listed collections are correct: two derived, counted metrics that use (before discarding) per-ping AS mapping? If so, I'll proceed with the review.

Comment 5

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

That's right; thanks!

Comment 6

[Chris H-C :chutten]

Comment on attachment 9126506 [details]
Data collection request

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

:tdsmith will provide information about where this documentation will live. I'm assuming in docs.telemetry.mozilla.org someplace?

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is performed on Telemetry, which can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :tdsmith is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+, pending :tdsmith providing the location of the documentation for this measure.

Comment 7

[Mark Reid [:mreid]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #3)

Capturing the ASN that appears to be announcing an IP address is strictly less informative than the IP address, but can still be sensitive (e.g. my IP address maps to an AS for the Vancouver Mozilla office, so you can make a strong inference about my identity). To mitigate that privacy risk, I want to avoid tagging individual pings with IP addresses or ASNs, and instead want to compute some aggregates for ASNs where we have more than a critical threshold of users.

Computing aggregates of this data without tagging individual pings is novel on the pipeline side as far as I know. Unless you mean "only tag individual pings with ASNs containing more than X reporting clients over the previous <time period>".

Tim, can you clarify what you have in mind for aggregates here?

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

(In reply to Chris H-C :chutten from comment #6)

Comment on attachment 9126506 [details]
:tdsmith will provide information about where this documentation will live.
I'm assuming in docs.telemetry.mozilla.org someplace?

Clearing ni? and acknowledging an obligation to provide this documentation before we begin performing this aggregation.

(In reply to Mark Reid [:mreid] from comment #7)

Computing aggregates of this data without tagging individual pings is novel on the pipeline side as far as I know.

Yes, I think so.

Schematically, I'd like the resulting table to look something like:

SELECT
  submission_date,
  as_number,
  COUNT(distinct client_id) AS n_clients,
  count of distinct client_ids reporting a DoH heuristics event,
  count of distinct client_ids reporting a DoH heuristics event where the canary was disabled
FROM ...
GROUP BY 1, 2
HAVING n_clients  > @some_value

Comment 9

[Jeff Klukas [:klukas] (UTC-4)]

We have a few ways we could try to produce the requested dataset.

1. Produce AS numbers as part of decoded output, but with a size filter applied

This would require an updated scope for data review, but it would follow an established operational pattern. We use a size filter for geo city information, throwing away city info if the city is not above a threshold based on a public dataset of city sizes. If no such reliable dataset already exists, we could potentially look at producing it ourselves, but that would be a project in itself.

I think this is the best option, but I don't think the necessary dataset exists.

2. Batch processing from payload_bytes_raw

This seems like the cleanest way to produce a dataset in the shape that :tdsmith mentioned; we would define a docker container that loads the MaxMind DB, queries the previous day's output from payload_bytes_raw, extracting AS numbers, and then write the aggregated results into the target table. There's an access question of whether we want to let an Airflow job be able to read payload_bytes_raw, which contains raw IPs and is thus heavily restricted. There's also a performance question since client_id is not available at the top level in payload_bytes_raw; we'd need to reach into the payload to extract it, which incurs query cost and also means we have to use a JS UDF for unzipping the payload, which may not be reliable at scale; we could potentially stream out the entire gzipped payload and process inside the container.

This is clunky, but it's doable and I believe we could make it reliable.

3. Stream processing

We could perform the aggregation in the decoder by performing AS number lookup, but not persisting it as part of the main output messages. We would instead introduce an additional branch of the pipeline to do the aggregation. This would be a pretty new paradigm and it would complicate our thinking around operation of the pipeline; in particular, it might place new restrictions on how and when we deploy the pipeline. We would also be introducing some error to the data because of imperfect deduplication within the streaming pipeline compared to batch processing at the end of the day.

Comment 10

[Anna Scholtz [:ascholtz]]

Attachment: GitHub Pull Request

Comment 11

[Anna Scholtz [:ascholtz]]

moz-fx-data-shared-prod.telemetry.asn_aggregates is now available.