1414388 - Audit Release Internet Health Telemetry to be sure it's being collected properly

Status: RESOLVED WORKSFORME

(Core :: Security: PSM, enhancement)

Description

[Chris H-C :chutten]

Ever since bug 1340021 we've meant to be collecting many network probes in opt-out fashion. In bug 1413258 we found out that it wasn't true for some of them due to extra checks.

Ensure the full list of probes opted-out in bug 1340021 are actually being collected in release.

Here's the list:

HTTP_{TRANSACTION|PAGELOAD}_IS_SSL (being handled in bug 1413258)
HTTP_SCHEME_UPGRADE
SSL_HANDSHAKE_VERSION
SSL_HANDSHAKE_RESULT
SSL_TIME_UNTIL_READY
SSL_TIME_UNTIL_HANDSHAKE_FINISHED
SSL_RESUMED_SESSION,
PWMGR_FORM_AUTOFILL_RESULT,
PWMGR_LOGIN_PAGE_SAFETY

Comment 1

[Georg Fritzsche [:gfritzsche]]

We are not the owners of these probes.
Moving to more suitable component for verification.

Comment 2

[Christoph Kerschbaumer [:ckerschb]]

David, is this something that you could take on? Or Jonathan? Or Tanvi?

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

> SSL_HANDSHAKE_VERSION

I think this one is fine - it's marked opt-out and there's no extra checks I can find.

> SSL_HANDSHAKE_RESULT

Same here.

> SSL_TIME_UNTIL_READY

Same.

> SSL_TIME_UNTIL_HANDSHAKE_FINISHED

Same.

> SSL_RESUMED_SESSION

Same.

I'm not familiar with the other histograms.

Comment 4

[Tanvi Vyas[:tanvi]]

(In reply to Chris H-C :chutten from comment #0)
> PWMGR_FORM_AUTOFILL_RESULT,
MattN may no more about this one.

> PWMGR_LOGIN_PAGE_SAFETY
We should continue collecting this probe in release as an opt-out probe.

What is the ask in this bug exactly?

Comment 5

[Tanvi Vyas[:tanvi]]

(In reply to Tanvi Vyas[:tanvi] from comment #4)
> (In reply to Chris H-C :chutten from comment #0)
> > PWMGR_FORM_AUTOFILL_RESULT,
> MattN may no more about this one.
>

Comment 6

[Chris H-C :chutten]

(In reply to Tanvi Vyas[:tanvi] from comment #4)
> (In reply to Chris H-C :chutten from comment #0)
> > PWMGR_FORM_AUTOFILL_RESULT,
> MattN may no more about this one.
> 
> > PWMGR_LOGIN_PAGE_SAFETY
> We should continue collecting this probe in release as an opt-out probe.
> 
> What is the ask in this bug exactly?

Ensure that the data that was asked to be collected in bug 1340021 is actually being received and analysed. We found two that weren't actually being sent due to errant code (bug 1413258), so this is to check that the others are actually doing what they're supposed to be doing, and providing the value they're supposed to be providing.

A quick check at sql.tmo for release main pings containing the probe should suffice. Or pointing to a dashboard that collates them.

Comment 7

[Christoph Kerschbaumer [:ckerschb]]

David, any chance you could make sure we are collecting that data? I guess this should be a P1 because otherwise we are facing a 6 or 12 weeks lag of that data.

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

The probes from comment 0 are all showing up in the tmo measurement dashboard, so I think we're good here.

Comment 9

[Chris H-C :chutten]

(In reply to David Keeler [:keeler] (use needinfo) from comment #8)
> The probes from comment 0 are all showing up in the tmo measurement
> dashboard, so I think we're good here.

That's not necessarily sufficient. TMO only reflects users that have opt-in Telemetry (so, all pre-release users and few (soon (Firefox 58) to be none) release users). It doesn't say anything about whether or not the data is being reported by all release users as expected.

So I went to the longitudinal dataset and asked it how many clients from its sampled dataset it has seen on release in the past six months for all of comment#0's probes:

HTTP_{TRANSACTION|PAGELOAD}_IS_SSL (being handled in bug 1413258) showed up in about 1% of the clients. As expected for something that's still opt-in-only (for about another week).

HTTP_SCHEME_UPGRADE
SSL_HANDSHAKE_VERSION
SSL_HANDSHAKE_RESULT
SSL_TIME_UNTIL_READY
SSL_TIME_UNTIL_HANDSHAKE_FINISHED
SSL_RESUMED_SESSION, <-- these all showed up in about 80% of the clients. No problems here.

PWMGR_FORM_AUTOFILL_RESULT,
PWMGR_LOGIN_PAGE_SAFETY <-- these two showed up in more than 25% of the clients. That's definitely more than opt-in users.

So the conclusion, that everything's fine here, seems to hold (barring bug 1413258).