Audit Release Internet Health Telemetry to be sure it's being collected properly

RESOLVED WORKSFORME

Status

()

enhancement
RESOLVED WORKSFORME
2 years ago
a year ago

People

(Reporter: chutten, Unassigned)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(firefox58 affected)

Details

(Reporter)

Description

2 years ago
Ever since bug 1340021 we've meant to be collecting many network probes in opt-out fashion. In bug 1413258 we found out that it wasn't true for some of them due to extra checks.

Ensure the full list of probes opted-out in bug 1340021 are actually being collected in release.

Here's the list:

HTTP_{TRANSACTION|PAGELOAD}_IS_SSL (being handled in bug 1413258)
HTTP_SCHEME_UPGRADE
SSL_HANDSHAKE_VERSION
SSL_HANDSHAKE_RESULT
SSL_TIME_UNTIL_READY
SSL_TIME_UNTIL_HANDSHAKE_FINISHED
SSL_RESUMED_SESSION,
PWMGR_FORM_AUTOFILL_RESULT,
PWMGR_LOGIN_PAGE_SAFETY
Priority: -- → P2
We are not the owners of these probes.
Moving to more suitable component for verification.
Component: Telemetry → DOM: Security
Priority: P2 → --
Product: Toolkit → Core
David, is this something that you could take on? Or Jonathan? Or Tanvi?
Flags: needinfo?(tanvi)
Flags: needinfo?(jkt)
Flags: needinfo?(dkeeler)
> SSL_HANDSHAKE_VERSION

I think this one is fine - it's marked opt-out and there's no extra checks I can find.

> SSL_HANDSHAKE_RESULT

Same here.

> SSL_TIME_UNTIL_READY

Same.

> SSL_TIME_UNTIL_HANDSHAKE_FINISHED

Same.

> SSL_RESUMED_SESSION

Same.

I'm not familiar with the other histograms.
Flags: needinfo?(dkeeler)
(In reply to Chris H-C :chutten from comment #0)
> PWMGR_FORM_AUTOFILL_RESULT,
MattN may no more about this one.

> PWMGR_LOGIN_PAGE_SAFETY
We should continue collecting this probe in release as an opt-out probe.

What is the ask in this bug exactly?
(In reply to Tanvi Vyas[:tanvi] from comment #4)
> (In reply to Chris H-C :chutten from comment #0)
> > PWMGR_FORM_AUTOFILL_RESULT,
> MattN may no more about this one.
>
Flags: needinfo?(MattN+bmo)
(Reporter)

Comment 6

a year ago
(In reply to Tanvi Vyas[:tanvi] from comment #4)
> (In reply to Chris H-C :chutten from comment #0)
> > PWMGR_FORM_AUTOFILL_RESULT,
> MattN may no more about this one.
> 
> > PWMGR_LOGIN_PAGE_SAFETY
> We should continue collecting this probe in release as an opt-out probe.
> 
> What is the ask in this bug exactly?

Ensure that the data that was asked to be collected in bug 1340021 is actually being received and analysed. We found two that weren't actually being sent due to errant code (bug 1413258), so this is to check that the others are actually doing what they're supposed to be doing, and providing the value they're supposed to be providing.

A quick check at sql.tmo for release main pings containing the probe should suffice. Or pointing to a dashboard that collates them.
David, any chance you could make sure we are collecting that data? I guess this should be a P1 because otherwise we are facing a 6 or 12 weeks lag of that data.
Component: DOM: Security → Security: PSM
Flags: needinfo?(dkeeler)
The probes from comment 0 are all showing up in the tmo measurement dashboard, so I think we're good here.
Status: NEW → RESOLVED
Last Resolved: a year ago
Flags: needinfo?(dkeeler)
Resolution: --- → WORKSFORME
(Reporter)

Comment 9

a year ago
(In reply to David Keeler [:keeler] (use needinfo) from comment #8)
> The probes from comment 0 are all showing up in the tmo measurement
> dashboard, so I think we're good here.

That's not necessarily sufficient. TMO only reflects users that have opt-in Telemetry (so, all pre-release users and few (soon (Firefox 58) to be none) release users). It doesn't say anything about whether or not the data is being reported by all release users as expected.

So I went to the longitudinal dataset and asked it how many clients from its sampled dataset it has seen on release in the past six months for all of comment#0's probes:

HTTP_{TRANSACTION|PAGELOAD}_IS_SSL (being handled in bug 1413258) showed up in about 1% of the clients. As expected for something that's still opt-in-only (for about another week).

HTTP_SCHEME_UPGRADE
SSL_HANDSHAKE_VERSION
SSL_HANDSHAKE_RESULT
SSL_TIME_UNTIL_READY
SSL_TIME_UNTIL_HANDSHAKE_FINISHED
SSL_RESUMED_SESSION, <-- these all showed up in about 80% of the clients. No problems here.

PWMGR_FORM_AUTOFILL_RESULT,
PWMGR_LOGIN_PAGE_SAFETY <-- these two showed up in more than 25% of the clients. That's definitely more than opt-in users.

So the conclusion, that everything's fine here, seems to hold (barring bug 1413258).
Flags: needinfo?(tanvi)
Flags: needinfo?(jkt)
Flags: needinfo?(MattN+bmo)
You need to log in before you can comment on or make changes to this bug.