1413258 - Make sure the LetsEncrypt data use-case is covered

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Georg Fritzsche [:gfritzsche]]

Due to bug 1406390 et al release data will no longer be aggregated from Firefox 58+ and will no longer be available for view on TMO.
Lets Encrypt seems to be using some of the release data published through aggregates.tmo, so we need to figure out a transition plan.

Comment 1

[Georg Fritzsche [:gfritzsche]]

JC, can you summarize what data LetsEncrypt needs?

Comment 2

[J.C. Jones [:jcj] (he/him)]

Georg,

Right now Let's Encrypt's https-over-time graph [1] is a Golang-clone [2][3] of rbarnes' time-series aggregator [4]. The intent is just to track adoption of HTTPS over time in some fashion that indicates how often users are negotiating TLS-protected sessions, as driving HTTPS-everywhere is the goal of Let's Encrypt.

Currently it's just pulling release data [3]. 

If I'm understanding correctly, the telemetry is not going away, it's just not going to be accessible on TMO? Perhaps this would be an opportunity to do the number-crunching on Mozilla's side and publish the summarized data directly?

[1] https://letsencrypt.org/stats/#percent-pageloads
[2] https://github.com/jcjones/ct-sql/tree/master/firefox-telemetry
[3] https://github.com/jcjones/ct-sql/blob/master/cmd/telemetry-update/main.go#L95
[4] https://ipv.sx/telemetry/general-v2.html?channels=release&measure=HTTP_PAGELOAD_IS_SSL&target=1&absolute=0&relative=1

Comment 3

[Chris H-C :chutten]

Uh oh.

Network telemetry is predicated on the value of toolkit.telemetry.enabled which limits its collection to pre-release, even though it has releaseChannelCollection: opt-out. This means that we have no representative SSL data for Firefox users beyond Beta.

This is due to nsHttpChannel checking if the nsHttpHandler has telemetry enabled[1] which checks toolkit.telemetry.enabled[2].

This needs to be changed if we want SSL data from the release population. Simply popping the HTTP_{PAGELOAD|TRANSACTION}_IS_SSL accumulations out of the if(gHttpHandler->IsTelemetryEnabled()) block ought to do.

Better yet, we could remove the whole IsTelemetryEnabled part from nsHttpHandler and have the net telemetry be unconditionally accumulated.(presupposing that it isn't expensive to generate the data for accumulation.) Telemetry takes care of deciding what to store and send so that callers shouldn't have to.

(At the very least we ought to convert it to use Telemetry::CanRecordPrerelease())

[1]: http://searchfox.org/mozilla-central/rev/af86a58b157fbed26b0e86fcd81f1b421e80e60a/netwerk/protocol/http/nsHttpChannel.cpp#2268
[2]: http://searchfox.org/mozilla-central/rev/af86a58b157fbed26b0e86fcd81f1b421e80e60a/netwerk/protocol/http/nsHttpHandler.cpp#1753

Comment 4

[Chris H-C :chutten]

Attachment: bug 1413258 - Collect SSL Telemetry on all channels

Ever since bug 1340021 we were supposed to be collecting *_IS_SSL probes on
release, but the code in nsHttpChannel prevented it from happening.

Review commit: https://reviewboard.mozilla.org/r/196374/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/196374/

Comment 5

[Patrick McManus [:mcmanus]]

thanks. I'll look at some of the other network data too. Some of it might be fairly expensive to create - but this is not.

Comment 6

[Patrick McManus [:mcmanus]]

Comment on attachment 8925126 [details]
bug 1413258 - Collect SSL Telemetry on all channels

https://reviewboard.mozilla.org/r/196374/#review201634

Comment 7

[Georg Fritzsche [:gfritzsche]]

(In reply to Chris H-C :chutten from comment #3)
> (At the very least we ought to convert it to use
> Telemetry::CanRecordPrerelease())

Should we have a follow-up bug on that?
That should avoid some misunderstandings.

Comment 8

[Georg Fritzsche [:gfritzsche]]

(In reply to J.C. Jones [:jcj] from comment #2)
> If I'm understanding correctly, the telemetry is not going away, it's just
> not going to be accessible on TMO? Perhaps this would be an opportunity to
> do the number-crunching on Mozilla's side and publish the summarized data
> directly?

Yes, with the fix in comment 3 the data will be available, just not publicly right now.
Lets use bug 1414839 to scope out what we need to do here.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0d15cd7ec41f
Collect SSL Telemetry on all channels r=mcmanus

Comment 10

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/0d15cd7ec41f