1533363 - Collect Telemetry on tracking resources traffic

Status: RESOLVED FIXED

(Core :: Networking, enhancement, P1)

Description

[Gary Chen [:xeonchen]]

We are going to analysis the amount of tracking resources, the interest values are daily statistics of

1. total number of HTTP transactions
2. total number of connections
3. total number of bytes

We're going to collect above values based on their Class of Service flags and the category on the tracking list.

Data Types

Class of Service

Flags	Load Type
nsIClassOfService::Leader	A
w/o nsIRequest::LOAD_BACKGROUND	B
w/ nsIRequest::LOAD_BACKGROUND	C

Tracking Category

Category	List Category
Content	I
Analytics/Advertising/Social	II
Fingerprinting	III

Telemetry Targets

There are 22 values, denoted by Y for both normal/private browsing mode.

Normal Mode

Y = 0 for first party
Y = 1 for non-listed third party type

Y	Type A	Type B	Type C
Category I	2	3	4
Category II	5	6	7
Category III	8	9	10

Private Mode

Y = 11 for first party
Y = 12 for non-listed third party type

Y	Type A	Type B	Type C
Category I	13	14	15
Category II	16	17	18
Category III	19	20	21

Comment 1

[Ethan Tseng [:ethan]]

Gary, thank you for filing this bug. Nice tables! :)

Comment 2

[Valentin Gosu [:valentin] (he/him)]

Gary, will you be submitting a patch? Or should we find someone on the necko team?

Comment 3

[Ethan Tseng [:ethan]]

(In reply to Valentin Gosu [:valentin] from comment #2)

Gary, will you be submitting a patch? Or should we find someone on the necko team?

I talked with Gary offline. He doesn't have cycles to implement this patch in next 2~3 weeks.
We would appreciate if someone on the Necko team can take it (It not, Gary will come back to work on this later).
Thank you, Valentin!

Comment 4

[Gary Chen [:xeonchen]]

Attachment: Bug 1533363 - Part 1: Add HttpTrafficAnalyzer service;

Comment 5

[Gary Chen [:xeonchen]]

This bug isn't related to prio.

Comment 6

[Gary Chen [:xeonchen]]

Attachment: Bug 1533363 - Part 2: Add HTTP Traffic category Telemetry;

Comment 7

[Gary Chen [:xeonchen]]

Attachment: request.md

Comment 8

[Chris H-C :chutten]

Comment on attachment 9054192 [details]
request.md

Preliminary note:

Please expand a touch on the data being collected. It is important to note that the transactions, connections, and bytes are being recorded per-"category" and that the number and disposition of categories is known ahead of time. Please list the categories that these will be divided into.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions files [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json) and [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml), and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 73.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :xeonchen is responsible for renewing or removing the collection before it expires in Firefox 73.

---
Result: datareview+, pending additional information about the data being collected (per the preliminary note)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c3488f465c29
Part 1: Add HttpTrafficAnalyzer service; r=dragana
https://hg.mozilla.org/integration/autoland/rev/99636457aa95
Part 2: Add HTTP Traffic category Telemetry; r=chutten

Comment 10

[Cristian Brindusan [:cbrindusan]]

Backed out 2 changesets (Bug 1533363) for build bustages at HttpTrafficAnalyzer.cpp.

Backout: https://hg.mozilla.org/integration/autoland/rev/dfe534711b4ad3ab68029d06b0efedec70732952

Push that started the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=99636457aa9599efea9fd166328c73d7c683026b

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=238214531&repo=autoland&lineNumber=13421

Comment 11

[Gary Chen [:xeonchen]]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3cf47ad8fe4c128f44a620a7c77f750e8864f94f

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/13bff8b50313
Part 1: Add HttpTrafficAnalyzer service; r=dragana
https://hg.mozilla.org/integration/autoland/rev/7c3de15af00b
Part 2: Add HTTP Traffic category Telemetry; r=chutten

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/13bff8b50313
https://hg.mozilla.org/mozilla-central/rev/7c3de15af00b

Comment 14

[Ethan Tseng [:ethan]]

Gary, please request uplift to Beta 67 for this bug and any required ones, such as:
Bug 1542069 - Enable Telemetry on tracking resources traffic
Bug 1541238 - Delays Tracking Loads

Comment 15

[Gary Chen [:xeonchen]]

Comment on attachment 9052886 [details]
Bug 1533363 - Part 1: Add HttpTrafficAnalyzer service;

Beta/Release Uplift Approval Request

• Feature/Bug causing the regression: N/A
• User impact if declined:
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1542069, Bug 1541238
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This bug only adds some telemetry probes.
  Bug 1542069 enables telmetry probing introduced in this bug.
  Bug 1541238 is default disabled.
• String changes made/needed: N/A

Comment 16

[Pascal Chevrel:pascalc (PTO until April 26)]

Please request uplift in Bug 1542069 and Bug 1541238 as well.

Comment 17

[Gary Chen [:xeonchen]]

(In reply to Pascal Chevrel:pascalc from comment #16)

Please request uplift in Bug 1542069 and Bug 1541238 as well.

Thanks for reminding :)

Comment 18

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9052886 [details]
Bug 1533363 - Part 1: Add HttpTrafficAnalyzer service;

Low risk, uplift approved for 67 beta 11, thanks.

Comment 19

[Oana Pop-Rus]

https://hg.mozilla.org/releases/mozilla-beta/rev/a258da1f2a67
https://hg.mozilla.org/releases/mozilla-beta/rev/e27ad01b9daa

Comment 20

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 4 changesets (Bug 1541238, Bug 1542069, Bug 1533363) for build bustages - missing Bug 1536488
https://hg.mozilla.org/releases/mozilla-beta/rev/ca196717df07

Comment 21

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 4 changesets (Bug 1541238, Bug 1542069, Bug 1533363) for build bustages - missing Bug 1536488

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=74c650902989c6b30b913c4b20639bf71ba11aef&selectedJob=240336243

Backout link: https://hg.mozilla.org/releases/mozilla-beta/rev/ca196717df0793c823865553b8907cf28573597f

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=240336243&repo=mozilla-beta&lineNumber=13796

[task 2019-04-15T12:55:45.628Z] 12:55:45 INFO - make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/xpconnect/tests/components/native'
[task 2019-04-15T12:55:46.569Z] 12:55:46 INFO - make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http'
[task 2019-04-15T12:55:46.577Z] 12:55:46 INFO - /builds/worker/workspace/build/src/clang/bin/clang++ -o Unified_cpp_protocol_http2.i_o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/netwerk/protocol/http -I/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/extensions/auth -I/builds/worker/workspace/build/src/netwerk/base -I/builds/worker/workspace/build/src/netwerk/cookie -I/builds/worker/workspace/build/src/netwerk/url-classifier -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fcrash-diagnostics-dir=/builds/worker/artifacts -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fomit-frame-pointer -funwind-tables -Werror -fprofile-instr-generate -MD -MP -MF .deps/Unified_cpp_protocol_http2.i_o.pp /builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http/Unified_cpp_protocol_http2.cpp
[task 2019-04-15T12:55:46.578Z] 12:55:46 INFO - In file included from /builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http/Unified_cpp_protocol_http2.cpp:29:
[task 2019-04-15T12:55:46.580Z] 12:55:46 ERROR - /builds/worker/workspace/build/src/netwerk/protocol/http/nsHttpChannel.cpp:1303:28: error: no member named 'CLASSIFIED_ANY_BASIC_TRACKING' in 'nsIHttpChannel::ClassificationFlags'; did you mean 'CLASSIFIED_ANY_TRACKING'?
[task 2019-04-15T12:55:46.581Z] 12:55:46 INFO - } else if (flags & CF::CLASSIFIED_ANY_BASIC_TRACKING) {
[task 2019-04-15T12:55:46.582Z] 12:55:46 INFO - ^
[task 2019-04-15T12:55:46.582Z] 12:55:46 INFO - CLASSIFIED_ANY_TRACKING
[task 2019-04-15T12:55:46.583Z] 12:55:46 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/include/nsIHttpChannel.h:200:5: note: 'CLASSIFIED_ANY_TRACKING' declared here
[task 2019-04-15T12:55:46.584Z] 12:55:46 INFO - CLASSIFIED_ANY_TRACKING = 125,
[task 2019-04-15T12:55:46.585Z] 12:55:46 INFO - ^
[task 2019-04-15T12:55:46.586Z] 12:55:46 INFO - 1 error generated.
[task 2019-04-15T12:55:46.587Z] 12:55:46 INFO - /builds/worker/workspace/build/src/config/rules.mk:805: recipe for target 'Unified_cpp_protocol_http2.i_o' failed
[task 2019-04-15T12:55:46.588Z] 12:55:46 ERROR - make[5]: *** [Unified_cpp_protocol_http2.i_o] Error 1
[task 2019-04-15T12:55:46.589Z] 12:55:46 INFO - make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http'
[task 2019-04-15T12:55:46.591Z] 12:55:46 INFO - make[5]: *** Waiting for unfinished jobs....
[task 2019-04-15T12:55:46.592Z] 12:55:46 INFO - make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/xpconnect/src'

Comment 22

[Bogdan Tara[:bogdan_tara | bogdant]]

Not sure that all the dependencies are met.

Comment 23

[(no longer active)]

We decided to hold off on backporting this to beta.

Comment 24

[Gary Chen [:xeonchen]]

(In reply to :Ehsan Akhgari from comment #23)

We decided to hold off on backporting this to beta.

↑