Closed Bug 1533363 Opened 8 months ago Closed 6 months ago

Collect Telemetry on tracking resources traffic

Categories

(Core :: Networking, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: xeonchen, Assigned: xeonchen)

References

Details

(Whiteboard: [necko-triaged])

Attachments

(3 files)

We are going to analysis the amount of tracking resources, the interest values are daily statistics of

  1. total number of HTTP transactions
  2. total number of connections
  3. total number of bytes

We're going to collect above values based on their Class of Service flags and the category on the tracking list.

Data Types

Class of Service

Flags Load Type
nsIClassOfService::Leader A
w/o nsIRequest::LOAD_BACKGROUND B
w/ nsIRequest::LOAD_BACKGROUND C

Tracking Category

Category List Category
Content I
Analytics/Advertising/Social II
Fingerprinting III

Telemetry Targets

There are 22 values, denoted by Y for both normal/private browsing mode.

Normal Mode

Y = 0 for first party
Y = 1 for non-listed third party type

Y Type A Type B Type C
Category I 2 3 4
Category II 5 6 7
Category III 8 9 10

Private Mode

Y = 11 for first party
Y = 12 for non-listed third party type

Y Type A Type B Type C
Category I 13 14 15
Category II 16 17 18
Category III 19 20 21

Gary, thank you for filing this bug. Nice tables! :)

Gary, will you be submitting a patch? Or should we find someone on the necko team?

Flags: needinfo?(xeonchen)
Priority: -- → P3
Whiteboard: [necko-triaged]

(In reply to Valentin Gosu [:valentin] from comment #2)

Gary, will you be submitting a patch? Or should we find someone on the necko team?

I talked with Gary offline. He doesn't have cycles to implement this patch in next 2~3 weeks.
We would appreciate if someone on the Necko team can take it (It not, Gary will come back to work on this later).
Thank you, Valentin!

Flags: needinfo?(xeonchen)
Assignee: nobody → xeonchen
Status: NEW → ASSIGNED
Priority: P3 → P1
Depends on: 1530481

This bug isn't related to prio.

No longer depends on: 1530481
Attached file request.md
Attachment #9054192 - Flags: data-review?(chutten)
Comment on attachment 9054192 [details]
request.md

Preliminary note:

Please expand a touch on the data being collected. It is important to note that the transactions, connections, and bytes are being recorded per-"category" and that the number and disposition of categories is known ahead of time. Please list the categories that these will be divided into.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions files [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json) and [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml), and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 73.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :xeonchen is responsible for renewing or removing the collection before it expires in Firefox 73.

---
Result: datareview+, pending additional information about the data being collected (per the preliminary note)
Attachment #9054192 - Flags: data-review?(chutten) → data-review+
Depends on: 1541238
Attachment #9054031 - Attachment description: Bug 1533363 - Part 2: Add HTTP Traffic category Telemetry → Bug 1533363 - Part 2: Add HTTP Traffic category Telemetry;
Pushed by xeonchen@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/c3488f465c29
Part 1: Add HttpTrafficAnalyzer service; r=dragana
https://hg.mozilla.org/integration/autoland/rev/99636457aa95
Part 2: Add HTTP Traffic category Telemetry; r=chutten
Flags: needinfo?(xeonchen)
Blocks: 1542069

Pushed by cbrindusan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/13bff8b50313
Part 1: Add HttpTrafficAnalyzer service; r=dragana
https://hg.mozilla.org/integration/autoland/rev/7c3de15af00b
Part 2: Add HTTP Traffic category Telemetry; r=chutten

Keywords: checkin-needed
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

Gary, please request uplift to Beta 67 for this bug and any required ones, such as:
Bug 1542069 - Enable Telemetry on tracking resources traffic
Bug 1541238 - Delays Tracking Loads

Flags: needinfo?(xeonchen)

Comment on attachment 9052886 [details]
Bug 1533363 - Part 1: Add HttpTrafficAnalyzer service;

Beta/Release Uplift Approval Request

  • Feature/Bug causing the regression: N/A
  • User impact if declined:
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: Bug 1542069, Bug 1541238
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This bug only adds some telemetry probes.
    Bug 1542069 enables telmetry probing introduced in this bug.
    Bug 1541238 is default disabled.
  • String changes made/needed: N/A
Flags: needinfo?(xeonchen)
Attachment #9052886 - Flags: approval-mozilla-beta?
Attachment #9054031 - Flags: approval-mozilla-beta?

Please request uplift in Bug 1542069 and Bug 1541238 as well.

Flags: needinfo?(xeonchen)

(In reply to Pascal Chevrel:pascalc from comment #16)

Please request uplift in Bug 1542069 and Bug 1541238 as well.

Thanks for reminding :)

Flags: needinfo?(xeonchen)

Comment on attachment 9052886 [details]
Bug 1533363 - Part 1: Add HttpTrafficAnalyzer service;

Low risk, uplift approved for 67 beta 11, thanks.

Attachment #9052886 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9054031 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Backed out 4 changesets (Bug 1541238, Bug 1542069, Bug 1533363) for build bustages - missing Bug 1536488

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=74c650902989c6b30b913c4b20639bf71ba11aef&selectedJob=240336243

Backout link: https://hg.mozilla.org/releases/mozilla-beta/rev/ca196717df0793c823865553b8907cf28573597f

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=240336243&repo=mozilla-beta&lineNumber=13796

[task 2019-04-15T12:55:45.628Z] 12:55:45 INFO - make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/xpconnect/tests/components/native'
[task 2019-04-15T12:55:46.569Z] 12:55:46 INFO - make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http'
[task 2019-04-15T12:55:46.577Z] 12:55:46 INFO - /builds/worker/workspace/build/src/clang/bin/clang++ -o Unified_cpp_protocol_http2.i_o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/netwerk/protocol/http -I/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/extensions/auth -I/builds/worker/workspace/build/src/netwerk/base -I/builds/worker/workspace/build/src/netwerk/cookie -I/builds/worker/workspace/build/src/netwerk/url-classifier -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fcrash-diagnostics-dir=/builds/worker/artifacts -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fomit-frame-pointer -funwind-tables -Werror -fprofile-instr-generate -MD -MP -MF .deps/Unified_cpp_protocol_http2.i_o.pp /builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http/Unified_cpp_protocol_http2.cpp
[task 2019-04-15T12:55:46.578Z] 12:55:46 INFO - In file included from /builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http/Unified_cpp_protocol_http2.cpp:29:
[task 2019-04-15T12:55:46.580Z] 12:55:46 ERROR - /builds/worker/workspace/build/src/netwerk/protocol/http/nsHttpChannel.cpp:1303:28: error: no member named 'CLASSIFIED_ANY_BASIC_TRACKING' in 'nsIHttpChannel::ClassificationFlags'; did you mean 'CLASSIFIED_ANY_TRACKING'?
[task 2019-04-15T12:55:46.581Z] 12:55:46 INFO - } else if (flags & CF::CLASSIFIED_ANY_BASIC_TRACKING) {
[task 2019-04-15T12:55:46.582Z] 12:55:46 INFO - ^
[task 2019-04-15T12:55:46.582Z] 12:55:46 INFO - CLASSIFIED_ANY_TRACKING
[task 2019-04-15T12:55:46.583Z] 12:55:46 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/include/nsIHttpChannel.h:200:5: note: 'CLASSIFIED_ANY_TRACKING' declared here
[task 2019-04-15T12:55:46.584Z] 12:55:46 INFO - CLASSIFIED_ANY_TRACKING = 125,
[task 2019-04-15T12:55:46.585Z] 12:55:46 INFO - ^
[task 2019-04-15T12:55:46.586Z] 12:55:46 INFO - 1 error generated.
[task 2019-04-15T12:55:46.587Z] 12:55:46 INFO - /builds/worker/workspace/build/src/config/rules.mk:805: recipe for target 'Unified_cpp_protocol_http2.i_o' failed
[task 2019-04-15T12:55:46.588Z] 12:55:46 ERROR - make[5]: *** [Unified_cpp_protocol_http2.i_o] Error 1
[task 2019-04-15T12:55:46.589Z] 12:55:46 INFO - make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/netwerk/protocol/http'
[task 2019-04-15T12:55:46.591Z] 12:55:46 INFO - make[5]: *** Waiting for unfinished jobs....
[task 2019-04-15T12:55:46.592Z] 12:55:46 INFO - make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/xpconnect/src'

Flags: needinfo?(xeonchen)
Flags: needinfo?(pascalc)
Depends on: 1536488

Not sure that all the dependencies are met.

We decided to hold off on backporting this to beta.

(In reply to :Ehsan Akhgari from comment #23)

We decided to hold off on backporting this to beta.

Status: REOPENED → RESOLVED
Closed: 7 months ago6 months ago
Flags: needinfo?(xeonchen)
Resolution: --- → FIXED
Flags: needinfo?(pascalc)
Flags: qe-verify-
Attachment #9054031 - Flags: approval-mozilla-beta+ → approval-mozilla-beta-
Attachment #9052886 - Flags: approval-mozilla-beta+ → approval-mozilla-beta-
Depends on: 1546796
Depends on: 1547389
Blocks: 1546699
You need to log in before you can comment on or make changes to this bug.