Closed Bug 1358149 Opened 7 years ago Closed 11 months ago

Address fingerprinting issues with AudioContext

Categories

(Core :: Web Audio, enhancement, P2)

enhancement

Tracking

()

RESOLVED FIXED
118 Branch
Tracking Status
relnote-firefox --- 118+
firefox-esr102 --- wontfix
firefox-esr115 --- wontfix
firefox55 --- wontfix
firefox116 --- wontfix
firefox117 --- wontfix
firefox118 --- fixed

People

(Reporter: padenot, Assigned: karlt)

References

(Depends on 1 open bug)

Details

(Keywords: perf-alert, Whiteboard: [tor 13017][fingerprinting][fp-triaged][fpp:m5])

Attachments

(10 files, 5 obsolete files)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
See this change in WebKit:

https://trac.webkit.org/changeset/214721/webkit

DoubleClick (and maybe others) are using an `AudioContext` that produces a silent output to fingerprint users. I haven't been able to confirm this first-hand yet.

I think we should be able to do more-or-less the same as WebKit here, we already have all the infra to detect silent AudioContext, and we need to track whether or not the page is in the foreground.

This is a compatibility risk because `ScriptProcessorNode` events won't fire if not connected to the destination AND the output of the `AudioContext` is silent AND the page is in the background.
Assignee: nobody → padenot
Priority: -- → P1
Or maybe even on the focused tab?
Rank: 17
This is an assigned P1 bug without activity in two weeks. 

If you intend to continue working on this bug for the current release/iteration/sprint, remove the 'stale-bug' keyword.

Otherwise we'll reset the priority of the bug back to '--' on Monday, August 28th.
Keywords: stale-bug
Mass change P1->P2 to align with new Mozilla triage process
Priority: P1 → P2
Whiteboard: [tor 13017]
I'd like to propose any fix to this bug be tied to "privacy.resistFingerprinting".
At the very least, yes. If we can figure out a way to not break a bunch of websites and also make this kind of fingerprinting not working, I'd look into making it the default, though.
This extension seems to be a temporary solution
https://addons.mozilla.org/en-US/firefox/addon/audioctx-fingerprint-defender/
Whiteboard: [tor 13017] → [tor 13017][fingerprinting]
No longer blocks: uplift_tor_fingerprinting
Whiteboard: [tor 13017][fingerprinting] → [tor 13017][fingerprinting][fp-triaged]

See [1]. Tom, I emailed you a copy.

Safari running in iOS, by default, automatically suspend new AudioContext’s that are not created in response to a tap-in display. Since our fingerprinting code is simply executed, this explains the existence of null values

The tests are a little old, Safari results are for v10 and v11. Assuming Safari still do this, as a first defense against Audio FP'ing, could implement this approach?

[1] A Web Browser Fingerprinting Method Based on the Web Audio API
~ The British Computer Society 2019
~ JORDAN S. QUEIROZ and E DUARDO L. FEITOSA
~ doi:10.1093/comjnl/bxy146


This extension seems to be a temporary solution
https://addons.mozilla.org/en-US/firefox/addon/audioctx-fingerprint-defender/

I would instead look at the methods/code used in CanvasBlocker. I believe the above addon is "flawed" and not very effective (from my travels and reading) - instead look at [1], [2] for better insight

[1] https://addons.mozilla.org/firefox/addon/canvasblocker/
[2] https://github.com/kkapsner/CanvasBlocker/

Related tickets (if anyone wants to link them)

Not sure if this was intentional (in tests on my windows machine)

  • FF61 and lower: an-smoothingTimeConstant gave high precision: 15 decimal places
  • FF62+: an-smoothingTimeConstant is 1 decimal place

If an-smoothingTimeConstant is reduced to 1 decimal place precision for all OSes, that's great (couldn't quickly spot a bugzilla)

(In reply to Simon Mainey from comment #8)

Not sure if this was intentional (in tests on my windows machine)

  • FF61 and lower: an-smoothingTimeConstant gave high precision: 15 decimal places
  • FF62+: an-smoothingTimeConstant is 1 decimal place

I find this really surprising.

(In reply to Paul Adenot (:padenot) from comment #9)

I find this really surprising.

Just to clarify: these are all "essentially" vanilla profiles, so no RFP (minor tweaks e.g. stop auto-updates), Firefox portables, final versions per major release - including ESRs - from FF60 through to Nightly: I have the same on a Linux Mint VM. I'll check those and get back to you.

My android is identical to windows machine (except for the addition of outputLatency and ac-sampleRate)

https://ghacksuserjs.github.io/TorZillaPrint/TorZillaPrint.html#audio - [run tests], show details

           ac-baseLatency: 0
            ac-sampleRate: 48000
                 ac-state: suspended
       ac-maxChannelCount: 2
        ac-numberOfInputs: 1
       ac-numberOfOutputs: 0
          ac-channelCount: 2
      ac-channelCountMode: explicit
 ac-channelInterpretation: speakers
               an-fftSize: 2048
     an-frequencyBinCount: 1024
           an-minDecibels: -100
           an-maxDecibels: -30
 an-smoothingTimeConstant: 0.8
        an-numberOfInputs: 1
       an-numberOfOutputs: 1
          an-channelCount: 1
      an-channelCountMode: max
 an-channelInterpretation: speakers

It'd be nice to get a larger set of results

FF 71.0 on Linux (Manjaro): 0.8

(In reply to Simon Mainey from comment #10)

My android is identical to windows machine (except for the addition of outputLatency and ac-sampleRate)

sorry: I meant ac-baseLatency , not ac-SampleRate

Now that Bug 531915 is fixed, we can pursue using the new fdlibm functions for Web Audio to eliminate the fingerprint.

Depends on: 531915

194 Firefox AudioContext key results, excluding ac-outputLatency and ac-sampleRate which are covered by RFP, from the data set of A Study of Feasibility and Diversity of Web Audio Fingerprints. The data set is very light on Android and Linux results

CONTROL: 183 results
  "ac-baseLatency":0,
  "ac-channelCount":2,
  "ac-channelCountMode":"explicit",
  "ac-channelInterpretation":"speakers",
  "ac-maxChannelCount":2,
  "ac-numberOfInputs":1,
  "ac-numberOfOutputs":0,
  "ac-state":"suspended",
  "an-channelCount":2,
  "an-channelCountMode":"max",
  "an-channelInterpretation":"speakers",
  "an-fftSize":2048,
  "an-frequencyBinCount":1024,
  "an-maxDecibels":-30,
  "an-minDecibels":-100,
  "an-numberOfInputs":1,
  "an-numberOfOutputs":1,
  "an-smoothingTimeConstant":0.8,

DIFF: 7 results: all windows
  "an-channelCount":1,

DIFF: 1 result: win10 x64; 78
  "ac-maxChannelCount":0,
  "an-channelCount":1,

DIFF: 1 result: win10 x64; 87
  "ac-maxChannelCount":8,

1 result: win10 x64; 87
  "ac-maxChannelCount":0,

1 result: win6.1 x64; 86
  "ac-maxChannelCount":1,

That seems like a fairly thin short tail. Paul, can those two items be covered by RFP?

Flags: needinfo?(padenot)

(In reply to Simon Mainey from comment #14)

DIFF: 7 results: all windows
"an-channelCount":1,

Thisi is real surprising. Per spec this defaults to 2, so this is likely something the fingerprinting code sets.

DIFF: 1 result: win10 x64; 78
"ac-maxChannelCount":0,
"an-channelCount":1,

This is when running on a machine that has no audio hardware I think. I don't understand the an-channelCount for the reasons described above.

DIFF: 1 result: win10 x64; 87
"ac-maxChannelCount":8,

This iis running on a machine that has a 7.1 soundcard.

1 result: win10 x64; 87
"ac-maxChannelCount":0,

Same as above.

1 result: win6.1 x64; 86
"ac-maxChannelCount":1

Again running with special hardware (mono audio device).

This ac-maxChannelCount is already covered by RFP: https://searchfox.org/mozilla-central/source/dom/media/webaudio/AudioContext.cpp#714-716 (hardcode to stereo which is the overwhelming majority)

Flags: needinfo?(padenot)

(In reply to Paul Adenot (:padenot) from comment #15)

Thisi is real surprising. Per spec this defaults to 2, so this is likely something the fingerprinting code sets.

None of the samples would have been running RFP, especially given that none of the RFP per-OS results for ac-outputLatency were present. So it's something else. Perhaps we can hardcode this to 2 if RFP is enabled to tighten things up, or debug why it doesn't return 2 without RFP. Who knows, maybe they were running some randomizing extension, which seems unlikely given other analysis

This ac-maxChannelCount is already covered by RFP: https://searchfox.org/mozilla-central/source/dom/media/webaudio/AudioContext.cpp#714-716 (hardcode to stereo which is the overwhelming majority)

Excellent. My bad.

(In reply to Simon Mainey from comment #16)

None of the samples would have been running RFP, especially given that none of the RFP per-OS results for ac-outputLatency were present. So it's something else. Perhaps we can hardcode this to 2 if RFP is enabled to tighten things up, or debug why it doesn't return 2 without RFP. Who knows, maybe they were running some randomizing extension, which seems unlikely given other analysis

It's always 2 unless set explicitely regardless of RFP, it's a normative spec requirement. It would be best to check the code to see where it sets it.

In this try run we might have figured out the AudioContext fingerprint; but https://audiofingerprint.openwpm.com/ still shows different OscillatorNode values for us. Our patches are here part one and part two.

I'm not sure what could be causing the difference; looking at OscillatorNode.cpp the only thing that jumps out to me in that file is float(2 * M_PI).... Paul do you have any ideas?

Flags: needinfo?(padenot)

RFP alone currently alters these OscillatorNode and hybrid OscillatorNode/DynamicsCompressor (I assume because some AudioContext keys are hardcoded and used downstream). Also the WPM test fails to get outputLatency correctly and always reports it as 0 (due to order/lag in requesting the info) but I don't think this matters in these tests

Tom, this is probably caused by this particular using AnalyzerNode.getFloatFrequencyData, and Firefox using a different FFT implementation than Chrome. We already compile the one they use (for libwebrtc), and it's faster than the one we use, so there's hope.

Flags: needinfo?(padenot)
Severity: normal → S3
Assignee: padenot → nobody
Keywords: stale-bug

The FFT implementation may be causing different fingerprints due to its use of system math libraries for sin() and cos(). If the goal is to minimize the fingerprinting entropy between Firefox users, then I would expect that persuading the FFT implementation to use fdlibm is more important than the choice of library.

The MOZ_LIBAV_FFT implementation is third party and regularly updated, so interposing math headers as was done with GDK headers may be a sustainable way to switch to fdlibm.

(In reply to Tom Ritter [:tjr] from comment #18)

In this try run we might have figured out the AudioContext fingerprint; but https://audiofingerprint.openwpm.com/ still shows different OscillatorNode values for us.

Was this between two Firefox users on the same OS?

See Also: → 1816389
See Also: → 1816381

The FFT implementation may be causing different fingerprints

FYI: AFAICT bug 1519004 changed the sum on my android. Previously I was 35.74996031448245, now I am a new unseen before value of 35.74995414912701 (you can use https://audiofingerprint.openwpm.com/). Whether than changes android entropy, who knows

all platforms, from research

	let knownGood = [
		// .7383...
		35.73832903057337, 35.7383295930922, 35.73833039775491, 35.73833402246237,
		// .7499
		35.74996018782258, 35.74996031448245, 35.7499681673944,
		35.74995414912701, // <- my android 113+: 1519004 ?
	]

Thanks for the confirmation, Simon, this was somewhat expected. The reason for changing this was unrelated to fingerprinting, it was a performance improvement, and consolidation of code to use less different fft libraries.

We have plans to move all of this again soon to another fft, this time using the same code on all platforms / CPU architectures, and there are other plans to change trig functions so that they're similar on all OSes / architectures as well.

openmax_dl's FFT, used on arm32, doesn't use system sin() and cos() because it has its own hard-coded tables for fixed and floating point.

We don't build ffvpx with CONFIG_HARDCODED_TABLES so av_rdft_init() is currently using system libraries to initialize its tables.

Whiteboard: [tor 13017][fingerprinting][fp-triaged] → [tor 13017][fingerprinting][fp-triaged][fpp:m3]
Summary: Address fingerprinting using AudioContext → Address fingerprinting issues with AudioContext

What we need to do:

  1. Make ffvpx use fdlibm for irrational math functions, by interposing headers. I would hope that the precise functions such as floor() and fabs() do not show differences in return values across system libraries, and so we may not need to use fdlibm for these.
  2. Finish D166710 or probably more durably use the same header interposition for dom/media/webaudio.
  3. Check that there are no significant performance regressions. Performance comparisons of fdlibm have showed varied results [1][2], but, skimming through the use of irrational functions in Web Audio, I didn't see any that I expect to be critical for performance. Exponentials and powers for AudioParams might be the more sensitive operations. It would be nice if we don't need to put fdlibm behind a pref in Web Audio.
  4. Verify that these changes reduce the fingerprint to no more than already exists across different architectures. I'll likely need some help with this.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=531915#c31
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=933257#c139

Whiteboard: [tor 13017][fingerprinting][fp-triaged][fpp:m3] → [tor 13017][fingerprinting][fp-triaged][fpp:m5]
Assignee: nobody → karlt
Status: NEW → ASSIGNED

(In reply to Paul Adenot (:padenot) from comment #15)

(In reply to Simon Mainey from comment #14)

DIFF: 7 results: all windows
"an-channelCount":1,

Thisi is real surprising. Per spec this defaults to 2,

This was resolved in bug 1666490.

This was resolved in bug 1666490.

Thanks. That explains it. I just looked back at the dataset (which the researchers gave me). The paper was July 2021 and all those with "an-channelCount":1, are FF78 or FF81 (guessing the tests took place around August 2020). Bug fix was FF83. Seems legit :)

This will facilitate making the functions available to C code.

These operations are used from dom/media/webaudio.

Depends on D185778

This provides a way to use fdlibm from third party C code without
modification, such as media/ffvpx, which is regularly updated and so this
avoids patching the vendored library.

A similar approach for C++ gets more complicated and is in a subsequent patch.

Depends on D185779

This includes the rdft FFT implementation.

Depends on D185780

This will facilitate switching these functions to fdlibm, which maps to math.h.

Depends on D185781

Depends on D185783

to avoid risk of log(2^order) / M_LN2 < order.

Depends on D185782

I was seeing hypot undefined in object files so NS_hypot's __builtin_hypot
call does not appear to be inlining the implementation.

Depends on D185893

Keywords: leave-open
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4adc8ab88a03
use prefix instead of namespace for fdlibm function names r=arai
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bd2eb4beadc5
use C language linkage for public fdlibm functions r=arai
Attachment #9348118 - Attachment is obsolete: true
Attachment #9348119 - Attachment is obsolete: true
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/08b3c95b3688
add log10f and hypotf from fdlibm r=arai
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6bf7d39ac480
switch inexact cmath function usage in Web Audio to math.h r=padenot
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cd6b30e9acce
use integer log2 function for FFT order r=padenot
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/60768557aa15
add a header to substitute fdlibm math functions for inexact system library functions r=arai
https://hg.mozilla.org/integration/autoland/rev/969e577218dc
use fdlibm for inexact math functions in ffvpx/libavcodec r=padenot,firefox-build-system-reviewers,glandium
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b6a176bcaf0b
use fdlibm for inexact math functions in webaudio r=padenot
Keywords: leave-open
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6eec009a00bf
use fdlibm_hypot instead of NS_hypot in AnalyserNode r=padenot
https://hg.mozilla.org/integration/autoland/rev/05da0ca6db46
use fdlibm's hypot and atan2 instead of libm's cabs and carg r=padenot

:karlt do you think this could be nominated for a relnote? https://wiki.mozilla.org/Release_Management/Release_Notes_Nomination

Flags: needinfo?(karlt)
Regressions: 1848468

(In reply to Pulsebot from comment #50)

Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b6a176bcaf0b
use fdlibm for inexact math functions in webaudio r=padenot

== Change summary for alert #39288 (as of Sat, 12 Aug 2023 11:10:34 GMT) ==

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
11% webaudio linux1804-64-shippable-qr fission webrender 134.17 -> 118.83 Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=39288

Hi Karl! Do you think the changes introduced by b6a176bcaf0bb5d9c9035ff8bed63bf0e79da30c could have caused this improvement ?

Depends on: 1849113

(In reply to Alex Finder from comment #54)

Hi Karl! Do you think the changes introduced by b6a176bcaf0bb5d9c9035ff8bed63bf0e79da30c could have caused this improvement ?

Thank you for pointing out that. Yes b6a176bcaf0bb5d9c9035ff8bed63bf0e79da30c would have been the cause of those changes.

For Linux and Mac, a comparison is available with a slightly narrower range.

Linux results have improved across all subtests. The most notable differences are:

Base New Delta Confidence Total Runs
Geometric Mean 135 ± 1.24 118.33 ± 0.87 -12.35% 20.76 (high) 6 / 6
Granular synthesis 300.67 ± 1.78 247.17 ± 1.82 -17.79% 18.75 (high) 6 / 6
Periodic Wave with Automation 293.67 ± 1.75 180 ± 0.86 -38.71% 52.00 (high) 6 / 6
Simple mixing (100 different buffers) 670.5 ± 1.75 628.33 ± 0.84 -6.29% 8.02 (high) 6 / 6
Simple mixing (100x same buffer) 665.67 ± 2.25 623.33 ± 1.37 -6.36% 6.02 (high) 6 / 6
Stereo Panning with Automation 245.17 ± 1.64 140 ± 0.45 -42.90% 63.29 (high) 6 / 6
Substractive synth 454.33 ± 1.61 266 ± 0.79 -41.45% 60.64 (high) 6 / 6
Synth 677.33 ± 1.96 545 ± 0.94 -19.54% 22.75 (high) 6 / 6

For Mac, subtests are mixed, but there are a couple of large regressions. MacOS appears to have some different implementations of sin and cos and exp2f at least.

Base New Delta Confidence Total Runs
Downmix without resampling (Stereo -> Mono) 45.8 ± 1.83 57 ± 8.95 24.45% 4.85 (med) 5 / 5
Geometric Mean 100.2 ± 0.83 102.6 ± 8.03 2.40% 0.65 (low) 5 / 5
Granular synthesis 303.6 ± 0.5 282.4 ± 7.85 -6.98% 2.13 (low) 5 / 5
Simple mixing (100 different buffers) 719.4 ± 0.74 777 ± 7.04 8.01% 2.34 (low) 5 / 5
Simple mixing (100x same buffer) 685.8 ± 0.69 739.6 ± 7.04 7.84% 2.30 (low) 5 / 5
Simple source test without resampling 25.2 ± 3.32 23.4 ± 10.73 -7.14% 1.52 (low) 5 / 5
Stereo Panning with Automation 100.6 ± 1.33 130 ± 8.75 29.22% 5.74 (high) 5 / 5

WINNT results the comparison has a slightly larger range due to build failures on intermediate revisions, but I expect the changes are still due to the same revision.

I see only improvements (or unchanged) for 32-bit WINNT subtests:

Base New Delta Confidence Total Runs
Geometric Mean 118.4 ± 0.46 111.8 ± 0.75 -5.57% 14.76 (high) 5 / 5
Granular synthesis 338.4 ± 0.45 319 ± 0.63 -5.73% 17.28 (high) 5 / 5
Simple mixing (100 different buffers) 768.8 ± 0.43 750.2 ± 0.67 -2.42% 6.94 (high) 5 / 5
Stereo Panning with Automation 278 ± 0.57 120.6 ± 1.26 -56.62% 160.65 (high) 5 / 5
Substractive synth 354.6 ± 1.76 320.8 ± 0.26 -9.53% 12.03 (high) 5 / 5
Synth 673 ± 0.32 648.4 ± 0.48 -3.66% 14.55 (high) 5 / 5

64-bit WINNT subtests have some notable regressions and a small improvement.

Base New Delta Confidence Total Runs
Geometric Mean 94.4 ± 2.76 95.8 ± 0.87 1.48% 1.14 (low) 5 / 5
Periodic Wave with Automation 136.8 ± 0.61 134 ± 0 -2.05% 7.48 (high) 5 / 5
Simple source test without resampling (Stereo and positional) 39 ± 0 40 ± 1.77 2.56% 3.16 (med) 5 / 5
Stereo Panning with Automation 90.6 ± 6.46 102.6 ± 1.11 13.25% 4.50 (med) 5 / 5
Substractive synth 221.6 ± 0.94 228.4 ± 0.66 3.07% 5.92 (high) 5 / 5
Synth 467.2 ± 0.95 543 ± 1.07 16.22% 23.24 (high) 5 / 5

I'll file some bugs to look at countering the regressions.

Results from https://github.com/arkenfox/user.js/issues/1701 and QA confirm that variations in AudioContext fingerprinting scripts across operating systems with fixed sample rates have been removed, as intended. The operating system is exposed to fingerprinters via other means but this result does assure that variations in OS libraries (potentially between versions or configurations) are no longer exposed through AudioContext.

Fingerprints still differ between ARM and Intel machines. The variation between x86 and amd64 on Windows 11 has been removed, which is an unexpected bonus. Remaining hardware differences are still plausible so we should keep a look out for them.

So there are notable reductions in entropy here, but the AudioContext sampleRate, outputLatency, and maxChannelCount carry potentially more entropy unless preferences are configured appropriately. The reductions here are of limited value unless privacy.resistFingerprinting = true, or fingerprintingProtection is enabled together with privacy.fingerprintingProtection.overrides = "+AudioContext,+AudioSampleRate", to remove these other sources of variation.

RFPTarget::JSMathFdlibm would also need to be set (as with privacy.resistFingerprinting = true, or with fingerprintingProtection and privacy.fingerprintingProtection.overrides = "+AudioContext,+AudioSampleRate,+JSMathFdlibm") to avoid exposure elsewhere of the same entropy removed here.

Some of these other sources of entropy might be removed by default in some browsing modes in future changes.

Release Note Request (optional, but appreciated)
[Affects Firefox for Android]:
Yes.
[Why is this notable]:
[Suggested wording]:
Web Audio in Firefox now uses the FDLIBM math library on all systems to
improve anonymity with resist fingerprinting mode.
[Links (documentation, blog post, etc)]:
https://support.mozilla.org/en-US/kb/firefox-protection-against-fingerprinting

relnote-firefox: --- → ?
Flags: needinfo?(karlt)
Keywords: perf-alert

Added a slightly reworded note to the Nightly release notes. Keeping the relnote? flag open to keep it on the radar for inclusion in our final release notes.

No longer depends on: 1849113
See Also: → 1849113
Depends on: 1849806
Depends on: 1851755
Depends on: 1851757
See Also: → 1851758
Attachment #9312037 - Attachment is obsolete: true
Attachment #9312036 - Attachment is obsolete: true
Attachment #9312035 - Attachment is obsolete: true
Duplicate of this bug: 1760669
Duplicate of this bug: 1658836
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: