Closed Bug 2045783 Opened 5 months ago Closed 4 months ago

Heap use-after-free read in [@ mozilla::net::merge_rrset] via cross-thread double finalization in TRRQuery::CompleteLookup

Categories

(Core :: Networking: DNS, defect, P1)

defect

Tracking

()

RESOLVED FIXED
154 Branch
Tracking Status
firefox-esr115 153+ fixed
firefox-esr140 153+ fixed
firefox151 --- wontfix
firefox152 --- wontfix
firefox153 + fixed
firefox154 + fixed

People

(Reporter: bugmon, Assigned: jesup)

Details

(6 keywords, Whiteboard: [necko-triaged][necko-priority-queue][adv-main153+r][adv-esr140.13+r][adv-esr115.38+r])

Attachments

(11 files, 8 external files)

1.91 KB, text/html
Details
1.54 KB, patch
Details | Diff | Splinter Review
1.57 KB, text/plain
Details
3.78 KB, text/plain
Details
868 bytes, text/plain
Details
1.73 KB, patch
Details | Diff | Splinter Review
9.48 KB, text/plain
Details
5.60 KB, text/plain
Details
234 bytes, text/plain
Details
89 bytes, text/plain
Details
14.25 KB, text/plain
Details

mozilla::net::TRRQuery (netwerk/dns/TRRQuery.h) performs its DNS-over-HTTPS completion bookkeeping without synchronization: mTrrLock only guards the mTrrA/mTrrAAAA/mTrrByType request pointers, while the result state used by the finalization path of TRRQuery::CompleteLookup — the RefPtr<AddrInfo> members mAddrInfoA/mAddrInfoAAAA, the mTrrAUsed/mTrrAAAAUsed state enums, mTRRSuccess, and mCalledCompleteLookup — is read and written with no lock. The completion accounting itself is a check/decrement/re-read TOCTOU on the Atomic<uint32_t> mTRRRequestCounter: 'if (mTRRRequestCounter) { mTRRRequestCounter--; pendingRequest = (mTRRRequestCounter != 0); }'. When the A and AAAA completions of a single TRRQuery execute concurrently on two threads, both threads can observe the separate re-read as 0 and both run the unlocked finalization tail. One thread passes mAddrInfoAAAA as a raw AddrInfo* into merge_rrset() (no reference held), while the other thread's finalization clears the member RefPtr and unwinds the last reference in TRR::ReturnData — the AddrInfo is freed while merge_rrset is dereferencing it, a classic cross-thread heap use-after-free. The merged result, partially read from freed memory, is then installed into the parent-process DNS cache, additionally enabling DNS cache poisoning.

The two-thread precondition is reachable in shipping builds: nsHostResolver::MaybeRetryTRRLookup (nsHostResolver.cpp:1328-1334, added by Bug 1825538) calls TRRService::DontUseTRRThread() whenever a TRR_ONLY lookup's DoH channel fails with NS_ERROR_PROXY_UNAUTHORIZED / NS_ERROR_PROXY_AUTHENTICATION_FAILED — a condition fully controlled by an on-path attacker or malicious proxy answering one DoH CONNECT with HTTP 407. After this one-way flip, all newly dispatched TRR requests run on the main thread using a plain nsHttpChannel (callbacks on the main thread), while TRR requests dispatched before the flip still complete on the socket thread via TRRServiceChannel. A malicious DoH server stalls a victim lookup's A and AAAA requests across the flip, answers the AAAA with a CNAME-only response (no addresses, RA flag clear) so TRR::FollowCname dispatches a successor request post-flip (main thread, inheriting the AAAA counter slot), and then releases the A response (socket thread) and the successor's AAAA response (main thread) near-simultaneously — putting both completions of one TRRQuery on two threads.

The natural race windows are instruction-scale, so the proof-of-concept build adds two PR_Sleep() race-wideners strictly inside the pre-existing windows (between the counter decrement and re-read, and between the raw-pointer argument fetch and its first dereference in merge_rrset); they change no logic, locking, or lifetime, and the MOZ_DIAGNOSTIC_* guards remain in place and are never hit (both threads pass the counter check while it is non-zero — the exact TOCTOU). On release builds the diagnostic guards are compiled out entirely, so naturally-hit races silently double-finalize. The crash was reproduced deterministically through the browser evaluator and independently verified: ASAN heap-use-after-free READ in merge_rrset <- TRRQuery::CompleteLookup on the Socket Thread, freed and allocated by the Main Thread in TRR::ReturnData via nsHttpChannel.

Build Info

Affected Code

File: netwerk/dns/TRRQuery.h, line 82-102

  Mutex mTrrLock;  // lock when accessing the mTrrA[AAA] pointers
  RefPtr<mozilla::net::TRR> mTrrA MOZ_GUARDED_BY(mTrrLock);
  RefPtr<mozilla::net::TRR> mTrrAAAA MOZ_GUARDED_BY(mTrrLock);
  RefPtr<mozilla::net::TRR> mTrrByType MOZ_GUARDED_BY(mTrrLock);
  // |mTRRRequestCounter| indicates the number of TRR requests that were
  // dispatched sucessfully. ... Note that nsHostResolver::CompleteLookup is
  // only called when this counter equals to 0.
  Atomic<uint32_t> mTRRRequestCounter{0};

  uint8_t mTRRSuccess = 0;  // number of successful TRR responses
  bool mCalledCompleteLookup = false;
  ...
  RefPtr<mozilla::net::AddrInfo> mAddrInfoA;     // no lock!
  RefPtr<mozilla::net::AddrInfo> mAddrInfoAAAA;  // no lock!

File: netwerk/dns/TRRQuery.cpp, line 230-240

  bool pendingRequest = false;
  if (mTRRRequestCounter) {
    mTRRRequestCounter--;
    pendingRequest = (mTRRRequestCounter != 0);  // TOCTOU: separate re-read;
                                                 // both racing threads can see 0
  } else {
    MOZ_DIAGNOSTIC_CRASH("Request counter is messed up");
  }
  if (pendingRequest) {  // There are other outstanding requests
    return LOOKUP_OK;
  }

File: netwerk/dns/TRRQuery.cpp, line 242-251

  if (mRecord->af == AF_UNSPEC) {
    // merge successful records
    if (mTrrAUsed == OK) {            // unlocked read
      newRRSet = mAddrInfoA;          // unlocked RefPtr read
      status = mAResult;
      if (mTrrAAAAUsed == OK) {       // unlocked read
        // mAddrInfoAAAA passed as RAW AddrInfo* — no reference held across
        // the call; the other finalizer can free it mid-merge
        newRRSet = merge_rrset(newRRSet, mAddrInfoAAAA);
      }

File: netwerk/dns/TRRQuery.cpp, line 18-29

static already_AddRefed<AddrInfo> merge_rrset(AddrInfo* rrto,
                                              AddrInfo* rrfrom) {
  MOZ_ASSERT(rrto && rrfrom);
  // UAF read site: rrfrom points at the AddrInfo freed by the other thread
  bool isIPv6 = rrfrom->Addresses().Length() > 0 &&
                rrfrom->Addresses()[0].raw.family == PR_AF_INET6;

File: netwerk/dns/TRRQuery.cpp, line 349-356

  mAddrInfoAAAA = nullptr;   // unlocked clear: drops the member reference the
  mAddrInfoA = nullptr;      // other thread is concurrently using raw

  MOZ_DIAGNOSTIC_ASSERT(!mCalledCompleteLookup,
                        "must not call CompleteLookup more than once");
  mCalledCompleteLookup = true;  // non-atomic check-then-set
  return mHostResolver->CompleteLookup(rec, status, newRRSet, pb, aOriginsuffix,
                                       aReason, aTRRRequest);

File: netwerk/dns/nsHostResolver.cpp, line 1328-1335

  if (NS_FAILED(aFirstAttemptStatus) &&
      (aChannelStatus == NS_ERROR_PROXY_UNAUTHORIZED ||
       aChannelStatus == NS_ERROR_PROXY_AUTHENTICATION_FAILED) &&
      aAddrRec->mEffectiveTRRMode == nsIRequest::TRR_ONLY_MODE) {
    LOG(("MaybeRetryTRRLookup retry because of proxy connect failed"));
    TRRService::Get()->DontUseTRRThread();  // one-way flip: new TRRs move to
    return DoRetryTRR(aAddrRec);            // the MAIN thread (nsHttpChannel),
  }                                          // in-flight ones stay on socket thread

File: netwerk/dns/TRR.cpp, line 740-746

  RefPtr<TRR> trr =
      new TRR(mHostResolver, mRec, mCname, mType, mCnameLoop, mPB);
  trr->SetPurpose(mPurpose);
  if (!TRRService::Get()) {
    return NS_ERROR_FAILURE;
  }
  return TRRService::Get()->DispatchTRRRequest(trr);  // successor dispatched at
                                                       // response time → post-flip
                                                       // it lands on the MAIN thread

The unsynchronized completion state lives in TRRQuery.h; the TOCTOU counter check, the raw-pointer merge, and the unlocked RefPtr clears are in TRRQuery::CompleteLookup. The thread-flip enabler is in nsHostResolver.cpp, and the post-flip successor dispatch is TRR::FollowCname.

Exploit Chain

  1. Attacker controls the victim's network path (malicious Wi-Fi / captive portal / compromised proxy): operates the HTTP proxy configured in the browser and the DNS-over-HTTPS endpoint (or intercepts DoH traffic). Firefox uses DoH (TRR); custom-server/TRR_ONLY-effective lookups exist whenever TRR mode 3 is in use.
  2. Victim browses; the page's hostnames are resolved through the attacker's DoH server. The server stalls the victim lookup's A and AAAA requests (stalls can be extended indefinitely; DoH 3xx redirects additionally defeat the 1.5s TRR timeout by replacing the cancellable channel).
  3. The attacker's proxy answers one TRR_ONLY DoH CONNECT with HTTP 407 → the channel fails with NS_ERROR_PROXY_AUTHENTICATION_FAILED → nsHostResolver::MaybeRetryTRRLookup calls TRRService::DontUseTRRThread(): a permanent, global flip that moves every subsequently dispatched TRR request to the main thread (plain nsHttpChannel, main-thread callbacks) while already-dispatched requests still complete on the socket thread (TRRServiceChannel).
  4. The DoH server answers the victim's stalled AAAA request with a CNAME-only response (no addresses, RA=0): TRR::FollowCname dispatches a successor AAAA request post-flip → it runs and completes on the MAIN thread, while still occupying the original AAAA slot of the same TRRQuery (counter unchanged).
  5. The server releases the victim's A response (socket thread) and the successor's AAAA response (main thread) ~400 ms apart: both threads enter TRRQuery::CompleteLookup for the SAME TRRQuery. Due to the mTRRRequestCounter check/decrement/re-read TOCTOU, both observe the re-read as 0 and both execute the unlocked finalization.
  6. The socket thread's finalization evaluates merge_rrset(newRRSet, mAddrInfoAAAA), passing mAddrInfoAAAA as a raw AddrInfo* with no reference held; the main thread's finalization concurrently clears mAddrInfoAAAA (TRRQuery.cpp:349) and returns through TRR::ReturnData, whose stack RefPtr was the last reference → the 80-byte AddrInfo is freed while the socket thread reads it (ASAN: heap-use-after-free READ in merge_rrset).
  7. Impact beyond the UAF read: with heap grooming the freed AddrInfo can be reclaimed with attacker-influenced data before the merge completes, and the merged AddrInfo (built from freed memory) is installed into the parent-process DNS cache for the victim hostname (DNS cache poisoning); the double finalization also drives nsHostResolver bookkeeping negative (record wedging) and double-fires resolver completion paths that perform unbalanced RefPtr releases on release builds where the MOZ_DIAGNOSTIC_* guards are compiled out.

Steps to Reproduce

  1. Generate a test CA and a TLS leaf certificate for doh.example.com (see setup.sh), and install the CA via autoconfig: copy autoconfig/defaults/pref/autoconfig.js to <objdir>/dist/bin/defaults/pref/ and firefox.cfg (with the CA's DER base64 substituted) to <objdir>/dist/bin/. Autoconfig is used only to install the certificate.
  2. Apply source_patch.diff to the Firefox tree (two PR_Sleep race-window wideners inside the pre-existing TOCTOU and raw-pointer windows in netwerk/dns/TRRQuery.cpp; demonstration aid only) and build with the ASAN fuzzing mozconfig (/app/mozconfigs/mozconfig.linux.asan.fuzzing) via the build_firefox tool.
  3. Start the attacker services: python3 doh_server.py (malicious DoH server on https://127.0.0.1:8443 with control API on http://127.0.0.1:8053) and python3 proxy_server.py (CONNECT proxy on 127.0.0.1:8444 that can answer DoH CONNECTs with 407, control API on http://127.0.0.1:8055).
  4. Run trr_uaf.html through mcp__clauditor__browser_evaluator with firefox_binary=<objdir>/dist/bin/firefox, timeout=90, and the exact prefs from prefs.json (TRR mode 3 pointed at the attacker DoH server, SSL proxy pointed at the attacker proxy, plus three network.http.* prefs that make the victim fetch use a single AF_UNSPEC DNS lookup so the merge path is exercised).
  5. The testcase stages two victim lookups (stalled A+AAAA), triggers the DontUseTRRThread flip via a 6-second window of 407 CONNECT replies during a flipper lookup, releases the victim AAAA as a CNAME-only (RA=0) response to spawn the post-flip main-thread successor, then releases the successor's AAAA (main thread) and the victim's A (socket thread) 400 ms apart.
  6. Expected result: browser_evaluator reports crashed=true (parent process) with the ASAN heap-use-after-free READ in merge_rrset <- TRRQuery::CompleteLookup on the Socket Thread, freed/allocated by the Main Thread in TRR::ReturnData via nsHttpChannel.

Security Impact

  • Severity: High
  • Attacker capability: A network-level attacker (malicious Wi-Fi/captive portal or compromised proxy) who controls or intercepts the victim's DoH traffic and proxy responses can trigger a use-after-free of an AddrInfo object in the Firefox parent process. The freed 80-byte allocation can be reclaimed between the free and the use (the attacker fully paces both events via response timing and can retry indefinitely with fresh hostnames), turning the UAF read into reads of attacker-groomed memory; the resulting merged AddrInfo is installed into the parent-process DNS cache, yielding a DNS cache-poisoning primitive, and the double finalization exercises further unbalanced-release paths. This is parent-process memory corruption potentially leading to remote code execution in the most privileged Firefox process.
  • Preconditions: Victim uses DNS-over-HTTPS (TRR) and their HTTPS traffic traverses an attacker-controlled proxy (configured proxy, PAC, or captive-portal environment) so the attacker can answer a DoH CONNECT with 407 and control DoH responses/timing. The unsynchronized code ships in all builds; the natural race windows are instruction-scale, so reliable triggering requires many attacker-paced attempts (the PoC widens the existing windows with two PR_Sleep calls for deterministic demonstration; MOZ_DIAGNOSTIC_* guards are not removed and are not on the demonstrated path, and are compiled out entirely on release builds).

ASAN Report

==33133==ERROR: AddressSanitizer: heap-use-after-free on address 0x799687ecf718 at pc 0x79265b003596 bp 0x79262fee1a30 sp 0x79262fee1a28
READ of size 8 at 0x799687ecf718 thread T10
    #0 nsTArray_base<nsTArray_RelocateUsingMemutils>::Length() const dist/include/nsTArray.h:365:37
    #1 mozilla::net::merge_rrset(mozilla::net::AddrInfo*, mozilla::net::AddrInfo*) netwerk/dns/TRRQuery.cpp:29:37
    #2 mozilla::net::TRRQuery::CompleteLookup(nsHostRecord*, nsresult, mozilla::net::AddrInfo*, bool, nsTSubstring<char> const&, nsITRRSkipReason::value, mozilla::net::TRR*) netwerk/dns/TRRQuery.cpp:260:20
    #3 mozilla::net::TRR::ReturnData(nsIChannel*) netwerk/dns/TRR.cpp:623:26
    #4 mozilla::net::TRR::On200Response(nsIChannel*) netwerk/dns/TRR.cpp:779:5
    #5 mozilla::net::TRR::OnStopRequest(nsIRequest*, nsresult) netwerk/dns/TRR.cpp:891:12
    #6 mozilla::net::TRRServiceChannel::OnStopRequest(nsIRequest*, nsresult) netwerk/protocol/http/TRRServiceChannel.cpp:1286:15  [Socket Thread]
    #8 nsInputStreamPump::OnStateStop() netwerk/base/nsInputStreamPump.cpp:735:15
    #12 mozilla::net::nsSocketTransportService::Run() netwerk/base/nsSocketTransportService2.cpp:1221:18

0x799687ecf718 is located 72 bytes inside of 80-byte region [0x799687ecf6d0,0x799687ecf720)
freed by thread T0 here:
    #2 mozilla::net::AddrInfo::Release() dist/include/mozilla/net/DNS.h:165:3
    #5 RefPtr<mozilla::net::AddrInfo>::~RefPtr() dist/include/mozilla/RefPtr.h:78:7
    #6 mozilla::net::TRR::ReturnData(nsIChannel*) netwerk/dns/TRR.cpp:627:3
    #7 mozilla::net::TRR::On200Response(nsIChannel*) netwerk/dns/TRR.cpp:779:5
    #8 mozilla::net::TRR::OnStopRequest(nsIRequest*, nsresult) netwerk/dns/TRR.cpp:891:12
    #9 mozilla::net::nsHttpChannel::ContinueOnStopRequest(nsresult, bool, bool) netwerk/protocol/http/nsHttpChannel.cpp:10518:15
    #11 mozilla::net::nsHttpChannel::OnStopRequest(nsIRequest*, nsresult) netwerk/protocol/http/nsHttpChannel.cpp:10143:12  [Main Thread]

previously allocated by thread T0 here:
    #3 mozilla::net::AddrInfo::AddrInfoBuilder::AddrInfoBuilder(mozilla::net::AddrInfo*) dist/include/mozilla/net/DNS.h:208:15
    #4 mozilla::net::AddrInfo::Build() dist/include/mozilla/net/DNS.h:238:36
    #5 mozilla::net::TRR::ReturnData(nsIChannel*) netwerk/dns/TRR.cpp:607:24  [Main Thread]

SUMMARY: AddressSanitizer: heap-use-after-free (libxul.so+0x13379595)
==33133==ABORTING
Attached file trr_uaf.html —
Attached file setup.sh —
Attached file proxy_server.py —
Attached file prefs.json —
Attached patch fix.patch — — Splinter Review
Attached file doh_server.py —
Attached file crash_stack.txt —
Attached file firefox.cfg —
Attached file autoconfig.js —
Attached file prefs.js —
Group: core-security → network-core-security
Attached file (secure) —
Assignee: nobody → rjesup
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Severity: -- → S2
Priority: -- → P1
Whiteboard: [necko-triaged][necko-priority-queue]

Comment on attachment 9595137 [details]
(secure)

Security Approval Request

  • How easily could an exploit be constructed based on the patch?: Hard
  • Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
  • Which branches (beta, release, and/or ESR) are affected by this flaw, and do the release status flags reflect this affected/unaffected state correctly?: all
  • If not all supported branches, which bug introduced the flaw?: None
  • Do you have backports for the affected branches?: No
  • If not, how different, hard to create, and risky will they be?: trivial
  • How likely is this patch to cause regressions; how much testing does it need?: no risk worth speaking of
  • Is the patch ready to land after security approval is given?: Yes
  • Is Android affected?: Yes
Attachment #9595137 - Flags: sec-approval?
Attached file (secure) —

Belt-and-suspenders; allows static analyisis

Comment on attachment 9595137 [details]
(secure)

Approved to land 7/2

Attachment #9595137 - Flags: sec-approval? → sec-approval+
Whiteboard: [necko-triaged][necko-priority-queue] → [necko-triaged][necko-priority-queue][reminder-landing 2026-07-02]

13 days ago, tjr placed a reminder on the bug using the whiteboard tag [reminder-landing 2026-07-02] .

jesup, please refer to the original comment to better understand the reason for the reminder.

Flags: needinfo?(rjesup)
Whiteboard: [necko-triaged][necko-priority-queue][reminder-landing 2026-07-02] → [necko-triaged][necko-priority-queue]
Attachment #9604051 - Flags: approval-mozilla-beta?

firefox-beta Uplift Approval Request

  • User impact if declined/Reason for urgency: sandbox escape
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: super-simple, safe
  • String changes made/needed?: none
  • Is Android affected?: yes
Attachment #9604052 - Flags: approval-mozilla-beta?
Attached file (secure) —

Belt-and-suspenders; allows static analyisis

Original Revision: https://phabricator.services.mozilla.com/D305601

Attachment #9604107 - Flags: approval-mozilla-esr140?

firefox-esr140 Uplift Approval Request

  • User impact if declined/Reason for urgency: sandbox escape
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: super-simple, safe
  • String changes made/needed?: none
  • Is Android affected?: yes
Attachment #9604108 - Flags: approval-mozilla-esr140?
Attached file (secure) —

Belt-and-suspenders; allows static analyisis

Original Revision: https://phabricator.services.mozilla.com/D305601

Attachment #9604111 - Flags: approval-mozilla-esr115?

firefox-esr115 Uplift Approval Request

  • User impact if declined/Reason for urgency: sandbox escape
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: super-simple, safe
  • String changes made/needed?: none
  • Is Android affected?: yes
Attachment #9604112 - Flags: approval-mozilla-esr115?
Attached file (secure) —

Belt-and-suspenders; allows static analyisis

Original Revision: https://phabricator.services.mozilla.com/D305601

Attachment #9604052 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9604051 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Group: network-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 154 Branch
Flags: needinfo?(rjesup)
Attachment #9604107 - Flags: approval-mozilla-esr140? → approval-mozilla-esr140+
Attachment #9604108 - Flags: approval-mozilla-esr140? → approval-mozilla-esr140+
QA Whiteboard: [sec] [qa-triage-done-c154/b153]
Attachment #9604111 - Flags: approval-mozilla-esr115? → approval-mozilla-esr115+
Attachment #9604112 - Flags: approval-mozilla-esr115? → approval-mozilla-esr115+
Whiteboard: [necko-triaged][necko-priority-queue] → [necko-triaged][necko-priority-queue][adv-main153+r][adv-esr140.13+r][adv-esr115.38+r]
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: