1885529 - Investigate increase in DNS queueing time with native HTTPS records

Status: VERIFIED FIXED

(Core :: Networking: DNS, defect, P2)

Description

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

https://glam.telemetry.mozilla.org/firefox/probe/dns_native_queuing/explore?currentPage=1&os=Windows&process=parent&ref=20240308044528&timeHorizon=QUARTER&visiblePercentiles=%5B99.9%2C99%2C95%5D

This seems to only affect windows. It's unclear whether some HTTPS requests are hanging forever, or there's a bug that makes records wait in the queue without being resolved.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1874464

Comment 2

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

It seems the DNS_LOOKUP_TIME is also spiking since we landed this:
https://glam.telemetry.mozilla.org/firefox/probe/dns_native_lookup_time/explore?currentPage=1&os=Windows&process=parent&timeHorizon=QUARTER&visiblePercentiles=%5B99.9%2C99%2C95%5D
From a maximum of 11-13s to over 60s.

That indicates an issue with DNSQuery_A not respecting the regular timeouts HTTPS queries.

(edit) This was wrong. Since queuing time is reported before we do the call to DNSQuery_A, the increase to the lookup_time may be due to requests hanging in the queue.
We need a different probe that only records the time we took calling the platform APIs.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1874464

Comment 4

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Attachment: Bug 1885529 - Measure time spent in native HTTPS call r=#necko

Comment 5

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Attachment: 1885529-data-review.md

Comment 6

[Chris H-C :chutten]

Comment on attachment 9394438 [details]
1885529-data-review.md

(I'll let Cameron take this one)

Comment 7

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Attachment: Bug 1885529 - Also update nsHostResolver::mActiveAnyThreadCount in CompleteLookupByTypeLocked when resolving via native DNS r=#necko

The issue here is only obvious on Windows, when network.dns.get-ttl=true
and network.dns.native_https_query=true.

GetTTL will cause another resolution to happen with a low priority,
but these don't get picked up unless
mActiveAnyThreadCount < MaxResolverThreadsAnyPriority()

This causes the DNS_NATIVE_QUEUING probe to increase since these
queries don't get picked up immediately.

Also decrementing mActiveAnyThreadCount for ByType records makes
sure it always has the correct value.

Comment 8

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Attachment: Bug 1885529 - Make sure HostRecordQueue::mPendingCount is accurate r=#necko

When calling MoveToAnotherPendig the record gets removed from the
pending queue, and InsertRecord is called. As InsertRecord again
increments mPendingCount, we need to decrement it when removing from
a pending queue to ensure it's accurate.

Depends on D206498

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b8d47203a6b
Also update nsHostResolver::mActiveAnyThreadCount in CompleteLookupByTypeLocked when resolving via native DNS r=necko-reviewers,jesup
https://hg.mozilla.org/integration/autoland/rev/e633b6ff34e8
Make sure HostRecordQueue::mPendingCount is accurate r=necko-reviewers,jesup

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/2b8d47203a6b
https://hg.mozilla.org/mozilla-central/rev/e633b6ff34e8

Comment 11

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

glam is broken, but I ran this in SQL https://sql.telemetry.mozilla.org/queries/98978/source#244273 and it shows us back to normal.

Comment 12

[[e-mail]]

Comment on attachment 9394438 [details]
1885529-data-review.md

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes.
​
Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
If the request is for permanent data collection, is there someone who will monitor the data over time?
​
Yes, [e-mail] and [e-mail] are responsible.
​
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 1, Technical.
​
Is the data collection request for default-on or default-off?
​
Default on for all channels.
​
Does the instrumentation include the addition of any new identifiers?
​
No.
​
Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
Does the data collection use a third-party collection tool?
​
No.
​

Result: datareview+

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ce4fc8cc12d
Measure time spent in native HTTPS call r=necko-reviewers,jesup

Comment 14

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/0ce4fc8cc12d