1987786 - Crash in [@ res_nsearch_2], dns_res_send, libresolv.9.dylib during exchange address account setup (macOS)

Status: RESOLVED FIXED

(Thunderbird :: Account Manager, defect, P1)

Description

[Wayne Mery (:wsmwk)]

(continued from Bug 1441694 - startup Crash in res_nsearch_2 or dns_res_send if chat account logs in at startup - which is unrelated)

Crash report: https://crash-stats.mozilla.org/report/index/a2122675-02d7-4320-9ae7-1f8b80250820

User comment "first time running, just added an O365 exchange account, clicked continue and it crashed". And User comments in the crashes almost all cite creating a new account. So, account hub?

Reason:

EXC_BAD_ACCESS / KERN_INVALID_ADDRESS

Top 10 frames:

0  libresolv.9.dylib  res_nsearch_2
1  libresolv.9.dylib  res_9_nsearch
2  XUL  ffi_call_SYSV
3  XUL  ffi_call_int  js/src/ctypes/libffi/src/aarch64/ffi.c:748
4  XUL  js::ctypes::FunctionType::Call(JSContext*, unsigned int, JS::Value*)  js/src/ctypes/CTypes.cpp:7144
5  XUL  CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::...  js/src/vm/Interpreter.cpp:501
5  XUL  js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstru...  js/src/vm/Interpreter.cpp:583
6  XUL  InternalCall(JSContext*, js::AnyInvokeArgs const&, js::CallReason)  js/src/vm/Interpreter.cpp:664
6  XUL  js::CallFromStack(JSContext*, JS::CallArgs const&, js::CallReason)  js/src/vm/Interpreter.cpp:669
6  XUL  js::Interpret(JSContext*, js::RunState&)  js/src/vm/Interpreter.cpp:3287

If we sort https://crash-stats.mozilla.org/topcrashers/?product=Thunderbird&version=142.0 by macOS, we see several crash signatures at the top, and if you dig into those they also start in version 142:

• res_nsearch_2
• dns_res_send
• libresolv.9.dylib
• sock_eq

Another approach is query on macOS and version 142

I found no crashes for other OS on version 142 (or any other version) that cite creating an account

Comment 1

[Magnus Melin [:mkmelin]]

Seems very likely bug 1976254 caused this (using the DNS api more, or possibly differently than we use it before).
I'm going to mark it as such, tentatively.

Comment 2

[Wayne Mery (:wsmwk)]

The crash rate for most of the signatures is quite steady over the past couple months.
I have not found a Windows crash signature that hits at the scale of these macOS crashes.

Can we expect this to be mitigated by Bug 1975860 fixed in 145 - Cannot add new imap account - account hub locks up - for autodiscovery pointing to account hosted at 3rd party (i.e. large German providers like IONOS,@online.de) ? But if the patches won't be uplifted, confirmation is mostly likely to happen in beta via crash signature dns_res_send which has decent beta crash rate.

Comment 3

[Wayne Mery (:wsmwk)]

Toby, this is likely critical to exchange support. Do you want this to block the tb-ms-exchange bug?

Most crashes are version 143

Comment 4

[Eleanor Dicharry]

I think I finally managed to duplicate this one. When setting up a new account, after going through the oauth flow, it's looking up the server autoconfig info. At that point, TB hangs, and when I quit DNS.worker.mjs is still running, and then it crashes. I haven't been able to verify that this is the same crash, but it seems plausible given the DNS worker, which definitely calls into the resolv library.

It's still intermittant, but at least this is a clue.

Comment 5

[Eleanor Dicharry]

OK, my running hypothesis is that we're not aborting the SRV lookup appropriately when something fails (or when the user aborts). I don't get a crash all the time, but sometimes if I kick off the discovery process and abort, I get the lingering DNS worker. I'm guessing that maybe on MacOS, having that zombie worker is messing up some state in libresolv? I've also noticed that in some cases, if I abort and then wait, autodiscover comes back anyway some indeterminate time later.

Comment 6

[Eleanor Dicharry]

Attachment: Bug 1987786 - Handle errors in libresolv native DNS lookup functions. r=#thunderbird-back-end-reviewers

Comment 7

[Eleanor Dicharry]

I'm marking a potential fix for this for checkin, but since I can't reproduce it, the only way to know if it worked is if the crash frequency changes, so I'm leaving this open while we see what happens.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/583bf659f5d4
Handle errors in libresolv native DNS lookup functions. r=mkmelin

Comment 9

[Eleanor Dicharry]

Comment on attachment 9520594 [details]
Bug 1987786 - Handle errors in libresolv native DNS lookup functions. r=#thunderbird-back-end-reviewers

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: This may fix one of our current top crashers.
• User impact if declined: No attempt will have been made to fix one of our top crashers.
• Is this code covered by automated tests?: No
• Has the fix been verified in Daily?: No
• Has the fix been verified in Beta?: No
• Needs manual test from QA?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's really just handling an error condition that should have been handled to begin with.
• Does the fix cause any migrations to be skipped?: Yes
• String changes made/needed:

Comment 10

[Corey Bryant]

Comment on attachment 9520594 [details]
Bug 1987786 - Handle errors in libresolv native DNS lookup functions. r=#thunderbird-back-end-reviewers

[Triage Comment]
Approved for beta

Comment 11

[Daniel Darnell [:dandarnell]]

Thunderbird 145.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/a67fec958606

Comment 12

[Eleanor Dicharry]

Attachment: Bug 1987786 - Perform bounds checking on libresolv functions. r=#thunderbird-back-end-reviewers

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/3e43c48b3424
Perform bounds checking on libresolv functions. r=mkmelin

Comment 14

[Wayne Mery (:wsmwk)]

FWIW, possibly a slight decrease in 145 beta crash rate - hard to say with certainty. Good to see another patch for 146.

Comment 15

[Eleanor Dicharry]

Once 145 releases, would we want to try and do an uplift of this one?

Comment 16

[Wayne Mery (:wsmwk)]

After burn in with both some positive impact and no ill effects. Another case of, wait and see.

Comment 17

[Wayne Mery (:wsmwk)]

Crashes continue for res_nsearch_2 for daily, beta (145.0b4 and onwards), and crash rate for release is unchanged after shipping 145.0.

bp-8adadac6-c77c-45d3-b328-1d8510251115 145.0
bp-662de77f-22d7-44dc-b00f-e9bb90251114 146.0b1

Comment 18

[Eleanor Dicharry]

Honestly, at this point, I think we should probably replace the javascript wrapper for libresolv, either with the DNS XPCOM class in m-c or, if that doesn't support the SRV lookups we need, our own native DNS implementation. I'll look at that this week.

Comment 19

[Patrick Cloke [:clokep]]

(In reply to Eleanor Dicharry from comment #18)

Honestly, at this point, I think we should probably replace the javascript wrapper for libresolv, either with the DNS XPCOM class in m-c or, if that doesn't support the SRV lookups we need, our own native DNS implementation. I'll look at that this week.

It doesn't support SRV lookups, which is why this code exists. :)

I think supporting it in necko would be the way to go instead of having a separate implementation? See bug 735967. It would also probably fix other things that the JavaScript code might bypass certain DNS/proxy configurations.

Comment 20

[Eleanor Dicharry]

Thanks for the pointer at the older issues. I asked the Necko team in chat if they'd be open to a contribution from us to support SRV records.

Comment 21

[Kai Engert [:KaiE:]]

According to the call stack, we call symbol res_9_nsearch, but I cannot find that name anywhere in our code.
How is this possible?

Comment 22

[Eleanor Dicharry]

(In reply to Kai Engert [:KaiE:] from comment #21)

According to the call stack, we call symbol res_9_nsearch, but I cannot find that name anywhere in our code.
How is this possible?

We call res_search, which is #defined on macos resolv.h to res_9_search. I was assuming that res_9_nsearch was just something that gets called below that in the libresolv internals, but I'm checking to verify.

Comment 23

[Kai Engert [:KaiE:]]

Our source code says we might call the "search" version of the API.
But the call stack says we're calling the "nsearch" version of the API.

"nsearch" has an additional parameter, a resolver state variable.

I could understand if the "nsearch" API would be called indirectly by something else, but according to the stack, it's our XUL library that directly calls into "nsearch".

I wonder if that means the following. We try to lookup the "search" version of the API in libresolv, but for some reason, we are given the address of the "nsearch" version. Then we call the function with an invalid parameter list, and crash?

Comment 24

[Kai Engert [:KaiE:]]

The user from the crash report (comment 0) was running macOS 14.3 in 2025-09.
But macOS 14.4 was already released in 2024-03.

Did we already ask all users who are affected to try to update their macOS, to ensure we aren't dealing with an OS level bug?

Comment 25

[Eleanor Dicharry]

In https://github.com/apple-opensource/libresolv/res_data.c res_search directly calls res_nsearch. But you're right, the compiler can't optimize directly to nsearch because it's a dynamic lib....That could be the case, but I don't know why that would happen.

Comment 26

[Eleanor Dicharry]

(In reply to Kai Engert [:KaiE:] from comment #24)

The user from the crash report (comment 0) was running macOS 14.3 in 2025-09.
But macOS 14.4 was already released in 2024-03.

Did we already ask all users who are affected to try to update their macOS, to ensure we aren't dealing with an OS level bug?

We have been seeing crashes across many macos versions for this signature. The percentage is higher for lower versions, it's true, but we've seen crashes in 15 as well. The one time I was able to get a crash that looked the same was in 15.6, just before I was upgraded to 26 and if you look through the 143 and 144 crash reports there are crashes in higher versions. We have applied two fixes for real issues, so maybe we're down to lower, but in general the bug is happening on multiple OS versions.

Comment 27

[Eleanor Dicharry]

If it is the case that we're getting the wrong symbol from libresolv, then that's more reason to move to Necko's DNS implementation since it means ctypes is giving us the wrong function when we try to query it by name from the library symbols.

Comment 28

[Kai Engert [:KaiE:]]

Maybe the symbols we're trying to lookup in the library are older aliases, and they are defined to map to the wrong symbol.

Now that we found this potential cause, maybe we could try an easy fix and switch to a more modern function.

I see that res_search is deprecated, and the newer replacement might be the series:

• res_ninit
• res_nsearch
• res_nclose

Comment 29

[Kai Engert [:KaiE:]]

I see that res_ninit requires that the caller passes a buffer, rather, a pointer to a struct.
That struct size is platform specific.
However, it should be known and fixed at compile time.

I believe that our JS code doesn't need to understand/access the contents of that struct, it can treat it just as a buffer.

Idea for the hack:
In our C code, we implement a sizeof(res_state) to obtain the size of that buffer, and make that constant queryable for our JS code. If we don't have a simpler idea, we could use an nsI* service, and a function returns the constant, that way our JS code can get it.

The JS code allocates a memory buffer of that size. We define the res_state parameter in the ctypes definition as a void pointer. We pass the pointer to the allocated buffer to ctypes. The functions use that buffer to store their state in our buffer.

Comment 30

[Eleanor Dicharry]

We can try that, but the res_n* functions are not documented as being part of the public API in the resolver manpage on macos or on FreeBSD. See https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/resolver.3.html and https://man.freebsd.org/cgi/man.cgi?query=resolver&sektion=3&apropos=0&manpath=FreeBSD+14.3-RELEASE+and+Ports and I confirmed this on my mac as well. The n* functions are in the resolv.h header, but it looks like the expectation on macos is that the res_* functions are called. I know the GNU documentation says the non n* functions are deprecated, but I'm not sure if that's the case for the BSDs and their derivatives including macos.

Comment 31

[Eleanor Dicharry]

Attachment: WIP: Bug 1987786 - Use Mac-native DNS lookups for SRV records.

Comment 32

[Eleanor Dicharry]

Attachment: Bug 1987786 - Use a single worker to avoid multiple threads making concurrent DNS requests. r=#thunderbird-reviewers

Comment 33

[Phabricator Automation]

Comment on attachment 9528414 [details]
WIP: Bug 1987786 - Use Mac-native DNS lookups for SRV records.

Revision D273608 was moved to bug 2003284. Setting attachment 9528414 [details] to obsolete.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/394c6c5104be
Use a single worker to avoid multiple threads making concurrent DNS requests. r=mkmelin

Comment 35

[Wayne Mery (:wsmwk)]

Patch apparently helped https://support.mozilla.org/en-US/questions/1541729