1450583 - Enabling DoH causes macOS kernel panic: assertion failed: inp->inp_flowhash != 0, file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.41.2/bsd/netinet/tcp_output.c

Status: RESOLVED FIXED

(Core :: Networking: DNS, defect, P2)

Description

[Chris Peterson [:cpeterson]]

STR:
1. Enable DoH in Nightly using the instructions in this gist:

https://gist.github.com/bagder/5e29101079e9ac78920ba2fc718aceec

network.trr.mode pref = 2
network.trr.uri pref = "https://mozilla.cloudflare-dns.com/dns-query" or "https://dns.google.com/experimental"

2. Restart Nightly.
3. Try loading a website.

RESULT:
macOS kernel panic!

panic(cpu 3 caller 0xffffff8009a0f8af): assertion failed: inp->inp_flowhash != 0, file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.41.2/bsd/netinet/tcp_output.c, line: 1860

This panic is 100% reproducible. I am using Firefox Nightly 61.0a1 (2018-04-01) on macOS High Sierra 10.13.3.

Comment 1

[Chris Peterson [:cpeterson]]

Here is the system report for the kernel panic:

Anonymous UUID:       F0FD1048-F4DC-DCE3-D924-5B2A46270593

Sun Apr  1 13:10:45 2018

*** Panic Report ***
panic(cpu 3 caller 0xffffff8009a0f8af): assertion failed: inp->inp_flowhash != 0, file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.41.2/bsd/netinet/tcp_output.c, line: 1860
Backtrace (CPU 3), Frame : Return Address
0xffffff91fa4db800 : 0xffffff800964f606 
0xffffff91fa4db850 : 0xffffff800977c654 
0xffffff91fa4db890 : 0xffffff800976e149 
0xffffff91fa4db910 : 0xffffff8009601120 
0xffffff91fa4db930 : 0xffffff800964f03c 
0xffffff91fa4dba60 : 0xffffff800964edbc 
0xffffff91fa4dbac0 : 0xffffff8009a0f8af 
0xffffff91fa4dbc60 : 0xffffff8009a1cbb4 
0xffffff91fa4dbcc0 : 0xffffff8009b722bc 
0xffffff91fa4dbde0 : 0xffffff8009b82b93 
0xffffff91fa4dbed0 : 0xffffff8009b82891 
0xffffff91fa4dbf40 : 0xffffff8009bfa978 
0xffffff91fa4dbfa0 : 0xffffff8009601906 

BSD process name corresponding to current thread: firefox

Mac OS version:
17D102

Kernel version:
Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
Kernel UUID: 18D901F1-4A03-3FF1-AE34-C26B2732F13C
Kernel slide:     0x0000000009200000
Kernel text base: 0xffffff8009400000
__HIB  text base: 0xffffff8009300000
System model name: MacBookPro10,1 (Mac-C3EC7CD22292981F)

System uptime in nanoseconds: 134511924132
last loaded kext at 28656789739: org.virtualbox.kext.VBoxNetAdp 5.2.0 (addr 0xffffff7f8cf05000, size 28672)
loaded kexts:
org.virtualbox.kext.VBoxNetAdp  5.2.0
org.virtualbox.kext.VBoxNetFlt  5.2.0
org.virtualbox.kext.VBoxUSB 5.2.0
org.virtualbox.kext.VBoxDrv 5.2.0
com.apple.driver.AudioAUUC  1.70
com.apple.driver.AppleHWSensor  1.9.5d0
com.apple.driver.AGPM   110.23.30
com.apple.driver.ApplePlatformEnabler   2.7.0d0
com.apple.driver.X86PlatformShim    1.0.0
com.apple.filesystems.autofs    3.0
com.apple.driver.AppleMikeyHIDDriver    131
com.apple.driver.AppleHDAHardwareConfigDriver   280.12
com.apple.driver.AppleHDA   280.12
com.apple.driver.AppleMikeyDriver   280.12
com.apple.driver.AppleGraphicsDevicePolicy  3.16.21
com.apple.AGDCPluginDisplayMetrics  3.16.2
com.apple.driver.AppleHV    1
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.AppleUpstreamUserClient    3.6.4
com.apple.iokit.IOBluetoothSerialManager    6.0.2f2
com.apple.driver.AGDCBacklightControl   3.16.2
com.apple.driver.pmtelemetry    1
com.apple.GeForce   10.2.8
com.apple.driver.AppleBacklight 170.10.2
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.driver.AppleIntelHD4000Graphics   10.3.0
com.apple.driver.ACPI_SMC_PlatformPlugin    1.0.0
com.apple.driver.AppleSMCPDRC   1.0.0
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.eficheck   1
com.apple.driver.AppleMuxControl    3.16.21
com.apple.driver.AppleMCCSControl   1.5.3
com.apple.driver.AppleLPC   3.1
com.apple.driver.AppleIntelFramebufferCapri 10.3.0
com.apple.nvidia.NVDAStartup    10.2.8
com.apple.driver.AppleOSXWatchdog   1
com.apple.driver.AppleSMCLMU    211
com.apple.driver.AppleFIVRDriver    4.1.0
com.apple.driver.AppleThunderboltIP 3.1.1
com.apple.iokit.IOBluetoothUSBDFU   6.0.2f2
com.apple.driver.AppleUSBTCKeyEventDriver   254
com.apple.driver.AppleUSBTCKeyboard 254
com.apple.driver.AppleUSBTCButtons  254
com.apple.filesystems.apfs  748.41.3
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.filesystems.hfs.kext  407.30.1
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.driver.AirPort.Brcm4331   800.21.30
com.apple.driver.AppleSDXC  1.7.6
com.apple.iokit.AppleBCM5701Ethernet    10.3.1
com.apple.driver.AirPort.Brcm4360   1220.18.1a2
com.apple.driver.AppleAHCIPort  329
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleACPIButtons   6.1
com.apple.driver.AppleRTC   2.0
com.apple.driver.AppleHPET  1.8
com.apple.driver.AppleSMBIOS    2.1
com.apple.driver.AppleACPIEC    6.1
com.apple.driver.AppleAPIC  1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 220.0.0
com.apple.nke.applicationfirewall   183
com.apple.security.TMSafetyNet  8
com.apple.security.quarantine   3
com.apple.IOBufferCopyEngineTest    1
com.apple.driver.AppleIntelCPUPowerManagement   220.0.0
com.apple.iokit.IOUSBUserClient 900.4.1
com.apple.kext.triggers 1.0
com.apple.driver.DspFuncLib 280.12
com.apple.kext.OSvKernDSPLib    526
com.apple.driver.AppleSSE   1.0
com.apple.iokit.IOSerialFamily  11
com.apple.nvidia.driver.NVDAGK100Hal    10.2.8
com.apple.nvidia.driver.NVDAResman  10.2.8
com.apple.driver.AppleBacklightExpert   1.1.0
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.AppleGPUWrangler  3.16.2
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IONDRVSupport   517.22
com.apple.driver.AppleHDAController 280.12
com.apple.iokit.IOHDAFamily 280.12
com.apple.iokit.IOAudioFamily   206.5
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleGraphicsControl   3.16.21
com.apple.driver.AppleSMBusController   1.0.18d1
com.apple.driver.AppleSMBusPCI  1.0.14d1
com.apple.iokit.IOAcceleratorFamily2    376.6
com.apple.iokit.IOSurface   209.2.2
com.apple.AppleGraphicsDeviceControl    3.16.21
com.apple.iokit.IOGraphicsFamily    517.22
com.apple.driver.X86PlatformPlugin  1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.driver.AppleThunderboltEDMSink    4.1.2
com.apple.driver.AppleThunderboltDPOutAdapter   5.0.2
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.2f2
com.apple.iokit.IOBluetoothHostControllerUSBTransport   6.0.2f2
com.apple.iokit.IOBluetoothHostControllerTransport  6.0.2f2
com.apple.iokit.IOBluetoothFamily   6.0.2f2
com.apple.driver.AppleUSBMultitouch 261
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.usb.cdc    5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleThunderboltDPInAdapter    5.0.2
com.apple.driver.AppleThunderboltDPAdapterFamily    5.0.2
com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.3
com.apple.driver.AppleXsanScheme    3
com.apple.iokit.IOAHCIBlockStorage  301.40.2
com.apple.driver.AppleThunderboltNHI    4.5.6
com.apple.iokit.IOThunderboltFamily 6.6.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IO80211Family   1200.12.2
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleUSBMergeNub   900.4.1
com.apple.iokit.IOAHCIFamily    288
com.apple.driver.usb.AppleUSBEHCIPCI    1.2
com.apple.driver.usb.AppleUSBEHCI   1.2
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.iokit.IOUSBFamily 900.4.1
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.AppleEFINVRAM  2.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.driver.AppleEFIRuntime    2.1
com.apple.iokit.IOSMBusFamily   1.1
com.apple.security.sandbox  300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.iokit.IOBufferCopyEngineFamily    1
com.apple.driver.DiskImages 480.30.2
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore  2
com.apple.driver.AppleUSBTDM    439.30.4
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.iokit.IOUSBMassStorageDriver  140.30.1
com.apple.iokit.IOSCSIBlockCommandsDevice   404.30.2
com.apple.iokit.IOSCSIArchitectureModelFamily   404.30.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost    1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController    1.0
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.IOSlaveProcessor   1
com.apple.iokit.IOReportFamily  31
com.apple.iokit.IOTimeSyncFamily    650.5
com.apple.iokit.IONetworkingFamily  3.4
com.apple.driver.AppleACPIPlatform  6.1
com.apple.driver.AppleSMCRTC    1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily    1.4
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.corecrypto    1.0

EOF

Comment 2

[Patrick McManus [:mcmanus]]

this sounds an awful lot like https://bugzilla.mozilla.org/show_bug.cgi?id=1439231

which is a mac TCP fast open TFO bug that's triggered by DoH. (I think it has something to do with ipv6 error cases - but at its core its a TFO bug).

It should have been resolved in nightly - so you can you give the buildconfig information of your nightly? (the nightly fix would just silently disable TFO < 10.13.4)

you can also test that by setting network.tcp.tcp_fastopen_enable to false.

thanks.

Comment 3

[Chris Peterson [:cpeterson]]

Yes, I think this bug is a dupe of TFO bug 1439231. The panic assertion is the same. Unfortunately, since filing this bug, my Mac has updated from macOS 10.13.3 to 10.13.4, so I can't confirm that setting network.tcp.tcp_fastopen_enable to false avoids the panic on 10.13.3. I can confirm that 10.13.4 does not panic with DoH.

Patrick: if this is a dupe, shouldn't bug 1444453 ("Require MacOS 10.13.4 for TFO") have prevented me from hitting this TFO panic when using DoH? 1444453 landed 23 days ago and I was running this morning's Firefox Nightly (buildID 20180401100341) on macOS 10.13.3.

Comment 4

[Patrick McManus [:mcmanus]]

Yes, a Apr-01 nightly should have worked around that on 10.13.3 and I had thought bagder verified the fix. is 10.13.4 still in beta (I'm not a macos user myself..)?

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Patrick McManus [:mcmanus] from comment #4)
> Yes, a Apr-01 nightly should have worked around that on 10.13.3 and I had
> thought bagder verified the fix. is 10.13.4 still in beta (I'm not a macos
> user myself..)?

macOS 10.13.4 was released on March 29. I received 10.13.4 through the normal macOS App Store updates.

https://support.apple.com/en-us/HT208533

Comment 6

[Daniel Stenberg [:bagder]]

I'm just so puzzled. It doesn't trigger a kernel-panic for me anymore.

I still run 10.13.3 here and I figured I should keep doing this while this bug remains so that I can try out the code.

When I verified mcmanus' fix for bug 1439231, I built my own version and tested so I figured maybe I screwed something up then so now I updated my "regular" nightly to the latest 61 and took it for a spin. Enabled TRR (mode 2), made sure that the TFO option is not modified. I can now use TRR fine (verified with about:networking).

But TFO is enabled!

I also updated my own build just now, ran with "MOZ_LOG=sync,nsHttp:5" and checked:

 [833:Main Thread]: D/nsHttp nsHttpHandler::SetFastOpenOSSupport version 17.4.0
 [833:Main Thread]: D/nsHttp nsHttpHandler::SetFastOpenOSSupport  supported.

I can't explain why TRR + TFO doesn't crash for me anymore.

It turns out this check[1] uses PR_GetSystemInfo() which returns the version number that uname shows, which is "17.4.0" for 10.13.3 and is "17.5.0" for 10.13.4 (see https://en.wikipedia.org/wiki/MacOS_High_Sierra#Releases)

Still, I'll patch the version check to do right as with my local patch it now logs:

 [2986:Main Thread]: D/nsHttp nsHttpHandler::SetFastOpenOSSupport version 17.4.0
 [2986:Main Thread]: D/nsHttp nsHttpHandler::SetFastOpenOSSupport not supported.

[1] = https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsHttpHandler.cpp#350

Comment 7

[Daniel Stenberg [:bagder]]

Attachment: bug 1450583 - require macOS 10.13.4 (uname 17.5.0) for enabling TFO

Review commit: https://reviewboard.mozilla.org/r/233266/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/233266/

Comment 8

[Patrick McManus [:mcmanus]]

Comment on attachment 8964539 [details]
bug 1450583 - require macOS 10.13.4 (uname 17.5.0) for enabling TFO

https://reviewboard.mozilla.org/r/233266/#review238818

move this to beta too.. not so much for trr, but just for tfo reasons

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a2ba99f199a
require macOS 10.13.4 (uname 17.5.0) for enabling TFO r=mcmanus

Comment 10

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/8a2ba99f199a

Comment 11

[Daniel Stenberg [:bagder]]

Comment on attachment 8964539 [details]
bug 1450583 - require macOS 10.13.4 (uname 17.5.0) for enabling TFO

Approval Request Comment
[Feature/Bug causing the regression]: 1450583
[User impact if declined]: kernel panic and reboot on macOS <= 10.13.3 (most easily triggered when TRR is enabled) or "mere" TCP problems
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: somewhat
[Needs manual test from QE? If yes, steps to reproduce]: 
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: the change is minimal
[String changes made/needed]:

Comment 12

[Julien Cristau [:jcristau]]

Comment on attachment 8964539 [details]
bug 1450583 - require macOS 10.13.4 (uname 17.5.0) for enabling TFO

bump min macos version for tfo, approved for 60.0b10

Comment 13

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8a0377c0d860