Bugzilla

Here's yet another revision of my dtrace script. I've now found the proximate cause of my own crashes, at least. It seems to be an Apple bug (in the AMDRadeonX4000 kernel extension's AMDRadeonX4000_AMDHWVMContext::mapVA() method). Eric and Sorin, I'm very interested to see the results of your tests with this script.

steve, since you have GeForce video hardware, you won't be able to use this script as is (or any of its previous versions). You'll need to comment out all the probes that contain "AMDRadeonX4000". I'd still very much like to see your results, though.

As best I can tell, Apple's bug is very simple. Here's reconstructed C++ code for the first few lines of AMDRadeonX4000_AMDHWVMContext::mapVA():

        bool AMDRadeonX4000_AMDHWVMContext::mapVA(uintptr_t startAddress, IOAccelMemory* arg2,
                                                  uintptr_t arg3, uintptr_t length,
                                                  AMDRadeonX4000_IAMDHWVMM::VmMapFlags arg5)
        {
          if (startAddress < vmRangeStart) {
            return false;
          }
          if (startAddress + length >= vmRangeEnd) {
            return false;
          }
          ...
        }

The bug is that the fourth line of the function should be:

          if (startAddress + length > vmRangeEnd) {

Eric and Sorin, please test with this script. I now suspect that the GART stuff was a red herring, and that you will have exactly the same results as I did.

The AMDRadeonX4000 kernel extension uses the IOKit's IORangeAllocator class to manage allocations from the GPU's VRAM. The address range of 0x400000000 through 0x2400000000 is baked into its source code. The IORangeAllocator code (which is available at https://opensource.apple.com/ as part of the xnu kernel source distro) tries hard to manage this range efficiently, minimizing fragmentation. But as the range fills up, inevitably some allocations from it are made towards its end. The bug is triggered when an allocation is made flush up to the very end of range.

I will report this bug to Apple when I know a bit more about it. In the meantime, it may be possible to work around it by messing with Firefox code to make it invoke odd-sized allocations. That might increase fragmentation and slow performance, and would presumably need to be hidden behind a preference. I'll keep digging and see what I can come up with.

Here's yet another revision of my dtrace script. I've now found the proximate cause of my own crashes, at least. It seems to be an Apple bug (in the AMDRadeonX4000 kernel extension's AMDRadeonX4000_AMDHWVMContext::mapVA() method). Eric and Sorin, I'm very interested to see the results of your tests with this script.

steve, since you have GeForce video hardware, you won't be able to use this script as is (or any of its previous versions). You'll need to comment out all the probes that contain "AMDRadeonX4000". I'd still very much like to see your results, though.

As best I can tell, Apple's bug is very simple. Here's reconstructed C++ code for the first few lines of AMDRadeonX4000_AMDHWVMContext::mapVA():

        bool AMDRadeonX4000_AMDHWVMContext::mapVA(uintptr_t startAddress, IOAccelMemory* arg2,
                                                  uintptr_t arg3, uintptr_t length,
                                                  AMDRadeonX4000_IAMDHWVMM::VmMapFlags arg5)
        {
          if (startAddress < vmRangeStart) {
            return false;
          }
          if (startAddress + length >= vmRangeEnd) {
            return false;
          }
          ...
        }

The bug is that the fourth line of the function should be:

          if (startAddress + length > vmRangeEnd) {

Eric and Sorin, please test with this script. I now suspect that the GART stuff was a red herring, and that you will have exactly the same results as I did.

The AMDRadeonX4000 kernel extension uses the IOKit's IORangeAllocator class to manage allocations from the GPU's VRAM. The address range of 0x400000000 through 0x2400000000 is baked into its source code. The IORangeAllocator code (which is available at https://opensource.apple.com/ as part of the xnu kernel source distro) tries hard to manage this range efficiently, minimizing fragmentation. But as the range fills up, inevitably some allocations from it are made towards its end. The bug is triggered when an allocation is made flush up to the very end of the range.

I will report this bug to Apple when I know a bit more about it. In the meantime, it may be possible to work around it by messing with Firefox code to make it invoke odd-sized allocations. That might increase fragmentation and slow performance, and would presumably need to be hidden behind a preference. I'll keep digging and see what I can come up with.

Here's yet another revision of my dtrace script. I've now found the proximate cause of my own crashes, at least. It seems to be an Apple bug (in the AMDRadeonX4000 kernel extension's AMDRadeonX4000_AMDHWVMContext::mapVA() method). Eric and Sorin, I'm very interested to see the results of your tests with this script.

steve, since you have GeForce video hardware, you won't be able to use this script as is (or any of its previous versions). You'll need to comment out all the probes that contain "AMDRadeonX4000". I'd still very much like to see your results, though.

As best I can tell, Apple's bug is very simple. Here's reconstructed C++ code for the first few lines of AMDRadeonX4000_AMDHWVMContext::mapVA():

        bool AMDRadeonX4000_AMDHWVMContext::mapVA(uintptr_t startAddress, IOAccelMemory* arg2,
                                                  uintptr_t arg3, uintptr_t length,
                                                  AMDRadeonX4000_IAMDHWVMM::VmMapFlags arg5)
        {
          if (startAddress < vmRangeStart) {
            return false;
          }
          if (startAddress + length >= vmRangeEnd) {
            return false;
          }
          ...
        }

The bug is that the fourth line of the function should be:

          if (startAddress + length > vmRangeEnd) {

Eric and Sorin, please test with this script. I now suspect that the GART stuff was a red herring, and that you will have exactly the same results as I did.

The AMDRadeonX4000 kernel extension uses the IOKit's IORangeAllocator class to manage allocations from the GPU's VRAM. The address range of 0x400000000 through 0x2400000000 is baked into its machine code. The IORangeAllocator code (which is available at https://opensource.apple.com/ as part of the xnu kernel source distro) tries hard to manage this range efficiently, minimizing fragmentation. But as the range fills up, inevitably some allocations from it are made towards its end. The bug is triggered when an allocation is made flush up to the very end of the range.

I will report this bug to Apple when I know a bit more about it. In the meantime, it may be possible to work around it by messing with Firefox code to make it invoke odd-sized allocations. That might increase fragmentation and slow performance, and would presumably need to be hidden behind a preference. I'll keep digging and see what I can come up with.

Here's yet another revision of my dtrace script. I've now found the proximate cause of my own crashes, at least. It seems to be an Apple bug (in the AMDRadeonX4000 kernel extension's AMDRadeonX4000_AMDHWVMContext::mapVA() method). Eric and Sorin, I'm very interested to see the results of your tests with this script.

steve, since you have GeForce video hardware, you won't be able to use this script as is (or any of its previous versions). You'll need to comment out all the probes that contain "AMDRadeonX4000". I'd still very much like to see your results, though.

As best I can tell, Apple's bug is very simple. Here's reconstructed C++ code for the first few lines of AMDRadeonX4000_AMDHWVMContext::mapVA():

        bool AMDRadeonX4000_AMDHWVMContext::mapVA(vm_address_t startAddress, IOAccelMemory* arg2,
                                                  vm_size_t arg3, vm_size_t length,
                                                  AMDRadeonX4000_IAMDHWVMM::VmMapFlags arg5)
        {
          if (startAddress < vmRangeStart) {
            return false;
          }
          if (startAddress + length >= vmRangeEnd) {
            return false;
          }
          ...
        }

The bug is that the fourth line of the function should be:

          if (startAddress + length > vmRangeEnd) {

Eric and Sorin, please test with this script. I now suspect that the GART stuff was a red herring, and that you will have exactly the same results as I did.

The AMDRadeonX4000 kernel extension uses the IOKit's IORangeAllocator class to manage allocations from the GPU's VRAM. The address range of 0x400000000 through 0x2400000000 is baked into its machine code. The IORangeAllocator code (which is available at https://opensource.apple.com/ as part of the xnu kernel source distro) tries hard to manage this range efficiently, minimizing fragmentation. But as the range fills up, inevitably some allocations from it are made towards its end. The bug is triggered when an allocation is made flush up to the very end of the range.

I will report this bug to Apple when I know a bit more about it. In the meantime, it may be possible to work around it by messing with Firefox code to make it invoke odd-sized allocations. That might increase fragmentation and slow performance, and would presumably need to be hidden behind a preference. I'll keep digging and see what I can come up with.