I was trying to build Thunderbird on Solaris, and I was determined to find out what xp_qsort.h was before commenting it out, and I couldn't find any other reference to it on Bugzilla. Since there was a mention of libxp in the source code comments above it, I thought it might be referring to a qsort function in the X11 print libraries. It wasn't, though. Here's xp_qsort.h:
And the references to it in various CVS source tarballs disappear sometime around 2001, where it seems to get replaced by nsQuickSort, which is still in the tree to this day.
Based on where this file is now, I think "libxp" might be what Netscape used to call XPCOM (or a close ancestor). You can tell it's loosely based on the same files because both xp_qsort.h and nsQuickSort.h contain this comment that seems to have been carried along for about 22 years:
/* We need this because Solaris' version of qsort is broken and
- causes array bounds reads.
The possible options I see here involve either putting a definition for XP_SORT into this file, rewriting this to use nsQuickSort, or else simply removing the patch and hoping that whatever issue existed back then has been fixed. Because right now, the only way people can get this to compile on Solaris is by removing this and just using qsort. People have probably just been ripping it out for a couple decades now.