My suggestion? If a system has no swap, you check for some cutoff of free RAM (and possibly cache?) and start discarding tabs as you approach it.
If swap is on, I would be curious to see the result of measuring pageout/pagein rates of the firefox process to indicate when to discard tabs. ps has a flag for showing major page faults (paging to disk) per process so presumably there is a POSIX call for this information.
My thought is, on something like a 4GB system if you have firfox plus some other stuff going on you can easily have 600-800MB of "junk" get swapped out (least recently used after all... I suppose it's the loaded but inactive daemons, various chunks of gnome, etc..) resulting in plenty of RAM for the working set. So measuring total system page rate will not provide useful info for when to discard tabs. Your two scenarios where you'd get firefox pageout; 1) You start some other large application, so it would begin to page firefox. Cuttings tabs then would presumably increase performance of both firefox and the other application. 2) firefox RAM usage has crept up enough to start paging, an appropriate time to cut RAM usage.