Consider the following scenario:
the buffered range is [14586, 14614]
currentTime is 14588
video decoding is slightly behind, say 14587, the next key frame is in the future at 14589
Now remove is called on the source buffer, making the new buffered range at [14588, 14614].
When we attempt to decode the next video frame, it's no longer in the source buffer. While the video decoding is late, the next keyframe isn't in the past, and as such the skip_to_next_keyframe logic isn't triggered.
So we stall.
There's no way for the JS player to understand or detect that particular stall. As far as it's concerned, the buffered range contains currentTime, currentState is HAVE_FUTURE_DATA
One possibility would be that if we find ourselves in the situation that a frame in the past can't be decoded, but we have a keyframe in the future, yet behind currentTime then we jump to that frame. It will break A/V sync slightly, but at this stage, A/V sync is already broken and the video appears as a slide show anyway.