We should probably rename it to GetTransformToProperAncestor and assert that aAncestor is a proper ancestor of aFrame, if all call sites are OK with that. If we do that, TransformAncestorRectToFrame and TransformFrameRectToAncestor should all get Proper in their names as well (and their callers have to be checked).
FrameLayerBuilder::GetThebesLayerScaleForFrame in FrameLayerBuilder calls nsLayoutUtils::GetTransformToAncestor (via PredictScaleForContent) and can in some cases pass an ancestor that's equal to the given frame. For example, when GetThebesLayerScaleForFrame is called on a frame with a border and a CSS transform, this will happen because the frame will have both a ContainerLayer for the transform and a ThebesLayer child of that ContainerLayer associated with it.
Some of the callers to TransformFrameRectToAncestor such as the ones in nsObjectFrame::ComputeVisibility, could also have aFrame == aAncestor. We could introduce TransformFrameRectToProperAncestor/TransformProperAncestorRectToFrame for some other callers but it doesn't seem worth it to me.