I was also thinking that returning true for the out-of-process document is wrong, but after some more reading the code around its call site, now I think returning true looks correct.
There is only one call site of nsCoreUtils::IsRootDocument, which is in DocManager::CreateDocOrRootAccessible. If we consider a given document is not root, we try to get the parent accessible document, then we call BindChildDocument. Apparently it won't work in fission.
So, I think we should rename nsCoreUtils::IsRootDocument to nsCoreUtils::IsRootDocumentInProcess, and make the function return true in the case where the given document is the root document of the out-of-process document tree.