I think currently docgroup is keyed on eTLD+1. This should really be scheme + eTLD+1 (i.e., https://html.spec.whatwg.org/#site).
If websites opt-in, we are planning on changing the keying from site to origin. See bug 1601594. (I suspect it would be best for any fix here to build on that, but ultimately defer to Tom Tung on that.)
I also believe that currently, FPI does not affect docgroup keying. Since the goal of FPI is that when A embeds B1 and B1 popups B2, B1 and B2 are as isolated from each other as A and B1 are (assuming that the Bs are same-site and A is cross-site from the Bs), B1 and B2 ought to be in different docgroups (i.e., agents).
The observable effect of this would be that B1 and B2 can have shared memory (that uses docgroup as a key), despite FPI trying to forbid that in spirit.
(This takes us further from suggestions in bug 1321158, but I think that ought to be INVALID/WONTFIX.)