FWIW, https://html.spec.whatwg.org/#the-list-of-available-images does not consider the referrer policy as a cache key and I don't think it should.
Bug 1174921 Comment 17 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
FWIW, https://html.spec.whatwg.org/#the-list-of-available-images does not consider the referrer policy as a cache key and I don't think it should. (Edit: it's being discussed again in https://github.com/whatwg/html/issues/5541 though.)