My understanding of Firefox's implementation of Wayland is that the toplevel surface is created by Gtk3's window code - Gtk3 draws the window shadows, borders, and background - then Firefox itself renders in a subsurface which is positioned and sized to appear only within the "window area" (excluding shadows) - and for fractional scaling, wl_viewport is only used on the subsurface, not the toplevel?
If so, the cause of this issue is almost certainly due to rounding errors in position and size caused by the differences between the toplevel surface (presumably rendered by Gtk3 at nearest integer buffer scale then upscaled or downscaled to the fractional scale by the compositor) and the subsurface (intended to be rendered directly at fractional scale). The fractional-scale-v1 protocol is only designed to be used for toplevel surfaces: "For toplevel surfaces, the size is rounded halfway away from zero. The rounding algorithm for subsurface position and size is not defined." https://wayland.app/protocols/fractional-scale-v1
I think to fix this fully and completely, Firefox probably would have to handle its own toplevel surfaces rather than going through Gtk (which would also mean that Gtk can't be used for drawing shadows, window borders, etc - and it probably also means that Gtk can't be used for input handling). Some of the later comments on https://bugzilla.mozilla.org/show_bug.cgi?id=1701123 are relevant.
A possible "hack" to avoid the rounding problem might be to have Firefox's subsurface cover the exact same area as Gtk's toplevel surface - i.e. have the subsurface also cover the area where the shadow is rendered. But even if this work, it depends on non-specified behaviour. Also, Firefox's own drawing would have to be padded within the subsurface to only be within the "window area", which brings up new and different rounding error problems (the edges of the window area on Gtk's toplevel surface might not line up with physical pixels).
I should note that on my screen (3840x2160 at 175% in GNOME 44), this issue is also affecting maximized windows. Gtk still draws a border (actually a thin shadow) around the edges of maximized windows, so the rounding error issue happens there too. Fullscreen windows also seem to be affected on my system but I don't have an explanation for why that is happening.