The static build work added to /embedding/browser/gtk/tests/Makefile.in needs a little more work. OpenVMS fails to link because the NSPR libraries are brought in before some of the modules (layout) that reference them. On OpenVMS (and other platforms I believe) order is relevant. I moved MOZ_COMPONENT_LIBS "down" to the end of EXTRA_LIBS and now it works. xpfe/bootstrap/Makefile.in gets it right if you want a good example.
The link order issue may be fixed by the patch in bug 88038 since I factored out the static linking code to a separate makefile. Can you try it?
Fixed by static makefile reorg in bug 88038.