For sites like ship-it.allizom.org there's only an internal IP, but allizom.org isn't in the list of search domains published by the Mozilla VPN, so I can't resolve it. FWIW, I have the 'Apply DNS settings simultaneously' option set in Viscosity, which leaves my ISPs resolvers in place, and adds our resolvers for the search domains declared by the VPN. If that pref is off the VPN resolvers take over all queries, adding a lot of latency and breaking some CDN content, since I'm not in N. America.
I'm not sure it's safe to publish allizom.org (which is exclusively for testing/staging sites, and not a permanent location for any production sites) in the DNS search domains for the VPN. This could lead to people unexpectedly making changes in testing/staging while expecting that they're using production, and the majority of users will want xyz.mozilla.org, not xyz.allizom.org.
Hmm, I think this could indeed have some unwanted effects. For instance the problem you are trying to solve here is that the resolver in use will selectively use different nameservers for different domains because of the search domains published by the VPN, however, adding things to this search path will *also* make the local resolver attempt to append the domain to unqualified hostnames and depending on order, could result in someone attempting to reach production ship-it.mozilla.org, but instead get taken to ship-it.allizom.org or vice versa. Also, there is a limit on how many domains we can add to the search list. I feel like in this case, since allizom.org is normally a public-facing zone, perhaps we should add the DNS entries for it to the public zone, rather than having the split horizon on the zone. This would result in the host always being resolvable by anyone, but still only reachable if on the VPN. I'll need to gather some further input on which is a more correct route for this issue.
I missed the part about you using Viscosity. I don't think we are using that setting and don't have the issue. Is there a reason to have the "Apply DNS settings simultaneously" vs. just letting the vpn do all the resolution?
I see your point about unexpectedly getting allizom.org sites when a FQDN isn't used. My reasons for using the simultaneous DNS are in comment #0, but to expand the main problem is that without it CDNs resolve to nodes in California (the ones close to our nameservers). The request doesn't go over the VPN though, and sometimes those nodes block connections from 'distant' IPs. The web pages then 'hang' loading images/css/other static files put on the CDN. Akamai does this for example. IIRC the new VPN doesn't allow routing all traffic through it, and I'm not sure I want to anyway. But this might all be moot. I typically use FQDN anyway, and it turns out I can manually add allizom.org to the list of search domains , so I can fix resolution of ship-it.allizom.org myself. RESOLVED WONTFIX ?  Viscosity Icon > Preferences > MozillaVPN connection > Edit > Networking > Domains
we'll go with WONTFIX for now since you have a local fix, thank you for documenting here for future users!