Closed Bug 166479 Opened 23 years ago Closed 23 years ago

DNS: user resinit() to update DNS list when lookup fails (changing networks, DHCP, etc.)

Categories

(Core :: Networking, defect, P3)

Product:
Core
Component:
Networking
Platform:
x86
Linux
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.2beta

People

(Reporter: benc, Assigned: darin.moz)

References

Details

Attachments

(1 file, 1 obsolete file)

v1.01 patch - removes #undef HAVE_RES_NINIT added there only for testing
1.93 KB, patch
dougt
: review+
blizzard
: superreview+
brendan
: approval+
Details | Diff | Splinter Review
STEPS: 1- Launch mozilla, surf anywhere using DNS. 2- Cause (or suffer) any DNS unavailability. Hostmasters w/ a test server can kill named. Multi-network users can switch networks. 3- Attempt to surf again, using DNS. OBSERVED BEHAVIOR: the DNS server continues to query the old DNS servers. If the servers are not available, a normal error message is returned. EXPECTED BEHAVIOR: If possible, an updated DNS list should be read from /etc/resolv.conf, via resinit(). If the new list is incorrect, and error message should be returned. NOTE: Currently, there is no accurate error for a DNS query that fails b/c the nameserver was unreachable. If possible, this needs to be fixed as well. Part of the delay in catching this regression was because the error message returned is not accurate. (bug 164715).
I also wonder what other platforms are affected. Win32 and Mac OS X seem unaffected. Solaris, OS/2, BeOS and BSD are on my curiosity list.
Keywords: mozilla1.2, nsbeta1, regression
There are security reasons we do not do this such as the princeton exploit.
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → WONTFIX
benc: what mozilla version are you testing? this WORKSFORME w/ a current trunk build under linux. res_ninit is called when a DNS lookup fails: http://lxr.mozilla.org/seamonkey/source/netwerk/dns/src/nsDnsService.cpp#1188 for security reasons, as dougt pointed out, we remember host:port->ipaddr mappings that result in a successful connection. we continue to use the same ipaddr for each host:port for the duration of the browser session. also note: toggling the offline/online UI clears the browsers memory of all host:port->ipaddr mappings.
Doug, please don't WONTFIX this so fast. There has to be a secure way for me to roam my laptop from home to work to home, and have DNS rebind. I do not believe "security" can justify breaking that. It sounds like the res_ninit code should still kick in. I'll test tomorrow and report back. Reopening for now, as this should either be WFM or a bug to fix, but not WONTFIX. /be
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
I'm not saying we should update the pinned named socket->addressed sockets, I'm talking about reloading the DNS server list. I'll pull the tomorrows trunk per darin, and provide steps.
hostname->ip are pinned. if you have foo resolving to some intranet machine and disconnect VPN, you will never get the external foo.com. As the bug was written up, I wasn't exactly sure what we were talking about.
(thought darin did something to make the content pinned to the ip:port number...) I'll provide complete steps later, then you can decided if you want to WONTFIX or not.
ip address pinning is host:port -> ipaddr ... and we only pin after a successful connection.
Doug's comment #6 talks about a non-problem -- I never want to resolve warp.mcom.com or whatever its FQDN is to two different machines, nor can I resolve it outside the corporate firewall. I wouldn't want warp to resolve outside either (I don't search com or anything crazy like that). Still at home, I'll roam in and then back out (out has been a problem for me, not sure why -- could be my own RH7.1 DNS config is screwed up, and nothing to do with Mozilla code). /be
STEPS: 1- Configure /etc/resolv.conf to a DNS server that is not responding. You can make up a DNS server IP address, I turned off my test DNS server to ensure I would get a failed lookup. 2- launch mozilla. If you have a home page, ignore its loading, because the results is probably in cache. 3- request page via DNS (most default links are to redhat.com, use these if you can). NOTE: there is a pause as the resolver timeout is exceeded. 4- "www.redhat.com" could not be found. Plese check the name and try again." 5- Fix the DNS server list by editing /etc/resolv.conf so it has working nameservers. (I used "pump" to reset the list via DHCP). 6- surf to www.redhat.com again. OBSERVED BEHAVIOR: DNS lookup is not successful, you get the same error again. EXPECTED BEHAVIOR: mozilla would decide that the DNS failure could be due to a stale DNS server list, and use resinit() to refresh the list, then try the DNS query again. This might be a duplicate, I've seen some people complain that our DNS services dies after a timeout or other unspecified failure. This also explains why this works for Darin, but not for Brendan. The likely litmus test seems to be: if you can ping your old nameserver, you will have this problem. If you ping your old nameserver and you get "network unreachable" then the DNS server refresh will seem to work. I've used both Mozilla 2002-09-04-trunk and Mozilla 1.0RC3 for these results (per Darin's request for trunk and following Brendan's comments about RC3 working).
interesting... the case you describe is meant to work. and it should be possible to make it work without violating any of the security restrictions. i'm going to try to repro the problem following benc's steps exactly.
benc: what version of redhat are you using? more importantly, what version of glibc do you have installed on your system?
This may also be platform specific. I just realized that while I was tormenting my Linux system by dribbling named up and down, I was also tormenting my Mac OS X system, but Mozilla on Mac OS X recovered w/o much problem. I had actually started some of my DNS and connectivity functional test case writing on Mac OS X, which explains why it was so hard for me to figure out what was going on here. I'll do some more head scratching and testing...
ok, interesting... i'm able to repro benc's steps. investigating...
ah hah!! my debug build, compiled on RH 7.3 works like a charm! however, the nightly build off of ftp.mozilla.org seems to have a problem with this. i suspect it might not be including the call to res_ninit or something like that. it is #ifdef HAVE_RESNINIT.
but wait, when i was roaming in and out of my VPN connection, i was using the same nighly build off of f.m.o. in that case, i never exercised the exact same steps reported by benc, so hmm...
i just checked config-defs.h on btek (which runs redhat 6.2) and i noticed that it does not include HAVE_RES_NINIT. iirc, the nightly builds are done using redhat 6.2, so i guess this explains the problem. i checked /usr/include/resolv.h, and there is unfortunately no sign of res_ninit anywhere. it does provide res_init, but as we know that isn't meant to be threadsafe. i'm not sure what we can do to fix this problem other than to start having separate nightly builds for redhat 6.x and redhat 7.x/8.x.
i suspect my success with the VPN software might have something to do with the way VPN works... perhaps moving in and out of the VPN was triggering a low-level sort of resinit within glibc or perhaps even lower. certainly it would have killed off any old socket connections. i'll play with this some more when i'm home...
If you're using SERA on Linux, I believe it actually redirects DNS requests at the device level.
yeah, i was starting to think that something like that might explain why trunk builds appear to work in my VPN scenario. benc and i spent some time testing a bunch of scenarios, watched DNS traffic, and pretty much concluded that this is simply broken. in fact, it has never worked [in nightly builds]. it works great in builds done on redhat 7.x systems, and i verified that it worked on the trunk probably using my VPN example.
While we are discussing the status of resinit(), I would be interested in hearing any security concerns, either here or in a new bug w/ a security flag.
Verifying this bug sucks. I, for one, don't know how to packet trace well in Linux, so I haven't been looking at this directly, I've been running snoop w/ a filter on my DNS server. Troubleshooting VPN problems are also hard, because you can't exactly run an external tracer on your local network. I've filed Bug 166796, and I'm surprised it took me so long to get off my tail and deal with this.
FWIW, res_ninit first appears in the BIND8 resolver library. Glibc 2.2.x uses that but glibc 2.1.x uses the older BIND4. That's an incompatibility that will eventually bite mozilla.
right, it sounds like the only solution to this bug is to have separate builds for rh6 and rh7.
Why do we care about RH6.x at this time? /be
I think that is what the official build machines are using.
Yes, the build machines use RedHat 6.2 for releases. IIRC, we decided to use RH6 as the base system configuration that we "support" due to dynamic loader issues with RH5.x (specifically glibc 2.0). I don't know about the market numbers but I would be very surprised if we could get away with dropping RH6 support at this time. Providing multiple builds would be easy enough but then you run into the age old problem of providing QA for multiple sets of builds on a daily basis. Another alternative would be to just start requiring a recent version of the resolver even if that means installing and linking against a separate copy of the resolver library. However, I just installed the bind9 devel rpms and I don't see res_ninit in any of the newly installed headers though it might be just a bind8 feature.
ulrich drepper advised us to use res_ninit.
another solution would be to proxy synchronously to the main thread, and from there call res_init. besides some fun locking issues, that shouldn't be too difficult to implement.
i take it back, proxying wouldn't help if someone tried to call gethostbyname on some background thread. hmm...
I suggest detecting res_ninit at run time. Just call the nspr equivalent of dlsym. That's how gfx find libfreetype and tests for the right version. You use either the real version or a stub. An indirect call for a function that's rarely used shouldn't have much impact. In fact, a more general capabities test may be in order. Glibc has functions to return the version and release. Mozilla could be clever and use its own res_ninit if it knew the libc version it was using.
*** Bug 162435 has been marked as a duplicate of this bug. ***
*** Bug 151683 has been marked as a duplicate of this bug. ***
[12:15:49] <blizzard> I guess that res_ninit() doesn't exist on older glibc versions [12:17:17] <foo> _res_ninit was introduced when we added IPv6 support to the resolver [12:18:29] <foo> and if you build a program referencing a new symbol which isn't present the startup will fail [12:18:34] <foo> the alternative is to use [12:19:15] <foo> dlsym (RTLD_DEFAULT, "__res_ninit") [12:19:31] <foo> and use it if available
The simplest solution that doesn't depend on knowing lib versions or anything else that can change randomly is to use a separate process(es) for name resolution.
*** Bug 168954 has been marked as a duplicate of this bug. ***
Blocks: 120105
working on a patch...
Assignee: new-network-bugs → darin
Status: REOPENED → NEW
Status: NEW → ASSIGNED
Priority: -- → P3
Target Milestone: --- → mozilla1.2beta
Attached patch v1 patch (obsolete) — — Splinter Review
this patch implements the call to dlsym(RTLD_DEFAULT, "__res_ninit") as suggested.
Keywords: regression → patch
Three questions: 1) shouldn't you use "res_ninit" rather than "__res_ninit"? 2) should you use RTLD_DEFAULT or RTLD_NEXT? 3) should you consider the possibility that a return value of 0 might actually be correct? The Linux dlsym man page goes on about this although it would require a rather perverse configuration to actually see it.
Ok, four questions. _res seems to be defined as (*__res_state()). Wouldn't it be better to use _res which also exists in BIND4.
> 1) shouldn't you use "res_ninit" rather than "__res_ninit"? No. There is no res_ninit function exported. > 2) should you use RTLD_DEFAULT or RTLD_NEXT? No. Why should it be "NEXT"? You haven't found a first instance. > 3) should you consider the possibility that a return value of 0 might > actually be correct? The Linux dlsym man page goes on about this > although it would require a rather perverse configuration to actually > see it. I don't know what you mean. dlsym() returning NULL can never be correct on any system in the last almost 7-8years since we didn't map anything in the first pages. > _res seems to be defined as (*__res_state()). Wouldn't it be better to > use _res which also exists in BIND4. This is correct. You can simply use _res. It makes the interface nicer. But that's the only difference.
> 1) shouldn't you use "res_ninit" rather than "__res_ninit"? the symbol name is __res_ninit, so that's what i need to lookup. > 2) should you use RTLD_DEFAULT or RTLD_NEXT? the first instance of __res_ninit should do fine, so RTLD_DEFAULT makes sense. > 3) should you consider the possibility that a return value of 0 might > actually be correct? The Linux dlsym man page goes on about this > although it would require a rather perverse configuration to actually > see it. well, i don't exactly think we should try invoking a function at address 0. > 4) _res seems to be defined as (*__res_state()). Wouldn't it be better to > use _res which also exists in BIND4. uhm, the point is that we don't have access to resolv.h, so we can't use _res unless we #define it ourselves, which doesn't seem to add much value.
1) Yes, you're right. I'm an idiot. 2) It only matters if some 3rd-party library is loaded with that symbol. May be important, may not. 3) I know 0x0 isn't normally mapped but it is possible. It's probably not a good idea but it is legal. That's why I called it perverse. It's not important. 4) I would think that you would want to include resolv.h if you're doing tricky stuff like this. Can you use NSPR services? A couple of minor points. 5) There's a small performance penalty if you do the dlsym every time. Couldn't this be in a constructor or some such. It's highly unlikely the function address will change over the process's lifetime. 6) Isn't keeping HAVE_RESINIT defeating the whole purpose here. The build machine isn't the run-time machine.
> 2) It only matters if some 3rd-party library is loaded with that symbol. > May be important, may not. No, RTLD_NEXT is plainly wrong and might very well fail or crash the application. You only can use RTLD_NEXT once you found a first instance of a function either by using RTLD_DEFAULT or by a normal function call. > 3) I know 0x0 isn't normally mapped but it is possible. It's probably > not a good idea but it is legal. That's why I called it perverse. It's > not important. This is a Linux specific change (at least I don't know any other system using __res_ninit). And we don't use memory at location zero in the last 7-8 years. But even *if* there would be a DSO wrapped, no function could start at that address since at offset zero you always find the ELF header. And wrapping around the end of the address space is invalid. There is no way a NULL result can mean anything but an error. > 4) I would think that you would want to include resolv.h if you're doing > tricky stuff like this. Can you use NSPR services? No comment on that. I'd expect <resolv.h> to be used, too. But there might very well be buid issues. > 5) There's a small performance penalty if you do the dlsym every time. > Couldn't this be in a constructor or some such. It's highly unlikely the > function address will change over the process's lifetime. How often is the function containing this code called? Caching the value is indeed a good way since the definition is always present and dlsym() does require some effort. > 6) Isn't keeping HAVE_RESINIT defeating the whole purpose here. The > build machine isn't the run-time machine. Unless the target system must be at least as recent as the deployment system (I don't know whether this is a requirement) this indeed looks wrong. For maximum portability of the binary you still could build on an old system but to get better binaries for modern systems you can build on a modern system.
> 4) I would think that you would want to include resolv.h if you're doing > tricky stuff like this. Can you use NSPR services? we build mozilla using a redhat 6.2 box (iirc), and redhat 6.2's glibc does not include support for res_ninit. hence, it has no #define for _res. hence, there is no point to #include'ing the header file if the autoconf test for res_ninit failed (i.e., #ifndef HAVE_RES_NINIT). > 5) There's a small performance penalty if you do the dlsym every time. > Couldn't this be in a constructor or some such. It's highly unlikely the > function address will change over the process's lifetime. dlsym is only called the first time __res_ninit is needed. that is what the static PRBool tried is all about. > 6) Isn't keeping HAVE_RESINIT defeating the whole purpose here. The > build machine isn't the run-time machine. see my comments above. the RH 7.x RPMS available on ftp.mozilla.org do not need these calls to dlsym, so why not just let RH 7.x specific builds link to __res_ninit explicitly? this patch leaves those builds unchanged, and i think that is a good thing.
> the RH 7.x RPMS available on ftp.mozilla.org do not need these calls > to dlsym, so why not just let RH 7.x specific builds link to > __res_ninit explicitly? I thought the idea was to allow a a binary built on a glibc 2.2 system, e.g. RH7, to work on a glibc 2.1 system, e.g. RH6.2. As you have it now that's impossible.
> But even *if* there would be a DSO wrapped, no function could start at > that address since at offset zero you always find the ELF header. And > wrapping around the end of the address space is invalid. There is no > way a NULL result can mean anything but an error. On Linux, ld puts the ELF headers at 0x08048000. 0x0 is available. You can mmap things there and read and write and execute. But let's forget this. It's just getting in the way.
well, that might be nice, but the explicit point of this patch is to make it work the other way around because mozilla.org ships a binary compiled on redhat 6.2 that is targeting redhat 7.x as well. dropping the #ifdef HAVE_RES_NINIT code should be another bug altogether IMO.
I don't think that compiling on RH7 and running on RH6 is a laudable goal here. There are lots of reasons why that won't work. Please leave in the check for the correct symbol name so that those of us compiling on RH7 use the right code path.
Comment on attachment 100260 [details] [diff] [review] v1.01 patch - removes #undef HAVE_RES_NINIT added there only for testing r=dougt
Attachment #100260 - Flags: review+
Comment on attachment 100260 [details] [diff] [review] v1.01 patch - removes #undef HAVE_RES_NINIT added there only for testing sr=blizzard
Attachment #100260 - Flags: superreview+
fixed-on-trunk
Status: ASSIGNED → RESOLVED
Closed: 23 years ago → 23 years ago
Resolution: --- → FIXED
Comment on attachment 100260 [details] [diff] [review] v1.01 patch - removes #undef HAVE_RES_NINIT added there only for testing a=brendan@mozilla.org for the 1.0 branch. /be
Attachment #100260 - Flags: approval+
fixed1.0.2
Keywords: fixed1.0.2
*** Bug 144847 has been marked as a duplicate of this bug. ***
benc: can you pls verify this as fixed on the 1.0 branch, then replace "fixed1.0.2" with "verified1.0.2"? thanks!
I've read through the discussion of code implementations, but can someone tell me what the expected design change would be? Will this work in RH 6 post-fix, or not? Also, what's the scope of this fix? In the previous bug, nobody ever told me if we fixed other UNIX versions, like Solaris. (from patch) +// try to dynamically resolve "__res_ninit" (we may want to enable this +// for other XP_UNIX platforms as well) +#if defined(HAVE_RES_NINIT) || defined(RESOLVE_RES_NINIT) // On platforms where res_ninit() is available and supported // by Mozilla, we shall attempt to reinitialize the dns
benc: this patch was linux only. and it only fixes the problem for a redhat 6 build when run on a redhat 7 or later system.
This is working for me now. Redhat 7.3. Thanks.
I've got only RedHat 6 easily available, so I can verify part 2 of what Darin said.
Ben, Charles has 7.3.
VERIFIED/branch 10-03-08-01 jimmy, thanks for the info... Linux 7.3: launch mozilla surf w/ DNS turn off dns server surf, see errors. chang resolv.conf surf some more, and it works. The only think I can't figure out, is in some situations, w/ a bad resolv.conf, the browser manages to find a DNS server after a while. I'm not that familar with name resolution in Linux, I've looked at the man pages for host.conf, checked to see there are no nscd-type services, but it still seems to magically work in some cases. Still, if we go one week w/o anyone complaining about this bug on the trunk, I'm pretty confident this works, even if my test case didn't completely hose DNS when I wanted it to go down.
Keywords: fixed1.0.2 → verified1.0.2
*** Bug 177781 has been marked as a duplicate of this bug. ***
*** Bug 134681 has been marked as a duplicate of this bug. ***
VERIFIED/trunk: I was unable to get the exact verification I wanted on Charles' linux 7 system, the OS seems to find a name server even if resolve.conf points to a single, downed server. However, nobody has provided a recent, confirmed problem report, and Darin has actually watched for resinit() calls to make sure it works as desired.
Status: RESOLVED → VERIFIED
I do not see this behaviour in the 1.2beta release.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: