OSX getaddrinfo returns concatenated results from /etc/hosts and DNS, which breaks /etc/hosts based ad blocking



15 years ago
15 years ago


(Reporter: jon, Assigned: darin.moz)


Mac OS X

Firefox Tracking Flags

(Not tracked)



(2 attachments, 2 obsolete attachments)



15 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.6a) Gecko/20031009 Camino/0.7+
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.6a) Gecko/20031009 Camino/0.7+

Before I just upgraded Camino from about a month ago nightly build, it was
properly using the OSX lookupd to find hosts. Now, it seems it is using its own
mechanism to find hosts.

The way I know this is not working in Camino and my machine is working is that
Safari works properly.

There is bug #194476, but that is a different problem so don't close this as a
duplicate of that one.

Here is my lookupd hosts configuration:

lookupd -configuration

LookupOrder: CacheAgent FFAgent DNSAgent
ValidateCache: NO
_config_name: Host Configuration
name: hosts

Here is my /etc/hosts localhost broadcasthost
::1             localhost a.tribalfusion.com

Now, when I go to friendster.com using this version of Camino, I see the ads
which are served by a.tribalfusion.com. Prviously it didn't show any ads. If I
use Safari, I don't see the ads.

I also modified my lookupd configuration and removed DNSAgent to confirm that
lookupd is properly reading its configuration and that is true. It is as if
Camino isn't using lookupd at all!

I also just upgraded to OSX 10.2.8 and thought it might be something with the
new version of OSX, but the lookupd version is the same as what is on 10.2.6.
The fact that Safari works also confirms that it isn't a difference in OSX
versions that is causing the problems.

I'm a bit surprised no one else has reported this issue.

Reproducible: Always

Steps to Reproduce:
see above
Actual Results:  
I saw the ads.

Expected Results:  
Don't show me the ads. =)


15 years ago
Severity: blocker → major
darin? did anything change?
Assignee: pinkerton → darin
Component: General → Networking
Product: Camino → Browser
QA Contact: benc
Version: unspecified → Trunk

Comment 2

15 years ago
I would love it if someone would try to just duplicate the problem and see if
they are having it as well.

It is very easy to do:




Comment 3

15 years ago
what versions of gecko are we comparing here?  was the old version of camino
using the 1.0 branch builds?  i haven't been following camino, so i don't know
what a one month old build means.

also i have no access to a mac with which to test this.  the patch for bug
205726 recently went into the tree.  that should not have had too much affect on
the OSX build, though possibly it means that we are now calling getaddrinfo
instead of gethostbyname on OSX.

Comment 4

15 years ago
adding a line like this to /etc/hosts a.tribalfusion.com

works fine (i.e., blocks URLs from loading from that host) with yesterday's
trunk build under linux (RH9).

again, without access to a mac, there's not much i can do with this bug.  it
might be useful if someone provided a mozilla socket log.  there are
instructions on this site which explain how to capture this log:


all i need is to see NSPR_LOG_MODULES=nsSocketTransport:5


Comment 5

15 years ago
Created attachment 133348 [details]
log of debugging output

Comment 6

15 years ago
exerpts from the log file:

29141392[1c23b60]:   trying address:
29141392[1c23b60]:   connection failed! [reason=804b000d]
29141392[1c23b60]: nsSocketTransport::RecoverFromError [this=2df1770 state=3
29141392[1c23b60]:   trying again with next ip address
29141392[1c23b60]:   trying address:


mozilla-trunk> host a.tribalfusion.com
a.tribalfusion.com has address

so, the conclusion i'd draw is that OSX is giving the application both
and as possible choices for a.tribalfusion.com.  since in
reality servers can be multi-homed, there is nothing mozilla can or should do to
work around this behavior.  i'd suggest filing a bug against the Apples since
this is clearly a system issue.

most likely we are now calling getaddrinfo instead of gethostbyname on OSX, and
that probably explains the difference in behavior.  we are calling getaddrinfo
in order to support IPv6, and moreover it is the preferred way to do host

i think this bug report is INVALID.
Last Resolved: 15 years ago
Resolution: --- → INVALID

Comment 7

15 years ago
heh.. "the Apples".. not sure where that came from.  i meant, "Apple" of course ;)

Comment 8

15 years ago
I understand that none of you seem to be OSX users (bah! =)), but I think that
you should switch back to using gethostbyname on OSX for three reasons:

#1. the number of people using IPv6 at this point is minimal compared with the
number still using IPv4. As with browser compatibility wars, sometimes it is
best to compromise with what the end user sees and wants as part of the user

#2. This broke functionality of the browser that is extremely useful. I don't
want to see ads and I really like this browser and don't want to have to switch
back to using Safari. Put yourself in my seat. I also don't expect that this is
going to be the last time we hear about this issue. As soon as all of the OSX
users out there who use a hosts file for blocking ads get the new version of
Mozilla, they are going to ask the same questions as I am.

#3. According to the OSX man page for getaddrinfo on 10.2.8, the implementation
is still experimental and non-standard and therefore can be taken as something
to not rely on as the preferred way to do things:

     This implementation is still very experimental and non-standard.  The
     current implementation assumes a one-to-one relationship between inter-
     faces and links, which is not necessarily true according to the specifi-

I will file a bug report with Apple about this issue, however, I think that
until that bug is fixed (does anyone have a copy of Panther to try?), the
previous 'working' behavior should be restored on OSX. Maybe even have Mozilla
conditionally use getaddrinfo only on 10.3 and on 10.2 go back to using

Thank you for your consideration on this matter.
Resolution: INVALID → ---

Comment 9

15 years ago
I filed this as problem ID 3454903 with Apple.

Comment 10

15 years ago
jon: ok, maybe there is something we can do...

hmm.. i wonder if other BSDs behave similarly.

wtc: do you have any thoughts on this bug?  apparently it seems that getaddrinfo
returns results from both /etc/hosts and DNS.  this breaks the ad blocking
technique of listing ad servers under to bypass DNS lookup.  i suspect
OSX is broken in doing this, but maybe there is no promise that it will not
query both /etc/hosts and DNS.  apparently, if we switch to calling
gethostbyname, this problem goes away.  that makes me suspect that the behavior
getaddrinfo is unintended.

one solution is probably to define _PR_INET6_PROBE, so that we will only call
getaddrinfo on IPv6 enabled clients.
Summary: Camino no longer uses lookupd to find dns entries → OSX getaddrinfo returns concatenated results from /etc/hosts and DNS, which breaks /etc/hosts based ad blocking

Comment 11

15 years ago
I think this pretty much says it...so, can I get you guys to put it back the way
it was on 10.2 now?


------ Forwarded Message
From: Marc Majka <majka@apple.com>
Date: Fri, 17 Oct 2003 09:55:45 -0700
To: darwin-development@lists.apple.com
Subject: Re: getaddrinfo bug

The implementation of getaddrinfo in Mac OS X 10.2 was written to 
support our initial version of IPv6.  It ended up getting used a lot 
more than we had expected.  This was good news, in that it showed that 
there is a lot of code ready to take advantage of IPv6.  We've done 
lots of work to improve IPv6 in 10.3.  Included in that effort was an 
overhaul of getaddrinfo and getnameinfo.  getaddrinfo no longer 
searches all sources for information (e.g. /etc/hosts + NetInfo + DNS). 
  In 10.3 it follows lookupd's host search order and stops at the first 
"match".  It's also faster and takes more advantage of local caching to 
gain better performance.

Marc Majka
darwin-development mailing list | darwin-development@lists.apple.com
Do not post admin requests to the list. They will be ignored.

------ End of Forwarded Message

Comment 12

15 years ago
We can certainly determine the runtime OS version
and use getaddrinfo() only if we are on Mac OS X
10.3 or later, and use gethostbyname() if we are
on 10.1 and 10.2.  Could one of you ask Apple to
review this decision?

Does anyone know what is the version of Darwin in
Mac OS X 10.3?  This is the output of the "uname -r"
uname -r returns: "7.0.0"

Comment 14

15 years ago
--> nspr
Component: Networking → NSPR
Ever confirmed: true
Product: Browser → NSPR
Version: Trunk → 4.2

Comment 15

15 years ago
What I just got back from Apple on the subject...

now i'm curious when this will get fixed. =)

------ Forwarded Message
From: devbugs@apple.com
Date: Tue, 21 Oct 2003 19:50:39 GMT
Subject: Re: Bug ID #3454903: getaddrinfo lookup behavior is not the same as
with gethostbyname

Please include the line below in follow-up emails for this request.

Follow-up:  2999907

Re: Bug ID #3454903:  getaddrinfo lookup behavior is not the same as with

Hello Jon,

Thank you for taking the time to bring this issue to our attention. 
Engineering has reviewed your report and has determined that this is
behavior intended by design and is working correctly per engineering

Below, we have provided comments from engineering to help further
explain this conclusion:

>From your bug report, it sounds as if you were trying to block ads in a 
web browser by putting bogus entries in /etc/hosts.  The Mac OS X 10.2 
implementation of getaddrinfo consulted both /etc/hosts and DNS.  This 
meant that the idea of blocking unwanted banner advertising by screening 
out unwanted hosts with bogus host entries in /etc/hosts did not work.  The
behavior of getaddrinfo changed in Mac OS X 10.3 so that it simply followed
the lookupd host search order and stopped at the first entry.  This means that 
in Mac OS X 10.3, one can block access to certain hosts with bogus hosts entries. 

We hope this information is helpful.  Please do not hesitate to contact
us should you wish to discuss this matter further.

Thank you for your time and effort.  Your help is greatly appreciated.

Best Regards,

Maurice Cusseaux
Apple Developer Connection
For follow-ups please contact devbugs@apple.com

Comment 16

15 years ago
Created attachment 134432 [details] [diff] [review]
v1 patch

simple patch to disable getaddrinfo on OSX.  this means that gethostbyname will
be called instead, which at the moment it treated as non-threadsafe under OSX. 
this is not true for OSX 10.2 and above.  so, i'm not entirely happy with this
patch alone.  i'm also not sure if OSX will return IPv6 addresses via
gethostbyname.	in which case this patch is definitely not good enough.

Comment 17

15 years ago
I think we want to use getaddrinfo on 10.3 (Darwin kernel 7.0) and higher. (10.3
has been released, in case you didn't know.) It's only 10.2 that has this
problem, correct? Let's not hamstring future users. Ideally, Mozilla should
automagically use gethostbyname on 10.0 - 10.2.x and getaddrinfo on 10.3 and up.
That means a runtime initialization probe, I think. Perhaps too complicated. A
better compile-time-oriented patch would do something like this:


Comment 18

15 years ago
ed: yes, i agree.. i'm not happy with the v1 patch either.  i just put it up as
an option.

Comment 19

15 years ago
A possible solution would be to use getipnodebyname instead of gethostbyname. It
supports IPv6 and IPv4-mapped addresses, is thread safe, and supports the
AI_ADDRCONFIG flag. See bug 68796 comment #43.

Comment 20

15 years ago
What we really need is a mechanism that specifically supports ad blocking.
Besides people that are hacking their /etc/hosts files, is anyone else having a
problem with this behavior?

I've always disliked this user hacking of /etc/hosts. If we want to support ad
blocking in this situation, can't we just decide to cancel the request for a
hostname if the IP address is

I think this is network-incoherent, at least I've never seen any technical
discussion that explains why an address is the whole IPv4 network should be
allowed in /etc/hosts. TCP/Illustrated says that this cannot be a valid
desintation address. I'm sure that is what people intend when they hack their
/etc/hosts files, but do we want to encourage this behavior? Do we know other
software will handle this bogus entry correctly?

Comment 21

15 years ago
Ha! the crux of the problem! =)

It would be great to have a simple list that says:

Hosts to block access to:

It should be available via scripting as well so that outside applications can
add/remove items from the list easily.

Note that one benefit of using /etc/hosts over just having an application level
blocking mechanism is that it works in all of the browsers that I use, not just
Camino. So, maybe coming up with another central database that all browsers
agree on using would be better. Until then /etc/hosts is gonna have to be that

p.s...here is my list...=) it is amazing how much faster web pages load and how
much **** i just don't see anymore...it almost making using a browser on a
non-configured machine unbearable... ar.atwola.com ads.intuit.com Garden.ngadcenter.net Ogilvy.ngadcenter.net ResponseMedia-ad.flycast.com Suissa-ad.flycast.com UGO.eu-adcenter.net VNU.eu-adcenter.net a32.g.a.yimg.com ad-adex3.flycast.com ad.adsmart.net ad.ca.doubleclick.net ad.de.doubleclick.net ad.doubleclick.net ad.fr.doubleclick.net ad.jp.doubleclick.net ad.linkexchange.com ad.linksynergy.com ad.nl.doubleclick.net ad.no.doubleclick.net ad.preferences.com ad.sma.punto.net ad.uk.doubleclick.net ad.webprovider.com ad08.focalink.com adcontroller.unicast.com adcreatives.imaginemedia.com adex3.flycast.com adforce.ads.imgis.com adforce.imgis.com adfu.blockstackers.com adimage.blm.net adimages.earthweb.com adimg.egroups.com admedia.xoom.com adpick.switchboard.com adremote.pathfinder.com ads.admaximize.com ads.bfast.com ads.clickhouse.com ads.enliven.com ads.fairfax.com.au ads.fool.com ads.freshmeat.net ads.hollywood.com ads.i33.com ads.infi.net ads.jwtt3.com ads.link4ads.com ads.lycos.com ads.madison.com ads.mediaodyssey.com ads.msn.com ads.ninemsn.com.au ads.seattletimes.com ads.smartclicks.com ads.smartclicks.net ads.sptimes.com ads.tripod.com ads.web.aol.com ads.x10.com ads.xtra.co.nz ads.zdnet.com ads01.focalink.com ads02.focalink.com ads03.focalink.com ads04.focalink.com ads05.focalink.com ads06.focalink.com ads08.focalink.com ads09.focalink.com ads1.activeagent.at ads10.focalink.com ads11.focalink.com ads12.focalink.com ads14.focalink.com ads16.focalink.com ads17.focalink.com ads18.focalink.com ads19.focalink.com ads2.zdnet.com ads20.focalink.com ads21.focalink.com ads22.focalink.com ads23.focalink.com ads24.focalink.com ads25.focalink.com ads3.zdnet.com ads5.gamecity.net adserv.iafrica.com adserv.quality-channel.de adserver.dbusiness.com adserver.garden.com adserver.janes.com adserver.merc.com adserver.monster.com adserver.track-star.com adserver1.ogilvy-interactive.de adtegrity.spinbox.net antfarm-ad.flycast.com au.ads.link4ads.com banner.media-system.de banner.orb.net banner.relcom.ru banners.easydns.com banners.looksmart.com banners.wunderground.com barnesandnoble.bfast.com beseenad.looksmart.com bizad.nikkeibp.co.jp bn.bfast.com c3.xxxcounter.com califia.imaginemedia.com cds.mediaplex.com click.avenuea.com click.go2net.com click.linksynergy.com cookies.cmpnet.com cornflakes.pathfinder.com counter.hitbox.com crux.songline.com erie.smartage.com etad.telegraph.co.uk fp.valueclick.com gadgeteer.pdamart.com gm.preferences.com gp.dejanews.com hg1.hitbox.com image.click2net.com image.eimg.com images2.nytimes.com jobkeys.ngadcenter.net kansas.valueclick.com leader.linkexchange.com liquidad.narrowcastmedia.com ln.doubleclick.net m.doubleclick.net macaddictads.snv.futurenet.com maximumpcads.imaginemedia.com media.preferences.com mercury.rmuk.co.uk mojofarm.sjc.mediaplex.com nbc.adbureau.net newads.cmpnet.com ng3.ads.warnerbros.com ngads.smartage.com nsads.hotwired.com ntbanner.digitalriver.com ph-ad05.focalink.com ph-ad07.focalink.com ph-ad16.focalink.com ph-ad17.focalink.com ph-ad18.focalink.com rd.yahoo.com realads.realmedia.com redherring.ngadcenter.net redirect.click2net.com regio.adlink.de retaildirect.realmedia.com s2.focalink.com sh4sure-images.adbureau.net spin.spinbox.net static.admaximize.com stats.superstats.com sview.avenuea.com thinknyc.eu-adcenter.net tracker.clicktrade.com tsms-ad.tsms.com v0.extreme-dm.com v1.extreme-dm.com van.ads.link4ads.com view.accendo.com view.avenuea.com w113.hitbox.com w25.hitbox.com web2.deja.com webads.bizservers.com www.PostMasterBannerNet.com www.ad-up.com www.admex.com www.alladvantage.com www.burstnet.com www.commission-junction.com www.eads.com www.freestats.com www.imaginemedia.com www.netdirect.nl www.oneandonlynetwork.com www.targetshop.com www.teknosurf2.com www.teknosurf3.com www.valueclick.com www.websitefinancing.com www2.burstnet.com www4.trix.net www80.valueclick.com z.extreme-dm.com z0.extreme-dm.com z1.extreme-dm.com spinbox.versiontracker.com netadsrv.iworld.com ads.admonitor.net ads.imaginemedia.com advertising.com kermit.macnn.com servedby.advertising.com ads.tucows.com service.bfast.com vsii.spinbox.net ds.sf.futurenet.com adserver.bizland-inc.net ads4.earthweb.com www.sponsorships.net spinbox.macpublishing.net www.adtagger.com adcreatives.radcity.net sponsors.waveshift.com ad.au.doubleclick.net ads.intelihealth.com oz.valueclick.com banner.linkexchange.com ads.adflight.com ads.applelinks.com adsrv.ilife.com iv.doubleclick.net images.go2net.com connect.247media.ads.link4ads.com links.quinstreet.com adserver.ads360.com www.1blink.com banners.freeware.ru a.tribalfusion.com ads.eudora.com ads2.eudora.com www.qksrv.net spinbox.techtracker.com ads.uniquemedia.net ads-e.focalink.com littlebuddy.apple.com server2.as5000.com adserver1.backbeatmedia.com tracker.tradedoubler.com us.a1.yimg.com ads1.ad-flow.com ads.quicken.com fl01.ct2.comclick.com adserv.sapo.pt mjxads.internet.com ads.businessweek.com ads.dealnews.com ads.courier-journal.com www.whitehouse.com www.actinxxx.com www.adultfilmarchive.com www.ainews.com www.avn.com www.barefacts.com www.eroticmoviecompendium.com www.iafd.com www.lukeford.com www.rame.net www.sextracker.com www.escapade.nl www.sexhound.com the.sextracker.com outlawedjapan.com www.bestamateur.com www.cumqueenkari.com www.virtuagirl.com www.thumbnetwork.com www.max-productions.com www.smutserver.com www.yahan.com www.vixenvideos.com www.wantonwife.com www4.smutserver.com www.suzi-wong.com www.100freexxxpics.com www.creamypies.com www.cumshotfantasies.com www.gigme.com www.terra.es pornstars.eroticism.com samples.celebclub.com www.girls4guys.com www.hardandfast.com programs.amateurpages.com www.kostenlosecams.de www.cumexcite.com maya21.myifriends.net www.algonet.se www.freesextgp.com www.thumbshack.com www.amateurfreehost.com www.sextext.com sextracker.com www.stickyfist.com www.eros-island.com smutserver.com www.pussyfiend.com www.ispyceleb.com www3.smutserver.com www2.smutserver.com www.just-raw-sex.com www.thehun.com ads.specificpop.com ads.osdn.com ads.as4x.tmcs.net z1.adserver.com 1.marketbanker.com www.marketbanker.com pagead2.googlesyndication.com graphics7.nytimes.com a.as-us.falkag.net netshelter.adtrix.com pr.atwola.com www.timeinc.net rmedia.boston.com statse.webtrendslive.com

Comment 22

15 years ago
Created attachment 136387 [details] [diff] [review]
v2 patch
Attachment #134432 - Attachment is obsolete: true


15 years ago
Attachment #136387 - Flags: review?(wchang0222)

Comment 23

15 years ago
Created attachment 136390 [details] [diff] [review]
v2.1 patch

revised patch.	wtc noted that _darwin.h needs to enable _PR_INET6_PROBE for
target platforms less than 10.3 not less than 10.2.


15 years ago
Attachment #136387 - Attachment is obsolete: true


15 years ago
Attachment #136387 - Flags: review?(wchang0222)


15 years ago
Attachment #136390 - Flags: review?(wchang0222)

Comment 24

15 years ago
Looks pretty good, Darin. Forgive the following newbie questions: What is the
current value of MACOS_DEPLOYMENT_TARGET and how is its value
assigned/determined? Will the runtime check you've added in the patch to ptio.c
be done just once or will that code be run with every connection made?

Comment 25

15 years ago
MACOS_DEPLOYMENT_TARGET is set by NSPR's configure script.  there is an argument
that can be passed to NSPR's configure script to alter the value.  the default
value appears to be 10.1.

this runtime check will only happen once at startup.

Comment 26

15 years ago
Comment on attachment 136390 [details] [diff] [review]
v2.1 patch

r=wtc.	I like this patch!
Attachment #136390 - Flags: review?(wchang0222) → review+


15 years ago
Target Milestone: --- → 4.5

Comment 27

15 years ago
Comment on attachment 136390 [details] [diff] [review]
v2.1 patch

would be good to get this in for beta.
Attachment #136390 - Flags: approval1.6b?

Comment 28

15 years ago
fixed on NSPR trunk and NSPRPUB_4_2_CLIENT_BRANCH.  this fix will appear in
mozilla 1.6 beta.
Last Resolved: 15 years ago15 years ago
Resolution: --- → FIXED

Comment 29

15 years ago
sadly, i'm on 10.3 now so this issue doesn't bug me personally anymore, but
thank you for fixing this! i think all the 10.2.x people will be eternally
grateful and it is the right thing to do anyway.

another satisified customer. =)

Comment 30

15 years ago
VERIFIED: fixed.
Added a entry to /etc/hosts.
Correct behavior would be to block hostname.

Mozilla 1.6a - incorrect behavior.
Mozilla 1.6f - correct behavior.
You need to log in before you can comment on or make changes to this bug.