1439780 - .local (avahi) domains result in "Hmm, we're having trouble finding that site."

Status: RESOLVED INVALID

(Core :: Networking, defect, P2)

Description

[toonn]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180216213452

Steps to reproduce:

Enter "host".local in the urlbar.


Actual results:

"Hmm, we're having trouble finding that site." page was loaded.


Expected results:

The site hosted at the local domain should have loaded.
Opera opens the site (only with http:// prefix, without it a google query is executed).
My current version is Mozilla Firefox 58.0.2, this only stopped working after updating from Mozilla Firefox 56.0.2.

Comment 1

[Gingerbread Man]

(In reply to toonn from comment #0)
> My current version is Mozilla Firefox 58.0.2, this only stopped working
> after updating from Mozilla Firefox 56.0.2.

It would help if you could find the regression range.
mozregression --good 56 --bad 58
https://mozilla.github.io/mozregression/documentation/usage.html

Comment 2

[toonn]

I specified 56 and 58 and the tool installed and tested nightly 57 and 59 respectively, both worked fine. My system firefox (58.0.2) still can't handle the .local domain.

Here's the result:

(py2env) $ mozregression --good 56 --bad 58
**********
You should use a config file. Please use the --write-config command line flag to help you create one.
**********

 0:01.34 INFO: Using date 2017-11-13 for release 58
 0:01.34 INFO: Using date 2017-08-02 for release 56
 0:10.74 INFO: Testing good and bad builds to ensure that they are really good and bad...
 0:10.74 INFO: Downloading build from: https://archive.mozilla.org/pub/firefox/nightly/2017/08/2017-08-02-10-03-02-mozilla-central/firefox-57.0a1.en-US.linux-x86_64.tar.bz2
===== Downloaded 100% =====
 0:29.53 INFO: Running mozilla-central build for 2017-08-02
 0:56.75 INFO: Launching /tmp/tmp_gX0jo/firefox/firefox
 0:56.75 INFO: Application command: /tmp/tmp_gX0jo/firefox/firefox -profile /tmp/tmp6kDREW.mozrunner
 0:56.77 INFO: application_buildid: 20170802100302
 0:56.77 INFO: application_changeset: 52285ea5e54c73d3ed824544cef2ee3f195f05e6
 0:56.77 INFO: application_name: Firefox
 0:56.77 INFO: application_repository: https://hg.mozilla.org/mozilla-central
 0:56.77 INFO: application_version: 57.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry' or 'exit' and press Enter): good
 1:29.12 INFO: Using local file: /tmp/tmppzEuM5/2017-11-13--mozilla-central--firefox-59.0a1.en-US.linux-x86_64.tar.bz2 (downloaded in background)
 1:29.12 INFO: Running mozilla-central build for 2017-11-13
 1:54.65 INFO: Launching /tmp/tmpGe_KiO/firefox/firefox
 1:54.65 INFO: Application command: /tmp/tmpGe_KiO/firefox/firefox -profile /tmp/tmpkGEtTG.mozrunner
 1:54.66 INFO: application_buildid: 20171113220112
 1:54.66 INFO: application_changeset: c616a6fd5e4b20cca139fcdd3957682afaa862b9
 1:54.66 INFO: application_name: Firefox
 1:54.66 INFO: application_repository: https://hg.mozilla.org/mozilla-central
 1:54.68 INFO: application_version: 59.0a1
Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry' or 'exit' and press Enter): good
 2:13.85 ERROR: Build was expected to be bad! The initial good/bad range seems incorrect.

Comment 3

[Valentin Gosu [:valentin] (he/him)]

You indicated to mozregressions that firefox 58 is good.
Either 58.0.2 is different from 58.0, or the problem is with your profile.

Could you try a few more things?
First of all, see if you can reproduce this with your addons disabled [1]
Run firefox -P and create a new profile [2]. Check if you can still reproduce the bug.
If that doesn't work backup your profile (switching to one version then back to an old one may corrupt it sometimes) and download the latest nightly from http://nightly.mozilla.org/ and try it, with your old profile, and with the new one.

[1] https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode
[2] https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

Comment 4

[toonn]

First tried disabling all addons, removed one legacy addon that didn't work anymore anyway, can still reproduce it.

Then created a new profile, still reproduces.

Does testing nightly still make sense? mozregression had me test nightly 59 and it worked fine so I expect nightly to just work.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

Not sure why/how this happens for you. I just tried it on 58 on my machine, and I can load http://raspberrypi.local/ just fine. Same in Firefox 60.

(In reply to toonn from comment #4)
> First tried disabling all addons, removed one legacy addon that didn't work
> anymore anyway, can still reproduce it.
> 
> Then created a new profile, still reproduces.
> 
> Does testing nightly still make sense? mozregression had me test nightly 59
> and it worked fine so I expect nightly to just work.

Yes. Please test. For some reason, your installed firefox doesn't work, but the one from mozregression does.
So we will need a few more details. Could you send us the details from about:support ?

Comment 6

[toonn]

Here's the about:support data: http://ix.io/NWG
I installed this firefox with nix, not sure if that matters.

Comment 7

[Valentin Gosu [:valentin] (he/him)]

(In reply to toonn from comment #6)
> Here's the about:support data: http://ix.io/NWG
> I installed this firefox with nix, not sure if that matters.

That's possible. You could try the download from https://www.mozilla.org/en-US/firefox/new/
If one works and the other doesn't, with the same profile, it means there is a problem with the nix packaged version.

Comment 8

[toonn]

First tried the firefox from the arch repos, because it was easier to get. Then tried the downloaded version you linked. Same issue in both.

Comment 9

[toonn]

Tried firefox-devedition installed with nix (Mozilla Firefox 59.0): same issue.
Then finally tried firefox nightly downloaded from mozilla.org (Mozilla Firefox 60.0a1): .local domains resolve properly.

Comment 10

[u408661]

Could you reproduce with http logging enabled (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)? (Perhaps also do the same but with a working browser and attach both logs.) That could help us track down the source of the issue.

Thanks.

Comment 11

[toonn]

Attachment: HTTP logging: log.txt-main.974

Last time I tested nightly I messed up my profile somewhat, still not sure if it's completely fixed now : s
I'm ok with testing nightly again to provide a log for a working version but I'd need some advice on how to do that without messing up my profile?

It generated 4 child logs I could provide those as well if necessary.
I attached log.txt-main.974 generated with ff58.0.2 (not working) to the bug report.

Comment 12

[Gingerbread Man]

(In reply to toonn from comment #11)
> Last time I tested nightly I messed up my profile somewhat, still not sure
> if it's completely fixed now : s
> I'm ok with testing nightly again to provide a log for a working version but
> I'd need some advice on how to do that without messing up my profile?

If for some reason you need to test your regular profile, it's simplest to create a new profile, then copy the contents of your regular profile into it while Firefox is closed. Then launch Nightly with your duplicate profile, using either the profile manager or the command
firefox -p ProfileName -no-remote

That being said, as indicated at comment 3, you were supposed to create a backup copy of your profile folder. You were warned that running a newer version of Firefox (60 i.e. Nightly), then an older one (Firefox 58), will corrupt the profile:

(In reply to Valentin Gosu [:valentin] from comment #3)
> If that doesn't work backup your profile (switching to one version then back
> to an old one may corrupt it sometimes) and download the latest nightly from
> http://nightly.mozilla.org/ and try it, with your old profile, and with the
> new one.
> 
> [2] https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

Comment 13

[u408661]

From the log:
> 2018-02-28 09:09:32.648203 UTC - [Socket Thread]: D/nsHostResolver Resolving host [selfoss.local].
> 2018-02-28 09:09:32.648215 UTC - [Socket Thread]: D/nsHostResolver   No usable address in cache for host [selfoss.local].
> 2018-02-28 09:09:32.648227 UTC - [Socket Thread]: D/nsHostResolver   DNS thread counters: total=2 any-live=0 idle=2 pending=1
> 2018-02-28 09:09:32.648235 UTC - [Socket Thread]: D/nsHostResolver   DNS lookup for host [selfoss.local] blocking pending 'getaddrinfo' query: callback [0x7f5e34fd0af0]
> 2018-02-28 09:09:32.648252 UTC - [DNS Resolver #18]: D/nsHostResolver DNS lookup thread - Calling getaddrinfo for host [selfoss.local].
> 2018-02-28 09:09:32.765400 UTC - [DNS Resolver #18]: D/nsHostResolver Calling 'res_ninit'.
> 2018-02-28 09:09:32.828027 UTC - [DNS Resolver #18]: D/nsHostResolver DNS lookup thread - lookup completed for host [selfoss.local]: failure: unknown host.
> 2018-02-28 09:09:32.828052 UTC - [DNS Resolver #18]: D/nsHostResolver nsHostResolver record 0x7f5e390e9f30 new gencnt
> 2018-02-28 09:09:32.828065 UTC - [DNS Resolver #18]: D/nsHostResolver Caching host [selfoss.local] negative record for 60 seconds.

Looks to me like getaddrinfo is failing.

Reporter: am I correct in assuming this is the nix installation this log comes from? If so, could you also post a log from a functioning instance for comparison? I'm curious if there's anything obviously different in the logs (beyond the getaddrinfo failure). (BTW, no need for the child logs, all networking happens on the parent process.)

Comment 14

[toonn]

I'm perfectly willing to provide logs from the working nightly but I don't want to fubar my profile again. Will just copying my profile directory and moving it back into place afterwards prevent all the weirdness? Last time after removing every other version of firefox the about window still said nightly 57 and firefox forgot and refused to remember my theme and customizations.

Comment 15

[Valentin Gosu [:valentin] (he/him)]

(In reply to toonn from comment #14)
> I'm perfectly willing to provide logs from the working nightly but I don't
> want to fubar my profile again. Will just copying my profile directory and
> moving it back into place afterwards prevent all the weirdness?

Yes, that should work.

Comment 16

[toonn]

Attachment: HTTP Logging log.txt-main.28580

This attachment is the log for firefox nightly 60.0a1 (2018-03-01) (64-bit).
I turned on logging in about:networking, opened a new window, entered the address (actually a keyword for a bookmark), the page loaded fine as I'd expect from any version, closed the window, stopped the logging.

Comment 17

[Honza Bambas (:mayhemer)]

Could 

2018-02-28 09:09:32.765400 UTC - [DNS Resolver #18]: D/nsHostResolver Calling 'res_ninit'.

be the cause here?  It's something I don't see in the "good" log from comment 16, at all.  Would also support the theory the failure comes from a different build package.

Comment 18

[Michal Novotny [:michal]]

Attachment: hook_getaddrinfo.c

I've created a simple hook for getaddrinfo to verify whether getaddrinfo is called and what it returns.

Compile it by running:

$ gcc -shared -ldl -fPIC hook_getaddrinfo.c -o hook_getaddrinfo.so

and then run firefox this way:

$ LD_PRELOAD=/path/to/so_file/hook_getaddrinfo.so ./firefox

Comment 19

[toonn]

(In reply to Michal Novotny (:michal) from comment #18)
> and then run firefox this way:
> 
> $ LD_PRELOAD=/path/to/so_file/hook_getaddrinfo.so ./firefox

I tried to do this, using the firefox on my path, nothing seemed to change, am I supposed to share logs again?

I'm not sure which firefox you wanted me to try with because you used ./firefox, regular installed by nix, nightly?

Comment 20

[Michal Novotny [:michal]]

This shouldn't change anything. Preloaded hook_getaddrinfo logs all getaddrinfo calls to console. You should run this with firefox where you see the problem. Then in the console you should see request for selfoss.local and the result that getaddrinfo returns to firefox.

Comment 21

[toonn]

First time I ran it with a relative path to the so file, these errors:
> ERROR: ld.so: object 'hook_getaddrinfo.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
> ERROR: ld.so: object 'hook_getaddrinfo.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
> ERROR: ld.so: object 'hook_getaddrinfo.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
> 
> (firefox:1368): Gtk-WARNING **: GTK+ module /nix/store/bzisdwddl278cj0wvv3pwryl8xcszq37-libcanberra-0.30/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
> GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.
> Gtk-Message: Failed to load module "canberra-gtk-module"

Then I used the full path:
> 
> (firefox:1403): Gtk-WARNING **: GTK+ module /nix/store/bzisdwddl278cj0wvv3pwryl8xcszq37-libcanberra-0.30/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
> GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.
> Gtk-Message: Failed to load module "canberra-gtk-module"

This is exactly the same output I get when I just run firefox, except the number changes 1368, 1403, 1466, so I think that's just the PID.
I get this gtk-message with most gtk programs installed through nix but the gtk-warning is unique to firefox. However I think it's unrelated since gtk shouldn't have anything to do with dns resolution.

Both times I tried to visit the local domain, nothing got printed to the console.

Comment 22

[Michal Novotny [:michal]]

Attachment: hook2.c

It looks like we might end up calling gethostbyname or gethostbyname_r if nspr is compiled without getaddrinfo support. Please try this new code. You obviously need to specify full path to the preloaded shared lib.

Do I understand correctly that all versions downloaded from mozilla.org work correctly and all versions downloaded with nix have the problem?

It would be good to have logs from console from both working and non-working versions.

Comment 23

[toonn]

Note that I said in comment 8 that 58 downloaded from mozilla didn't work either. It was only nightly that worked. I actually think nix doesn't alter the firefox binary in any way, otherwise they wouldn't use the firefox branding. They don't include the branding for thunderbird by default because they alter it in some way (probably just patching some library paths).

Comment 24

[toonn]

Attachment: Log for getaddrinfo

This is the log for my installed firefox (58.0.2), note that I compiled `hook2.c` with the name `hook_getaddrinfo2.so`.
Noticed ff59 was released, should I update or do you want me to keep testing this version?

Comment 25

[Daniel Stenberg [:bagder]]

Just because it hasn't been properly spelled out previously in this bug, I'd like to add...

Firefox does nothing special for resolving ".local" domain names. It resolves them *exactly* the same way it resolves every other name[*] so when a name like this has a problem, it is something in the fundamental level. This is how Firefox has resolved names for a very long time.

I just tried Firefox 52 (ESR), 59 and current Nightly 61 (on Debian unstable) and they all resolve my ".local" host names fine.  This makes me believe that we need something else done locally as well to trigger this problem.

@toonn: I think you can continue testing on 59. It should work the same way...

[*] = with the exception of TRR, but that's not involved here

Comment 26

[Michal Novotny [:michal]]

Attachment: hook3.c

(In reply to toonn from comment #23)
> Note that I said in comment 8 that 58 downloaded from mozilla didn't work
> either. It was only nightly that worked. I actually think nix doesn't alter
> the firefox binary in any way, otherwise they wouldn't use the firefox
> branding. They don't include the branding for thunderbird by default because
> they alter it in some way (probably just patching some library paths).

What I don't understand is that nothing was logged with the previous attempt (comment #21). The newer version of the function hooks didn't change anything for getaddrinfo and in this log it's logged. Did you run different version of firefox?

In this log I can see that getaddrinfo fails with error EAI_SYSTEM. Attached is another version that logs more input arguments and errno. It would be good to find out what's the difference between 58 and nightly. Please run this test with failing as well as working version of firefox and provide logs.

Then have a look at https://wiki.archlinux.org/index.php/avahi. Do you have mdns_minimal in nsswitch.conf? If yes, is there any change if you use mdns4_minimal instead?

Comment 27

[toonn]

Attachment: Creation of a testing profile (to show the gtk-warnings without the hooks)

Comment 28

[toonn]

Attachment: First hook with 58

Comment 29

[toonn]

Attachment: 2nd hook with 58

Comment 30

[toonn]

Attachment: 3rd hook with 58

Comment 31

[toonn]

Attachment: First hook with nightly

Comment 32

[toonn]

Attachment: 2nd hook with nightly

Comment 33

[toonn]

Attachment: 3rd hook with nightly

Comment 34

[toonn]

Attachment: Output of --version for good measure

Comment #26 made me retry the first hook, which did output logs but only if firefox wasn't already running when I ran the command. Not sure why I tried that with the second hook but not the first, I remember I created a new profile for testing the second hook so maybe I closed firefox to do that and didn't reopen.

I've attached a log for the creation of a testing profile, just to show the gtk-warnings that are likely unrelated to the problem.
Then there's logs with all three hooks for the two versions and in this latest attachment the output of --version for the firefox on my path (58 installed with nix) and firefox nightly downloaded from mozilla.

Also, to come back on something I said earlier. Nix does distribute the mozilla binary for firefox as a package named firefox-bin but the "firefox" package I installed is built from the "Firefox source tree." It used to be that the firefox installed with that package was branded nightly because of trademarks.

Comment 35

[Michal Novotny [:michal]]

(In reply to toonn from comment #34)
> Comment #26 made me retry the first hook, which did output logs but only if
> firefox wasn't already running when I ran the command.

OK, that makes sense because it attached to already running process and opened a new window. You can use parameters --no-remote or --new-instance to avoid that.

Anyway, the logs show that we call getaddrinfo in the same way in both cases:

getaddrinfo node=selfoss.local, service=(null), hints=0x7fadeac71d90, retval=-11
  hints.ai_family=0
  hints.ai_socktype=1
  hints.ai_protocol=0
  hints.ai_flags=32
  errno=16

getaddrinfo node=selfoss.local, service=(null), hints=0x7fdfb8268d50, retval=0
  hints.ai_family=0
  hints.ai_socktype=1
  hints.ai_protocol=0
  hints.ai_flags=32
  IPv6 address: 2a02:1810:4405:7100:ac3b:a0ff:fefc:a657
  IPv4 address: 192.168.0.244

Looks like error 16 might come from https://github.com/bminor/glibc/blob/master/sysdeps/posix/getaddrinfo.c#L942
I have really no idea what's going on here :-/

Comment 36

[Michal Novotny [:michal]]

(In reply to Michal Novotny (:michal) from comment #35)
> Looks like error 16 might come from
> https://github.com/bminor/glibc/blob/master/sysdeps/posix/getaddrinfo.c#L942

If getinfoaddr really fails here the multicast DNS request would not be probably sent. You can verify it by watching tcpdump's output while trying to load http://selfoss.local

$ sudo tcpdump -n -i any udp and port 5353

Comment 37

[toonn]

Attachment: tcpdump while starting nightly

I got no output with ff58, either just navigating to the url or starting it up. I got some output while I started up nightly but not when I navigated to the url.
I think maybe this is because it was included in favorites/top sites on the new tab page and was then cached from that lookup?

Comment 38

[Michal Novotny [:michal]]

This is a problem of nix libraries. I installed nix on my computer and I see the same problem. No program installed within nix can resolve names using mDNS. E.g. install telnet using nix and try to connect to you machine, it'll fail with "Resolver internal error". Output of strace shows that libnss_mdns4_minimal.so.2 cannot be find in the search path:

openat(AT_FDCWD, "tls/haswell/x86_64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/haswell/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/x86_64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "haswell/x86_64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "haswell/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "x86_64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/84h2zni7h805k0i1ys2bba3dsp1cqnhh-glibc-2.26-131/lib/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

The reason why you thought you were able to reproduce it with version 58 downloaded from mozilla.org is that you made the mistake described in comment #35. You didn't use --no-remote parameter and only new window of already running firefox from nix environment was opened.

Comment 39

[toonn]

:michal thank you. Didn't think of that.
I'll take the bug to one of the nixos issue trackers.