491543 - make lookup of autoconfig domains smart enough to detect google apps hosted domains

Status: RESOLVED DUPLICATE of bug 551519

(Mozilla Messaging Graveyard :: Server Operations, defect)

Description

[David Ascher (:davida)]

The autoconfig code fails for domains which are hosted by Google Apps for Your Domain.

The client code doesn't have access to DNS to determine whether a domain's MX maps to the google servers, so can't figure it out.

This bug is about making the code that serves autoconfig XML files, first look in the static DB for configs, but if no config is found, to look in the MX record to see if it should return a config that corresponds to the right google servers, etc.

I don't know the details of the DNS config that we're looking for, or the details of the autoconfig files.

This probably needs security review as well.

Comment 1

[Ben Bucksch (:BenB)]

I think the server serving the XML files should stay dumb. It should be a general service, which can be used by other mail clients or completely other consumers (MailServerList is explicitly free for any use of any kind). They may want to make other choices.

I'd always prefer to keep the logic in the client.

The client is, however, currently incapable of making an MX lookup (Mozilla simply has no DNS code IIRC, but relies on OS). Can we make a web service that just does an MX lookup for us? We give it a domain name, it looks up the MX, and returns that (just that)? The client then makes another query to the DB for the domain found.
I can implement that. All I'd need is the webservice to make the MX lookup.

Comment 2

[Ben Bucksch (:BenB)]

If we do the original proposal, please make it optional (e.g. smart=true or whatever).

Comment 3

[David Ascher (:davida)]

As per IRC, I think putting this logic in the server is going to be harder to maintain over time, esp. if e.g. google changes their model, or if we want to add eventual new google-apps-like providers.

Comment 4

[Ben Bucksch (:BenB)]

I think davida meant "putting this logic in the *client* is going to be harder". David A. prefers the server model.
In this case, please see comment 2 (make it optional).

Comment 5

[Philippe M. Chiasson (:gozer)]

(In reply to comment #1)
> I think the server serving the XML files should stay dumb. It should be a
> general service, which can be used by other mail clients or completely other
> consumers (MailServerList is explicitly free for any use of any kind). They may
> want to make other choices.
> 
> I'd always prefer to keep the logic in the client.

I'd agree here too.

> The client is, however, currently incapable of making an MX lookup (Mozilla
> simply has no DNS code IIRC, but relies on OS). Can we make a web service that
> just does an MX lookup for us?

That would be very, very simple.

> We give it a domain name, it looks up the MX,
> and returns that (just that)? The client then makes another query to the DB for
> the domain found.
> I can implement that. All I'd need is the webservice to make the MX lookup.

I've thrown something together here:

http://ectoplasm.org/bin/mx/[domain]

i.e.:

<http://ectoplasm.org/bin/mx/mozilla.org>

Comment 7

[Ben Bucksch (:BenB)]

I think this is a dup of bug 551519. That bug would solve this one and many other cases, too.

Comment 8

[Philippe M. Chiasson (:gozer)]

I've tweaked a little the code the FastMail.fm folks wrote, and turned it on as a 404 handler for /autoconfig/ requests. So it kicks in *only* for domains we don't know about, and only returns data for google apps / fastmail.fm domains.

Comment 9

[Ben Bucksch (:BenB)]

Wait a second, please. I was spending most of last week to do this (DNS MX) in the client, as discussed with bwinton some time ago, specifically to avoid this. As said, I'd like the DB to be a pure DB.

Comment 10

[Philippe M. Chiasson (:gozer)]

(In reply to comment #9)
> Wait a second, please. I was spending most of last week to do this (DNS MX) in
> the client, as discussed with bwinton some time ago, specifically to avoid
> this. As said, I'd like the DB to be a pure DB.

Same here, I 100% think that having the client be able to DNS MX (and SRV/TXT) lookups on it's own is a very good thing. 

Closing this bug might have been premature, but it's because it just dawned on me (with a poke by davida) that making the server-side be able to detect google apps domains was literally just a few lines of code on my part.

So for now, autoconfig requests for these domains will start working for our users, and possibly, once we have a smarter client, I can even disable that from the server-side and revert back to a better, dumber ISP database.

Makes sense ?

Comment 11

[Ben Bucksch (:BenB)]

Well, from one perspective, but it doesn't solve my problem.
I'd need a) a clear and definite date/point (in the near future) when this is going to be turned off again and b) a way to avoid this and get just the data.

Please note that you'll now make a DNS lookup for 66% of the domains (all misses).
Also, Fastmail have (or are currently rolling out) an autoconfig server now, so it's not needed there.

As said, I invested a whole week of work in this client-side DNS MX lookup, because it was important for me to keep the design as an open, simple DB. It's not particularly encouraging.

Comment 12

[David :Bienvenu]

Ben, isn't your DNS code in gecko? I wouldn't bet the farm on getting that into 1.9.2 before we ship 3.1, so I think have a server-side fallback until we get client support is a reasonable thing to do, especially if it only takes a little bit of gozer's time.

Comment 13

[David Ascher (:davida)]

Agree w/ comment #12 -- I don't see a big downside in not doing this in the server _until_ it _ships_ in a main release from my POV.  

I don't understand why we would care about making extra DNS queries given the scale of this.

Making 3% of all account configs Just Work is high value IMO.

--da

Comment 14

[Ludovic Hirlimann [:Usul]]

So I tested it with my domain this morning and with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.4pre) Gecko/20100413 Lanikai/3.1b2pre.

It doesn't really work :
 It detects pop.hirlimann.net unsecure and mail.hirlimann.net on port 25. 

Instead of imap.google... and smtp.goo....

Plus :
Work-2:~ ludo$ ping mail.hirlimann.net
ping: cannot resolve mail.hirlimann.net: Unknown host
Work-2:~ ludo$ ping pop.hirlimann.net
ping: cannot resolve pop.hirlimann.net: Unknown host

Error might be in the code because I'm getting these in the -console when trying  (that's why I'm adding blake to the bug).:

-- Exception object --
+ _message (string) 'Supplied value not in allowed list'
+ stack (string) 2964 chars
+ message (string) 'Supplied value not in allowed list'
+ toString (function) 3 lines
*
-- Stack Trace --
Exception("Supplied value not in allowed list")@chrome://messenger/content/accountcreation/util.js:179
MalformedException("allowed_value.error",null)@chrome://messenger/content/accountcreation/sanitizeDatatypes.js:231
(null,[object Object],-1)@chrome://messenger/content/accountcreation/sanitizeDatatypes.js:216
("hirlimann.net",false,null,null,0)@chrome://messenger/content/accountcreation/guessConfig.js:427
guessConfig("hirlimann.net",(function (type, hostname, port, ssl, done, config) {gEmailWizardLogger.info("progress callback host " + hostname + " port " + port + " type " + type);if (type == "imap" || type == "pop3") {config.incoming.type = type;config.incoming.hostname = hostname;config.incoming.port = port;config.incoming.socketType = ssl;config.incoming._inprogress = !done;} else if (type == "smtp" && !me._userPickedOutgoingServer) {config.outgoing.hostname = hostname;config.outgoing.port = port;config.outgoing.socketType = ssl;config.outgoing._inprogress = !done;}me.updateConfig(config);}),(function (config) {me.foundConfig(config);gEmailWizardLogger.info("in success, incomingState = " + me._incomingState + " outgoingState = " + me._outgoingState);if (me._incomingState == "done" && me._outgoingState == "done") {me.stopSpinner("found_settings_guess");_hide("stop_button");_show("edit_button");} else if (me._incomingState == "done" && me._outgoingState != "probing") {if (me._outgoingState == "failed") {me.stopSpinner("failed_to_find_settings");} else {me.stopSpinner("found_settings_guess");}me.editConfigDetails();} else if (me._outgoingState == "done" && me._incomingState != "probing") {if (me._incomingState == "failed") {me.stopSpinner("failed_to_find_settings");} else {me.stopSpinner("found_settings_guess");}me.editConfigDetails();}if (me._outgoingState != "probing" && me._incomingState != "probing") {me._probeAbortable = null;}}),(function (e, config) {gEmailWizardLogger.info("guessConfig failed: " + e);me.updateConfig(config);me.stopSpinner("failed_to_find_settings");me._probeAbortable = null;me.editConfigDetails();}),(function (e, config) {gEmailWizardLogger.info("guessConfig failed for incoming: " + e);me._setIconAndTooltip("incoming", "failed", "");me._incomingState = "failed";config.incoming.hostname = -1;me.updateConfig(config);}),(function (e, config) {gEmailWizardLogger.info("guessConfig failed for outgoing: " + e);me._setIconAndTooltip("outgoing", "failed", "");me._outgoingState = "failed";if (!me._userPickedOutgoingServer) {config.outgoing.hostname = -1;}me.updateConfig(config);}),[object Object],"both")@chrome://messenger/content/accountcreation/guessConfig.js:223
("hirlimann.net",[object Object],"both")@chrome://messenger/content/accountcreation/emailWizard.js:554
([object Object])@chrome://messenger/content/accountcreation/emailWizard.js:463
(false)@chrome://messenger/content/accountcreation/fetchhttp.js:213
([object ProgressEvent])@chrome://messenger/content/accountcreation/fetchhttp.js:145

I have a test account if you guys need one.

Comment 15

[Ben Bucksch (:BenB)]

OK, I investigated, and there are several bugs here:
1. The exception you see is irrelevant. It's coming from guessConfig, as can
   be seen in the stack, which we fall into when we can't get any or no
   proper config from the ISPDB.
   I have fixed this in bug 549045, IIRC, so I don't feel like debugging that
   on 3.1.
2. the client is not actually getting a config back from the ISBDB for
   hirlimann.net . I see "fetchConfigFromDB: Not found" on the console when
   I turn on logging (see prefs, search for "wizard").
   However, https://live.mozillamessaging.com/autoconfig/hirlimann.net
   does return the Google config, so I don't know what's up. That's what
   prevents it from working.
3. Actually, also
   https://live.mozillamessaging.com/autoconfig/nonsense/hirlimann.net
   returns the config, so does .../v1.0/... and .../v1.2/... It shouldn't.
4. The returned config does not include the queried domain. We currently
   don't enforce that, but it's invalid per spec and way may enforce it.

Comment 16

[Ben Bucksch (:BenB)]

> 2. the client is not actually getting a config back from the ISBDB for
>    hirlimann.net . I see "fetchConfigFromDB: Not found" on the console when
>    I turn on logging (see prefs, search for "wizard").
>    However, https://live.mozillamessaging.com/autoconfig/hirlimann.net
>    does return the Google config, so I don't know what's up. That's what
>    prevents it from working.

Ah, found it. The ISPDB returns error code 404, with the config as error page.
The client sees error code 404 and understands "not found", and considers it an error.

May I ask: why has this been commited,
- with ignoring my concerns, given that I designed and initiated this system
  and database, and ignoring the fact that I was working on client-side
  code particularly to support this
- without any review, although we require *two* reviews for every stupid
  ISP config file change. This was more risky than a change of the static
  foobar.de XML file.
- not even considering the fact that the DNS lookups made from the server
  leak information about domains queried by clients (and I can't even see
  whether there are any other security problems, because the source is not
  attached)
- no testing against the client at all
?

Comment 17

[Ben Bucksch (:BenB)]

Ping? Could you please
- disable this for now
- attach the source
At the minimum, we need to fix the return code and remove the fastmail stuff (given that they have an autoconfig server).

Comment 18

[David Ascher (:davida)]

I think it'd be really good to keep doing this centrally until you have the client-side version deployed in a production build.  That said, we should fix the bugs!

Comment 19

[Philippe M. Chiasson (:gozer)]

(In reply to comment #16)
> > 2. the client is not actually getting a config back from the ISBDB for
> >    hirlimann.net . I see "fetchConfigFromDB: Not found" on the console when
> >    I turn on logging (see prefs, search for "wizard").
> >    However, https://live.mozillamessaging.com/autoconfig/hirlimann.net
> >    does return the Google config, so I don't know what's up. That's what
> >    prevents it from working.
> 
> Ah, found it. The ISPDB returns error code 404, with the config as error page.
> The client sees error code 404 and understands "not found", and considers it an error.

Yes, and that's a bug in how I duct-taped the lookup to happen only when encountering a 404 looking for the static XML files.

> May I ask: why has this been commited,
> - with ignoring my concerns, given that I designed and initiated this system
>   and database, and ignoring the fact that I was working on client-side
>   code particularly to support this

Not by ill intent, that's for sure. I just suddently saw a very simple/easy way to plug the code the fastmal guys shipped us a while back into the existing ispdb lookup that litterally took me all of 5 minutes. And since a few % or our ispdb hits fail on google hosted domains, figured it was a quick solution to make them happy.

> - without any review, although we require *two* reviews for every stupid
>   ISP config file change. This was more risky than a change of the static
>   foobar.de XML file.

Yes, I definitely jumped the gun here and should have obtained feedback before moving forward.

> - not even considering the fact that the DNS lookups made from the server
>   leak information about domains queried by clients (and I can't even see
>   whether there are any other security problems, because the source is not
>   attached)

Source was submitted by the fastmail guys in a bug I can't quite find at the moment, but the code itself lives here:

[svn.mozilla.org]/mozillamessaging.com/sites/ispdb.mozillamessaging.com/trunk/tools/MozAutoConfig.pm

And has been live at

https://live.mozillamessaging.com/autoconfig-dyn-test/hirlimann.net

for a while now

> 3. Actually, also
>   https://live.mozillamessaging.com/autoconfig/nonsense/hirlimann.net
>   returns the config, so does .../v1.0/... and .../v1.2/... It shouldn't.

Another bug in MozAutoConfig.pm, it's a bit too generous in how it gets the domain from the url, basically last path element: wrong.

> 4. The returned config does not include the queried domain. We currently
>   don't enforce that, but it's invalid per spec and way may enforce it.

Ah, a good point too, needs fixing as well.

> Ping? Could you please
> - disable this for now

Okay, let's disable this for now, at least until the current implementation bugs are fixed. Then we can discuss what to do with it again.

> - attach the source

Source is in svn.

Comment 20

[Ben Bucksch (:BenB)]

> Not by ill intent, that's for sure. ...
> Yes, I definitely jumped the gun here and should have obtained feedback
> before moving forward.

Apology appreciated and accepted. :-)

> > - attach the source
> Source is in svn.

At svn://svn.mozilla.org/mozillamessaging.com/sites/ispdb.mozillamessaging.com/trunk/tools/MozAutoConfig.pm I assume. Thanks.


> that's a bug in how I duct-taped the lookup to happen only when
> encountering a 404 looking for the static XML files.

Could you please remove the duct tape until this is at least taped on properly? Because it's currently not helping at all, the client entirely ignores it.

Comment 21

[Ben Bucksch (:BenB)]

Bug 491543 fixes this.

Comment 22

[Ben Bucksch (:BenB)]

eh, I mean: Bug 551519 fixes this.

Comment 23

[Ben Bucksch (:BenB)]

Once the tree opens and I can commit bug 551519, this is fixed (in TB 3.1).
I tested again [e-mail] and it works nicely.

gozer, could you now please remove the changes made to the server here, given that they don't work anyways and are superceded? Once you've done that, we can close this as dup. Thanks!

Comment 24

[Philippe M. Chiasson (:gozer)]

(In reply to comment #23)
> Once the tree opens and I can commit bug 551519, this is fixed (in TB 3.1).
> I tested again [e-mail] and it works nicely.
> 
> gozer, could you now please remove the changes made to the server here, given
> that they don't work anyways and are superceded? 

If you mean the automatic ispdb fallback on the DNS based method, that's been disabled for a while now.

Comment 25

[Ben Bucksch (:BenB)]

Ah, great, thanks.
I'll mark this a dup, then. Bug 551519 together with bug 561918 fixes this.

_tsk_, please verify once bug 551519 is commited that the Google Apps domains you know of work.

Thanks all!