1662433 - MCD autoconfig disfunctional for Thunderbird 78, affecting enterprise deployment (function processLDAPValues(values) does not return any values in Thunderbird 68.3.1+)

Status: RESOLVED FIXED

(MailNews Core :: LDAP Integration, defect, P1)

Description

[marinar]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36 Edg/85.0.564.41

Steps to reproduce:

In Thunderbird 68 and earlier we use MCD autoconfig to read user information for LDAP
..Thunderbird\Default\pref\autoconfig.js points to autoconfiguration script that uses
function processLDAPValues (values) to read users attributes.
Starting Thunderbird 69 the MCD script does not return values.
Is any way to fix MCD or any other way to auto setup users profiles and preferences?

Actual results:

MCD script and ProcessLDAPValues function does not read user information. Sets LDAP address books, and other preferences.

Expected results:

MCD autoconfig script reads user information from LDAP and sets user account and profile preferences.

Comment 1

[marinar]

Does anybody has any latest information how to make MCD autoconfig work or any other information how to setup user profile/account in Enterprise environment.
Any sample scripts that work with Thunderbird 78.

Thank you

Comment 2

[Magnus Melin [:mkmelin]]

https://mail.mozilla.org/pipermail/tb-enterprise/2019-October/001604.html ?

Comment 3

[marinar]

In TB 68 we did disable Sandbox to make autoconfig script work.

This is C:\Program Files (x86)\Mozilla Thunderbird\defaults\pref\autoconfig.js
//Pointer to Autoconfiguration script
pref('general.config.obscure_value',0);
pref('general.config.filename', 'Mozilla.cfg');
pref('general.config.vendor', 'Mozilla');
pref('general.config.sandbox_enabled', false);

Any other new features in TB 69 and later, block MCD autoconfig scripts?

Comment 4

[Thomas D. (:thomas8)]

Hmmm, the document on MDN presents MCD (Mission Control Desktop) in "Archive of obsolete content"... ?

https://developer.mozilla.org/en-US/docs/Archive/Misc_top_level/MCD,_Mission_Control_Desktop_AKA_AutoConfig

(more via matrix-PM to Wayne today)

Comment 5

[Mike Kaply [:mkaply]]

I'm very confused. I have people that said it didn't work in 68. You're saying it broke between 68 and 78?

Comment 6

[Mike Kaply [:mkaply]]

Moving to LDAP. Not an Autoconfig issue.

Comment 7

[Kai Engert [:KaiE:]]

If LDAP is working correctly, then this isn't an LDAP issue.
It was necessary to change LDAP, as described in the other bug, because of platform changes.

If that change caused something else to fail - then that other code needs to be adjusted, IIUC.

I don't know the autoconfig code, so I'm purely speculating. Could the autoconfig code currently assume that a call will synchronously return data, which it doesn't?

Then you need to track down which code that is, and change it to wait for results asynchronously - potentially this requires introducing some new mechanism that calls a callback - once data is available.

Comment 8

[Mike Kaply [:mkaply]]

The code is here:

https://searchfox.org/mozilla-central/source/extensions/pref/autoconfig/src/prefcalls.js#157

It's calling

https://searchfox.org/comm-central/source/ldap/xpcom/public/nsILDAPSyncQuery.idl#24

which is a sync function by design.

Comment 10

[Thomas D. (:thomas8)]

This should probably be fixed as a matter of priority, as it prevents large organisational/enterprise consumers with thousands of users from deploying/using Thunderbird.

• Magnus, should this be assigned to someone?
• Ben and Kai: Per Mike's analysis (summarized in Bug 1592922 Comment 37), after he has fixed the MCD API part in bug 1592922, this is now a regression caused by our bug 1576364, which is said to break LDAP in autoconfig. Do you agree? Is the failure understood, maybe along Kai's comment 7 combined with Mike's reply of comment 8, sync vs. async? Any ideas how to address this?

Comment 11

[Kai Engert [:KaiE:]]

With the latest information, I agree, it looks like a regression in LDAP code.

I don't know how to fix this at this time, this needs analysis.

Comment 12

[marinar]

I agree that fixing MCD should be a priority.
Users love Thunderbird as email client, but if we can not autoconfigure user profiles, we will be forced to look for alternatives.

Comment 13

[Wayne Mery (:wsmwk)]

Seems like this should P1, so that we don't have users languishing on version 68

Comment 14

[Magnus Melin [:mkmelin]]

Mike, thanks for the help so far!

Is it possible to mark getLDAPAttributes async in the API? https://searchfox.org/mozilla-central/rev/c409dd9235c133ab41eba635f906aa16e050c197/extensions/pref/autoconfig/src/prefcalls.js#228 (and do other needed changes to make it return the values, when it has them)

I don't quite understand how the API is used? Are these functions that can be put into js code for autoconfig.js?

Comment 15

[Mike Kaply [:mkaply]]

I tied calling getLDAPAttributes async, but it didn't work.

the issue is the entire CPP file:

https://searchfox.org/comm-central/source/ldap/xpcom/src/nsLDAPSyncQuery.cpp

Is built around Sync LDAP.

Comment 16

[Ben Campbell]

Attachment: 1662433-add-error-handling-to-nsLDAPSyncQuery-1.patch

This patch should fix it.
There are a couple of things happening here:

1. I broke the nsLDAPSyncQuery when I added the onLDAPError() callback to nsILDAPMessageListener. (The "part-three" patch applied for Bug 1659947). This means it would hang if an LDAP error occured.
2. nsLDAPSyncQuery.getQueryResults() didn't really report any errors. This patch makes sure that meaningful errors are thrown.

CAVEAT:

• With this patch, I can connect and query my local slapd server, but my server isn't set up with any data, so I haven't tested a query that actually returns anything! I'm using a unit test file to drive it, and unit tests aren't allowed to connect to non-local servers. So it'd be good to see this working in a real use case. Any volunteers?

NOTES:

• previously, connection errors (eg no server at the given host/port) wouldn't cause an error, just an empty return string. Now connection errors will be thrown up. This might potentially trip up existing uses? (but it's still the Right Thing(tm) to do)
• security errors can now be caught here (eg connecting to a server with a self-signed cert, but without having that certificate registered as a security exception in Thunderbird). So you can check for this by looking at the error code. However, there's currently no way to access the failing certificate (so you can't pop up the add-an-exception dialog and retry). I'm not sure this would be desirable for enterprise-style setups or not... If needed I can add a mechanism to access the failed certificate when such an error occurs (probably by adding a .failedSecInfo attribute to the nsILDAPSyncQuery interface).

Comment 17

[Ben Campbell]

Attachment: 1662433-HACKHACKHACK-nsLDAPSyncQuery-test-1.patch

Just in case it's useful to anyone else - here's my hacky little unit test file for driving the nsLDAPSyncQuery.
Would be nice to turn it into a proper set of tests and add it, but we don't currently have a dummy LDAP server to run against.

Comment 18

[Mike Kaply [:mkaply]]

I'll take care of building and testing. It looks like this doesn't make ldapQuery async so no change needed to the caller?

Comment 19

[Ben Campbell]

(In reply to Mike Kaply [:mkaply] from comment #18)

I'll take care of building and testing. It looks like this doesn't make ldapQuery async so no change needed to the caller?

Yep, it's just a fix to the existing implementation. Same API.

Comment 20

[Magnus Melin [:mkmelin]]

(In reply to Ben Campbell from comment #17)

Would be nice to turn it into a proper set of tests and add it, but we don't currently have a dummy LDAP server to run against.

We do actually have an LDAP mock server now, since bug 1608304. Check browser_ldap_search.js and others. Doesn't do LDAP TLS (yet) though.

Comment 21

[Mike Kaply [:mkaply]]

Attachment: Firefox.cfg for testing

I gave this a try and it works differently (blank instead of null), but still doesn't show results.

I've attached a firefox.cfg that works in Thunderbird 68.2.2. It shows two results on the JS console at startup.

You can use this autoconfig.js in defaults/pref to enable it.

pref("general.config.obscure_value", 0);
pref("general.config.filename", "firefox.cfg");
pref("general.config.sandbox_enabled", false);

This is what I see in the console on 62.2.2

givenName=Aaron
givenName=Sheryl firefox.cfg:11
processLDAPValues firefox.cfg:11
getLDAPAttributes prefcalls.js:156
<anonymous> firefox.cfg:16

Comment 22

[Mike Kaply [:mkaply]]

Note I didn't add a try/catch to check for errors. I'll do that next.

Comment 23

[Mike Kaply [:mkaply]]

No errors, and it's definitely blank.

Comment 24

[Ben Campbell]

OK, obviously something else going on as well. I'll keep investigating. Having the script in Comment 21 and a mock LDAP server should help a lot.
I think the patch is still valid - all the cool kids are handling errors these days - but yeah, not much use if it doesn't actually return any data!

Comment 25

[Mike Kaply [:mkaply]]

That cfg file I attached should make it really easy to test.

Comment 26

[Ben Campbell]

Attachment: 1662433-fix-nsLDAPSyncQuery-output-1.patch

Thanks Mike - that was very helpful!
I think I've got it working. Seems like a simple fix, and I'm not entirely sure how it ever worked in the past...

Anyway. Want to give this one a try and see if it does what you want?
The previous error-handling patch is still required. I was going to roll it all into one, but it seems there are two separate and unrelated issues to fix, so I'll leave them separate.

If this does fix things, I'll code up some unit tests to go with it using our shiny new fake LDAP server.

Comment 27

[Ben Campbell]

(In reply to Mike Kaply [:mkaply] from comment #22)

Note I didn't add a try/catch to check for errors. I'll do that next.

Ahh yes. That won't work unless you write your own getLDAPAttributes() function. The one there in M-C catches the exception and displays an error dialog before it returns.

Actually given that prefcalls.js is the only place in M-C that seems to reference any of the nsILDAP classes, there's probably an argument for removing all the LDAP helpers there and make people supply their own LDAP code... (maybe with some helpers in C-C to make it trivial).

Comment 28

[Mike Kaply [:mkaply]]

Actually given that prefcalls.js is the only place in M-C that seems to reference any of the nsILDAP classes, there's probably an argument for removing all the LDAP helpers there and make people supply their own LDAP code... (maybe with some helpers in C-C to make it trivial).

I thought about that, but I'm not sure it's worth the effort, especially if it means folks having to modify their Autoconfig. This is stuff that has worked for 10+ years, so probably not worth reinventing.

As far as the try/catch goers, the actual call to query the server is in prefcalls.js, so we should be able to try/catch it there for failure, shouldn't we?

Comment 29

[Ben Campbell]

(In reply to Mike Kaply [:mkaply] from comment #28)

Actually given that prefcalls.js is the only place in M-C that seems to reference any of the nsILDAP classes, there's probably an argument for removing all the LDAP helpers there and make people supply their own LDAP code... (maybe with some helpers in C-C to make it trivial).

I thought about that, but I'm not sure it's worth the effort, especially if it means folks having to modify their Autoconfig. This is stuff that has worked for 10+ years, so probably not worth reinventing.

Yup - that's kind of what I figured. It'll all be in-house too, so no real public record of anyone using this stuff at all, even if it's critical for their organisation!

As far as the try/catch goers, the actual call to query the server is in prefcalls.js, so we should be able to try/catch it there for failure, shouldn't we?

That's what I meant - prefcalls.js is already catching the error and displaying a dialog box, so if you wanted to catch and handle any error conditions yourself (eg installing certificate exceptions, say), you'd have to do the server call yourself.

Comment 30

[Mike Kaply [:mkaply]]

Comment on attachment 9184417 [details] [diff] [review]
1662433-add-error-handling-to-nsLDAPSyncQuery-1.patch

Shouldn't this one be returning rv of the converted status?

@@ -154,7 +159,7 @@ nsresult nsLDAPSyncQuery::OnLDAPSearchEn
NS_WARNING(
"nsLDAPSyncQuery:OnLDAPSearchEntry(): "
"aMessage->GetAttributes() failed");

• FinishLDAPQuery();

• FinishLDAPQuery(NS_ERROR_FAILURE);
  return rv;
  }

Comment 31

[Mike Kaply [:mkaply]]

Patch works for me. Thank you!

I'm getting the Autoconfig fix into mozilla central.

Also, I caused some things to error out, and it worked as expected, showing the errors. So I don't think we need any changes to prefcalls.

Comment 32

[Ben Campbell]

Attachment: 1662433-add-error-handling-to-nsLDAPSyncQuery-2.patch

Minor tweak to the error-handling patch, based on Comment 30 (Also I realised I'd forgotten to set my diff output context to 8 lines instead of the default 4!)

Mike - you've obviously had a peek through the code, so I shall lumber you with a review for my two patches :-)

Comment 33

[Magnus Melin [:mkmelin]]

Any progress with getting the tests for it going?

Comment 34

[Ben Campbell]

I did start looking at this, but the existing fake LDAP server requires the test to manual pump data in and out of it, which means it's tricky to use with the nsLDAPSyncQuery (which doesn't return until the query completes). I think it'd be possible to use some javascript voodoo to get around this, but I'm rather leaning toward implementing more of the LDAP server - just enough to load it up with some test data and handle enough of the protocol to cover our pretty modest requirements.
Will investigate, and maybe break it out into a new bug.

Comment 35

[Magnus Melin [:mkmelin]]

These patches need un-bitrotting before landing

Comment 36

[Ben Campbell]

Attachment: 1662433-add-error-handling-to-nsLDAPSyncQuery-3.patch

Comment 37

[Ben Campbell]

Attachment: 1662433-fix-nsLDAPSyncQuery-output-2.patch

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/2538e2f3517e
Add error handling to nsLDAPSyncQuery.getQueryResults(). r=mkaply
https://hg.mozilla.org/comm-central/rev/c8169ad6ff96
Fix result attribute iteration & output in nsLDAPSyncQuery. r=mkaply

Comment 39

[Magnus Melin [:mkmelin]]

Comment on attachment 9185300 [details] [diff] [review]
1662433-fix-nsLDAPSyncQuery-output-2.patch

[Approval Request Comment]
User impact if declined: MDC ldap setup doesn't work
Testing completed (on c-c, etc.): on c-c, beta
Risk to taking this patch (and alternatives if risky): it's broken as is

Approval for both patches.

Comment 40

[Wayne Mery (:wsmwk)]

Comment on attachment 9185300 [details] [diff] [review]
1662433-fix-nsLDAPSyncQuery-output-2.patch

[Triage Comment]
Approved for esr78

Comment 41

[Wayne Mery (:wsmwk)]

Comment on attachment 9185298 [details] [diff] [review]
1662433-add-error-handling-to-nsLDAPSyncQuery-3.patch

[Triage Comment]
Approved for esr78

Comment 42

[Rob Lemley [:rjl]]

Thunderbird 78.5.1:
https://hg.mozilla.org/releases/comm-esr78/rev/6c60a6a605a8
https://hg.mozilla.org/releases/comm-esr78/rev/ebb170e1fa47

Comment 43

[marinar]

Hello
I had tried to plug the same autoconfig settings as in 68 to Thunderbird 78.5.1, but it seems it is not even reading config file.
Anybody had any success in making MCD working in Thunderbird?

Configuration below worked for all Thunderbird versions up to 68.3
...default\prefs\autoconfig.js

pref('general.config.obscure_value',0);
pref('general.config.filename', 'mozilla.cfg');
pref('general.config.vendor', 'mozilla');
pref('general.config.sandbox_enabled', false);

mozilla.cfg
lockPref("general.config.vendor", "oakton");
lockPref("autoadmin.global_config_url","https://******/thunderbird.cgi");
lockPref("autoadmin.append_emailaddr",false);
thunderbird.cgi reads .cfg that runs processLDAPValues function.
But it is not even reading configuraiton file, usually at this point failover.js is created in user profile.

Please advise.
Thank you

Comment 44

[Thomas D. (:thomas8)]

Hi Mike, any idea what might be wrong with the autoconfig setup of this user?
I've tried to offer some informed guesses...

(In reply to marinar from comment #43)

Hello
I had tried to plug the same autoconfig settings as in 68 to Thunderbird 78.5.1, but it seems it is not even reading config file.
Anybody had any success in making MCD working in Thunderbird?

Configuration below worked for all Thunderbird versions up to 68.3
...default\prefs\autoconfig.js

pref('general.config.obscure_value',0);
pref('general.config.filename', 'mozilla.cfg');

Hi Marinar, can you please try to move the following two lines into mozilla.cfg, and report back?
All the examples are just showing two lines of code in autoconfig.js as a pointer to actual configuration file.
Maybe it won't read more than the top two lines, which would make sense.

pref('general.config.vendor', 'mozilla');
pref('general.config.sandbox_enabled', false);

mozilla.cfg
lockPref("general.config.vendor", "oakton");
lockPref("autoadmin.global_config_url","https://******/thunderbird.cgi");

Also, I am not sure if the .cgi in between will work, because you're asking Thunderbird to run this file and I'd think it expects javascript files only, as the example in documentation suggests:

Centralized Management
The AutoConfig file can be managed centrally. To do so, reference the location of
a secondary AutoConfig file in the primary AutoConfig file:
pref("autoadmin.global_config_url","https://yourdomain.com/autoconfigfile.js");

lockPref("autoadmin.append_emailaddr",false);
thunderbird.cgi reads .cfg that runs processLDAPValues function.
But it is not even reading configuraiton file, usually at this point failover.js is created in user profile.

Please advise.
Thank you

Comment 45

[marinar]

Hello
Thank you for suggestions.
I tried these changes, but configuration did not work.

The described above setup was working for 15 years and originally were setup from this article
https://developer.mozilla.org/en-US/docs/Archive/Misc_top_level/MCD,_Mission_Control_Desktop_AKA_AutoConfig

Currently above setup works for Thunderbird 68.2.2 with pref('general.config.sandbox_enabled', false);

Any documentation how to setu Mission Control Autoconfig for Thunderbird 78

Thank you

Comment 46

[Mike Kaply [:mkaply]]

I verified that local autoconfig works fine with getLDAPAttributes in the local CFG file.

We didn't test getLDAPAttribute in a remote CFG file.

My autoconfig.js is:

pref("general.config.obscure_value", 0);
pref("general.config.filename", "firefox.cfg");
pref("general.config.sandbox_enabled", false);

My mozilla.cfg is:

//
var ldapHost = "ldap.adams.edu";
var ldapBase = "ou=people,dc=adams,dc=edu";

var ldapFilter = "surname=Abeyta";

var ldapAttrs = new Array( "givenname" );

// Define how to process LDAP results before we make the call
function processLDAPValues(queryResults)
{
Components.utils.reportError(queryResults);
}

// Call upon LDAP for the values in ldapAttrs array
// Uses the previous processLDAPValues()
getLDAPAttributes( ldapHost, ldapBase, ldapFilter, ldapAttrs.join(",") );

I'll try to convert this into a remote file for testing.

Comment 47

[Mike Kaply [:mkaply]]

I converted to a remote file and it worked fine for me.

Can you please run thunderbird with --jsconsole and see if there are any errors?

Comment 48

[marinar]

Thank you very much.

Autoconfiguration is working now from local cfg and directly from remote cfg, bypassing cgi call script in the middle.

Really appreciate your help.

Note: Some older code should be removed from .cfg, otherwise Thunderbird can not read configuration file.
Components.utils.import("resource:///modules/gloda/log4moz.js");
var log = Log4Moz.getConfiguredLogger("MCD");

Comment 49

[Thomas D. (:thomas8)]

Magnus, apart from the file name change (.js --> .jsm), is there anything else which could prevent enterprise users from importing this resource via MCD AutoConfig? Does the long form Components.utils work here or they must use Cu instead?

Thunderbird.cfg (AutoConfig file, local or external)

Components.utils.import("resource:///modules/gloda/log4moz.jsm");
var log = Log4Moz.getConfiguredLogger("MCD");

(In reply to marinar from comment #48)

Thank you very much.
Autoconfiguration is working now from local cfg and directly from remote cfg, bypassing cgi call script in the middle.

Really appreciate your help.

Most welcome, we're here to support you so that your enterprise scenarios will work well. So the .cgi was indeed part of the problem as I hinted in my comment 44.

Note: Some older code should be removed from .cfg, otherwise Thunderbird can not read configuration file.
Components.utils.import("resource:///modules/gloda/log4moz.js");

Marinar, this module still exists, but the file name has changed, it's now log4moz.jsm.
Can you please try again with the correct file name and report back?

https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/Log4moz.jsm

 var log = Log4Moz.getConfiguredLogger("MCD");

This should still work after the file name correction to .jsm, unless if we have another permissions/API problem here. Note that the logger name is case sensitive.

https://searchfox.org/comm-central/rev/5640a40e0d1d800e461050adadc46e29bdbe3982/mailnews/db/gloda/modules/Log4moz.jsm#47-72

(In reply to Thomas D. (:thomas8) from comment #44)

lockPref("autoadmin.global_config_url","https://******/thunderbird.cgi");

Also, I am not sure if the .cgi in between will work, because you're asking Thunderbird to run this file and I'd think it expects javascript files only, as the example in documentation suggests

Comment 50

[Thomas D. (:thomas8)]

Thanks a lot Mike for your expertise and guidance here!

(In reply to Mike Kaply [:mkaply] from comment #46)

I verified that local autoconfig works fine with getLDAPAttributes in the local CFG file.

(In reply to Mike Kaply [:mkaply] from comment #47)

I converted to a remote file and it worked fine for me.

Comment 51

[Magnus Melin [:mkmelin]]

(In reply to Thomas D. (:thomas8) from comment #49)
Any code constructs that have changed/been removed obviously won't work. Cu will work, but Cu.import doesn't import the same way as before. You can find code refs on https://searchfox.org/
Don't add log4moz usage either since we'll get rid of that soon.

If anyone has problems, strip down the file to minimum, without additional imports or other custom code that may be obsolete.

Comment 52

[marinar]

Hello

Just for information since it will be removed soon.
I tested configuration file with
Components.utils.import("resource:///modules/gloda/log4moz.jsm");
var log = Log4Moz.getConfiguredLogger("MCD");
and received error "Failed to read the configuration file. Please contact your system administrator."

Works perfectly without those lines

Thank you