Closed Bug 1573690 Opened 5 years ago Closed 2 years ago

[meta] Problems caused by userName/realuserName and hostName/realhostName

Categories

(Thunderbird :: General, defect, P2)

Product:
Thunderbird
Component:
General
Type:
defect

Tracking

(thunderbird_esr91 wontfix)

Status:
RESOLVED FIXED
Milestone:
102 Branch
Tracking Flags:
Tracking Status
thunderbird_esr91 --- wontfix

People

(Reporter: jorgk-bmo, Assigned: rnons)

References

Details

(Keywords: meta)

Let's start a collection.

Someone would have to do the digging on the exact background and usage of realhostname, but there must be some better way than keep having both hostname and realhostname and get them wrong half the time. We need to investigate how we can get rid of realhostname.

Keywords: meta

Khushil, this could be something to investigate for you.

I find it likely we can remove it, but probably it needs some migrator code for the cases we already have.

Assignee: nobody → khushil324
Priority: -- → P2
Depends on: 1676596
Assignee: khushil324 → benc

Ben, does https://support.mozilla.org/en-US/questions/1315845 fit into one of the blockers?

Flags: needinfo?(benc)

(In reply to Wayne Mery (:wsmwk) from comment #3)

Ben, does https://support.mozilla.org/en-US/questions/1315845 fit into one of the blockers?

Yes, that does sound like the kind of problem that this might cause!

I did have a quick poke around the other day, but came away a little baffled over the purpose of the username/realUsername split... it must have been added for some good reason, but I can't quite figure it out. Will try and do a deeper dive soon.

Flags: needinfo?(benc)

It's used for the case where you have imap.foo.com set up, and then the server changes (maybe the company was bought up) and you should start using imap.bar.com. At that point, what's on disk will have a directory named according to the older name, etc. But I don't know the exact details of why it was chosen just not to do needed renames at that point. It's certainly caused lots of subtle issues, and I think we should see if we can't just do the needed adjustments where hostname changes, and not have the two different hostnames in history forever.

From matrix:

Standard8
mkmelin: oh, I think I vaguely remember.
I think it is because for the various accounts we use the uris which include hostname + username.
However, if the user comes along and changes either or both of those, we still need to reference the old URIs in various places iirc.
I think password manager might have been one of those.
I vaguely remember discussing this with Bienvenu a long time ago.
mkmelin
It's certainly before hg time...
Standard8
yeah I wouldn't be surprised if this was in the suite.
and netscape
mkmelin
Needing to enter password again if you change hosts doesn't sound so bad. There could be worse cases that need changing of course.
Standard8
It'd be nice to tidy up the old one though 😉
Thinking about it, it might be more viable now we don't have RDF
but you'd probably want to audit if these are being used anywhere, in URIs etc.
mkmelin
Yeah... Ever so often we encounter problems due to the hostname/realhostname split. I think certs overrides is most recent one. Proxies is another one. One for OAuth I recently fixed.
Standard8
Definitely worth a look around again I think, with the various changes that have been done.
Oh and as you said in the bug, directory names as well...
Certainly feels like you should be able to not expose those at the interface levels, though I can't remember how that lot is structured.
jcranmer
Standard8's description matches up with my intuition
we use username + hostname to uniquely/persistently identify accounts in URIs in several places, and changing host name shouldn't change that ID
e.g., the file location storage is based on hostname + realhostname
were it up to me, I'd map realhostname -> hostname and hostname -> originalhostname (ditto for username), and then try to carefully audit all places where we actually use originalX to see if we actually need to persist it
retrofitting that onto our existing account setup… I'm not sure the realhostname is the worst thing in our system so far
Standard8
it might not be, but it certainly can be confusing... and I like the idea of changing to use original host name, much clearer.
jcranmer
it might be feasible now that XUL extensions are dead

Depends on: 1681489
Depends on: 1692334

(In reply to Magnus Melin [:mkmelin] from comment #6)

From matrix:
...
jcranmer
Standard8's description matches up with my intuition
we use username + hostname to uniquely/persistently identify accounts in URIs in several places, and changing host name shouldn't change that ID
e.g., the file location storage is based on hostname + realhostname
were it up to me, I'd map realhostname -> hostname and hostname -> originalhostname (ditto for username), and then try to carefully audit all places where we actually use originalX to see if we actually need to persist it
retrofitting that onto our existing account setup… I'm not sure the realhostname is the worst thing in our system so far
Standard8
it might not be, but it certainly can be confusing... and I like the idea of changing to use original host name, much clearer.
jcranmer
it might be feasible now that XUL extensions are dead

In which bug report do you want this work to be done?
A good project for new staff?

Flags: needinfo?(mkmelin+mozilla)

Probably best to do the work to get rid of it in this bug.

Flags: needinfo?(mkmelin+mozilla)
Depends on: 1429040

(In reply to Magnus Melin [:mkmelin] from comment #5)

It's used for the case where you have imap.foo.com set up, and then the server changes (maybe the company was bought up) and you should start using imap.bar.com. At that point, what's on disk will have a directory named according to the older name, etc. But I don't know the exact details of why it was chosen just not to do needed renames at that point. It's certainly caused lots of subtle issues, and I think we should see if we can't just do the needed adjustments where hostname changes, and not have the two different hostnames in history forever.

Wouldn't it be helpful to name the local folders NOT according to the hostnames, but use the same key as in the settings (e.g. server9)? Then the change of a server address would not require and folder renaming.

Also, what I noticed a couple of times is that in some cases, error messages (e.g. when no connection can be made) report that "hostname" has a problem even though TB tries to connect to "realhostname". This confused me until today when I figured out that a hostname change will maintain the original hostname value...

Depends on: 1727677

Because of .directory and .directory-rel prefs, I don't think folders on disk is a problem.

Seems to me two prefs are still needed, because after changing hostname, all the folder/message urls still use the old hostname, unless we

  • require a restart
  • propogate hostname change everywhere without a restart

Things can be improved to fix bug 1762301, bug 1727677, bug 1765754

  1. when constructing server/folder/message urls, use realhostname and fallback to hostname
  2. when finding server by folder/message urls, use realhostname and fallback to hostname
  3. on startup, replace hostname with realhostname, then remove realhostname

This may be a litter easier after bug 1707547

Requiring restart for such a special case sounds reasonable to me.

(In reply to Magnus Melin [:mkmelin] from comment #11)

Requiring restart for such a special case sounds reasonable to me.

That would be easy. On hostname change, we can set a pref (e.g. pendinghostname) and show a line Hostname change will take effect after a restart [Restart now], and nothing else.

Then on restart, we merge realhostname/pendinghostname into hostname, and remove realhostname/pendinghostname.

Do you want me to take this?

Yes please!

Assignee: benc → remotenonsense

Fixed in bug 1483485.

Status: NEW → RESOLVED
Closed: 2 years ago
status-thunderbird_esr91: --- → wontfix
Resolution: --- → FIXED
Target Milestone: --- → 102 Branch

So I am seeing a lot of fallout due to the unannounced renaming of server and username as it changes all Folder URIs on affected accounts - and my Add-on(s) store folder URIs in order to persist information for my users (In the case of QuickFolders, these are folder tabs and special settings such as reply identities and visual configurations that are stored on a per folder basis).

I have 2 questions:
Is there any way to reconstruct the old folder URI to write a mitigating tool that fixes these problems? Or maybe a log entry somewhere on what URIs were merged in this way? Not all users have backups of user.prefs for reconstruction.

Is this a once-off fix that happens during the migration to Thunderbird 102 or can it happen again once the accounts are migrated?

Related issue - I hope I summarized this correctly.

https://github.com/RealRaven2000/QuickFolders/issues/281

I would be very very grateful for some feedback there. I need to find a way of mitigating fallout from 1000s of users migrating their profiles and ending up with broken QuickFolders tabs.

Flags: needinfo?(remotenonsense)

This might be effecting filter rules in general on upgrading from 91 to 102. Bug 1777731

See Also: → 1777731

(In reply to Axel Grude from comment #15)

I have 2 questions:
Is there any way to reconstruct the old folder URI to write a mitigating tool that fixes these problems? Or maybe a log entry somewhere on what URIs were merged in this way? Not all users have backups of user.prefs for reconstruction.

At the point of upgrading, the old hostname/username still exists. After that, they are gone.

Is this a once-off fix that happens during the migration to Thunderbird 102 or can it happen again once the accounts are migrated?

The folder uri changes every time username/hostname get changed. Sorry, I don't have any good solution at the moment.

Flags: needinfo?(remotenonsense)

Just an update on this - here is how I remedy the problem from an Add-ons writer's perspective. QuickFolders now also stores the account id (e.g. "account1" together with the URI to store its folder shortcuts; once the upgrade to Thunderbird 102 happens (or a server is renamed), Thunderbird can detect invalid URIs and tries to rebuild the URI with the new root and fixes them quietly. Note that when you use Services.io.newURIto generate a new valid uri from the old one, you cannot use "imap" unless the folder exists for certain - POP3 and Local host have no such problem (hence the replacement with http)

    function reconstructUri(e) {
      let ac = MailServices.accounts.getAccount(e.account), acType = "N/A";
      if (ac && ac.incomingServer) {
        acType = ac.incomingServer.type || "unknown";
        if (acType!="im") { // do not reconnect to chat accounts - these are not supported.
          let oldUri = e.uri.replace("imap://", "http://"); // non-existent imap URIs throw!!
          let tempUri = Services.io.newURI(oldUri); // generate a fresh URI to retrieve path portion
          // or oldUri.filePath - not sure whether query params make sense or are allowed for  folder URIS
          return ac.incomingServer.serverURI + tempUri.pathQueryRef; 
        }
      }
      throw new Error(`Failed to reconstruct URL for ${e.uri} of account ${e.account} - account type ${acType}`);
    }

I am hoping that this will address the issue when Thunderbird 102 switches to manual / automatic update for all current users of 91.

(In reply to Ping Chen (:rnons) from comment #18)

(In reply to Axel Grude from comment #15)

I have 2 questions:
Is there any way to reconstruct the old folder URI to write a mitigating tool that fixes these problems? Or maybe a log entry somewhere on what URIs were merged in this way? Not all users have backups of user.prefs for reconstruction.

At the point of upgrading, the old hostname/username still exists. After that, they are gone.

Is this a once-off fix that happens during the migration to Thunderbird 102 or can it happen again once the accounts are migrated?

The folder uri changes every time username/hostname get changed. Sorry, I don't have any good solution at the moment.

Getting quite a few requests from quickFIlters users who notice that their filters have invalid target folders. Which bug do I point them to that manages the actual change of converting from realhostname to hostname (without backing up the aliases) ?

Filters were migrated in bug 1769732.

You need to log in before you can comment on or make changes to this bug.