Closed Bug 1211160 Opened 9 years ago Closed 8 years ago

Thunderbird Version 38.3 Buttons not working menu items and attachments disappearing

Categories

(Toolkit :: Add-ons Manager, defect)

defect
Not set
major

Tracking

()

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: Fabian_Finlay, Assigned: rkent)

References

Details

(Whiteboard: DUPETOME)

Attachments

(3 files, 5 obsolete files)

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36

Steps to reproduce:

Fixed it by reinstalling 38.2


Actual results:

Bug in 38.3 though problem did not happen till possibly 2 days after update installed.
Buttons do not work.  Help Menu does not show on menu bar though Menu bar visible.
Could not make attachments visible.  The attachment icon was shown but double clicking it did nothing.
Could not change folders to view.

I tried the following.

Downloaded 38.3 and reinstalled.
No change

Deleted msf files using Thunderfix and allowed Thunderbird to rebuild.
No change

Downloaded 38.2 and reinstalled.
All problems solved.

Conclusion: Bug in 38.3



Expected results:

Program should have worked.  It didn't.
What happens when you start 38.3.0 in safe mode?
 https://support.mozilla.org/en-US/kb/safe-mode
 http://getthunderbird.com/
Flags: needinfo?(Fabian_Finlay)
I am afraid I cannot answer that question now as I have resolved the problem by reinstalling Version 38.2 and the system now works.  I did however try some additional alternatives first which I forgot in my first post set out below:


Emptied Junk Folder
No change

Compacted all folders
No change

Cure for delete button not working found in Forum.  Deleted foldertree.json, panacea.dat and session.json with Trash and Trash.msf
No change

I also tried running with addons disabled but since at that stage as I was getting virtually no response from Thunderbird I am not sure if it actually ran with Add Ons disabled or not. I did not get the confirmation page suggested in the Help link above.  I only tried running with Add Ons disabled by clicking on the link on the help menu.  Nothing happened.  The Help menu item was missing from the Menu Bar but I was able to access it through the three bar menu button.

Another detail.  When I right clicked on blank space in the tab row I got a little box with a button for Customise.  Clicking on Customise like any other button did nothing.  The Menu Bar and Menu Toolbar options were missing.

I really don't want to reinstall 38.3 just to see if it stops working again but if it happens again I now know to start Thunderbird while holding down the shift key to start Thunderbird Safe Mode.
Flags: needinfo?(Fabian_Finlay)
Fabien, did you report this anywhere else?  And, if you already tested 38.3.0 all those ways, that is excellent and more than sufficient.

I recall seeing in the past week a report like this about help menu, but cannot remember where, in bugzilla or support.
Component: Untriaged → Mail Window Front End
Flags: needinfo?(Fabian_Finlay)
Just a note.  Foldertree.rdf is obsolete, replaced by XULStore.json.

Are you using the default theme?
Wayne, no I thought this was meant to be the place to report a bug so no report elsewhere.  I could find nothing about it on the User forum.

Now that I know how to launch safe mode from the desktop I see that what I needed to do was launch it in Safe Mode and then if it worked gradually reinstate each of my Add Ons.  I suspect one of them may be the cause of the problem because my wife's Thunderbird Version 38.3 works fine but she does not use the same Add Ons as me.

I am travelling today so it will have to wait but when I am settled I will reinstall 38.3 and if the problem reappears try that to see if i can pin it down.

Matt, I don't even know what the default theme is so I guess that means I am using it.
Flags: needinfo?(Fabian_Finlay)
Commenting to say that I have experienced this problem too on WinXP and Win7 with different users. Even clearing out the installation-folder completly and reinstaling didn't work.
I've tried everything that Fabian Finlay also tried and in the end installed version 38.2.0 and disabled updates for now.

There are also quite a few threads about this in the help-forum on support.mozilla.org with no answers that led to a solution.
Ok when I started using my laptop while travelling it showed the same problems as my desktop.  Installing 38.2 solved the problem. 

I then disabled all add ons while running 38.2, closed it down and installed 38.3.  It started with same problems shown instantly by no Help Menu.  All addons were disabled still except the integrated Lightning 4.03.  I disabled it and restarted and the Help menu reappeared.  Repeated the process to check it was not a fluke.

The problem seems to be with Lightning which is now meant to be integrated.

Over the next couple of days I will gradually restore my other addons and check for other problems.
A lot of the computers in my company suffer the same issue.
Thunderbird 38.3
Lightning 4.0.3

When I restart thunderbird in safe mode, everthing works fine.
Then I activate lightning 4.0.3, once again nothing works.

Maybe Lightning 4.0.3 dosen't work with the gui of Thunderbird 38.3
Exact same problem here. 
I downgraded to 38.2, disabled Lightning, applied the update... and then it worked !
Lightning was updated and enabled after the update (is this normal ?), but TB fully working.

On a computer where 38.3 was already installed, I add to start in safe mode with addons disabled, remove Lightning 4.0.3, restart, and install Lightning via the addons menu (the only proposed version was 4.0.2.1 though). This worked, but I have an older Lighting version. 

The problem seems to lie in the upgrade process from the separate Lightning addon to integrated 4.0.3 addon.
http://forums.mozillazine.org/viewtopic.php?p=14355767#p14355767 indicates reports in German forums.

and from bug 1211291 (I'm leaving that open in Thunderbord product)...
http://forums.mozillazine.org/viewtopic.php?f=39&t=2965781
https://support.mozilla.org/en-US/questions/1087066
https://support.mozilla.org/en-US/questions/1087126
Blocks: 1211291
Severity: normal → major
Component: Mail Window Front End → General
Keywords: regression
Product: Thunderbird → Calendar
Version: unspecified → Lightning 4.0.3
When the TB is stuck is it possible to access tools->error console and see any errors in there?
Nothing in my error logs
(In reply to :aceman from comment #12)
> When the TB is stuck is it possible to access tools->error console and see
> any errors in there?

See http://forums.mozillazine.org/viewtopic.php?p=14355323#p14355323:
"The error console won't open either!"
I agree with Eckard.  The Tools console will not open while Lightning is enabled.  So while it was disabled I cleared the log.  Then I enabled it restarted, disabled it and restarted.

The tools Console now has the following messages.


Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Thunderbird/chrome.manifest'.
Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Thunderbird/extensions/%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D/chrome.manifest'.


These mean nothing to me but I hope they may be of some use.

The following 2 messages appeared subsequently so are probably not relevant

Timestamp: 06/10/2015 17:53:49
Error: path is null
Source File: resource://gre/modules/osfile/ospath_win.jsm
Line: 220

Timestamp: 06/10/2015 17:54:49
Warning: This site makes use of a SHA-1 Certificate; it's recommended you use certificates with signature algorithms that use hash functions stronger than SHA-1.
Source File: https://aus4.mozilla.org/update/3/Thunderbird/38.3.0/20150928051427/WINNT_x86-msvc/en-GB/release/Windows_NT%206.3.0.0%20(x64)/default/default/update.xml
Line: 0
Thanks Fabian
Status: UNCONFIRMED → NEW
Ever confirmed: true
Starting TB 38.3 in safe mode, disable Lightning, restart TB, downgrade no need to remove first) Lightning to 4.0.2.1, enable Lightning, restart TB, problem resolved.

The problem is obviously Lightning 4.0.3, as 4.0.2.1 works perfectly.
Can you go one step further, downgrade to Thunderbird 38.2.0 again, then upgrade and make sure that Lightning gets updated to 4.0.3 again and the problem occurs again? This would help figure out how to reproduce.

I don't think it is 4.0.3 by itself, I think it is rather the corrupted update. Here is the not-yet-uploaded AMO version of Lightning 4.0.3, that should work equally well: https://mozilla.kewis.ch/releases/ltn403/

This version differs from the packaged version, but I've had someone check the packaged versions and the problem is not in the packaging itself. I hope we can get this fixed soon.
As with bug 1196666, user at https://support.mozilla.org/en-US/questions/1086709 encountered this problem with prior updates of Thunderbird
I can duplicate the symptoms of this by taking the bad chrome.manifest that Fallen sent me, and replacing that in the Lightning install in the profile.

I wonder if there is some way that we can at least prevent a Lighting issue from hosing all of Thunderbird, using that approach to reproduce?
I did some forensics on SUMO into the August and early september time frame. There are only a few posts about update to 38.2.0 affecting message list - not nearly as many as we just had for 38.3.0. I found only 5
https://support.mozilla.org/en-US/questions/1078753
https://support.mozilla.org/en-US/questions/1079165
https://support.mozilla.org/en-US/questions/1079629
https://support.mozilla.org/en-US/questions/1080973
plus https://support.mozilla.org/en-US/questions/1078326 which was blamed on Theme Font & Size Changer addon

So consider - at least half of users going to version 38.2.0 would have come from 31.*. Does updating within a release, eg 38.n to 38.n+1 increase the probability of problems?  versus 31 to 38 which seemed to have less problems.
Here's some analysis by me, repeating also things others have said.

Looking at the working and broken examples, it appears that the broken case updated at some point Lightning to 4.2.1 (through presumably AMO) then updated Thunderbird to 4.3.0.

When Thunderbird updated, all of the files for Lightning 4.3.0 were successfully copied from the distribution directory to the profile except for chrome.manifest. That file appears to still be the version from the 4.2.1. If I place that chrome.manifest in my profile directory with Lightning 4.3.0, I can reproduce the symptoms of this bug (my comment 22).

The extension copying code in Gecko is quite robust. I tried to simulate the effect of a locked chrome.manifest during the update process. Gecko successfully detected that the file could not be copied, and rolled back the copy so that Lightning remained at version 4.2.1, and the symptoms did not appear.

So I am left with something external to Gecko that did not allow the file to be copied, yet did not correctly report an error - maybe an AntiVirus program of some sort?

Without STR and a clear cause, how can we respond in the sort run? (In the 45 time frame, hopefully system addon capability will be implemented and we can use that See "Bug 1207287: Move the app-shipped system add-ons somewhere where they will be included in up date MARs." for both recent work there, and possible issues related to this bug).

One choice would be to set "extensions.installDistroAddons" to false, which will prevent the update of Lightning from the distribution directory. In the future, we could have code that runs with this initially true, but sets it to false in the profile once the initial profile is created (and presumably Lightning is now installed) I think that we could live in the long run with only installing Lightning once in a new profile, then relying on AMO for updates. 

A related option would be to add a preference to only install from the distro if the addon is missing, but not if the addon is already installed. That would only be a few lines of code, and I think that we could do that safely.
Attachment #8672124 - Flags: feedback?(philipp)
These two patches provide one possible approach to this issue. Basically, we only allow Lightning to be installed once from distribution, relying on AMO for additional updates.

I would only set this preference in comm-esr38. For TB45, we'll need to consider if we want to allow update from distribution or not.
Attachment #8672127 - Flags: feedback?(philipp)
Can we somehow either force a restart or advise the user to restart following the update.  Users will be in support complaining that lightning does not work in some way because they have not restarted to install the AMO update unless we do.  They have been doing it until now.  But it was an annoyance I was hoping to move on from with the inclusion of Lightning in the distribution.
In mozilla.support.calendar a user complained today that Lightning 4.0.3.1 could not be installed on top in Thunderbird 38.3. They only get a red message bar "Thunderbird can not modify the needed file".

1) Do we know that the problem is caused by an incorrect chrome.manifest file shipped in a setup package or mar package? Or is a correct file shipped but it failed to replace an older chrome.manifest file? Maybe wrong file attributes?

2) Comment 28 made me think about the following workflow, maybe it is related to the error or not. Thunderbird 38.2 is running. In the background Lightning 4.0.2.1 was downloaded from AMO and is waiting for a Thunderbird restart to install. User gets Thunderbird 38.3 update and restarts Thunderbird. What Lightning version will be installed? 4.0.2.1 from AMO or 4.0.3 from distribution folder?

3) Does the install from distribution only once patch affect users that don't use AMO? For example Linux distributions that either ship Lightning as a separate package or included in the Thunderbird package?
(In reply to Stefan Sitter from comment #29)
> In mozilla.support.calendar a user complained today that Lightning 4.0.3.1
> could not be installed on top in Thunderbird 38.3. They only get a red
> message bar "Thunderbird can not modify the needed file".

Enabling extensions.logging.enabled would provide more information eventually.
 
> 1) Do we know that the problem is caused by an incorrect chrome.manifest
> file shipped in a setup package or mar package? Or is a correct file shipped
> but it failed to replace an older chrome.manifest file? Maybe wrong file
> attributes?

Given the provided examples in bug 1211358, the shipped extension itself is consistent in distribution/extensions, so this looks like a problem in the addon installation code [1]. The code for installing the an addon is deleting the previous copy of the addon from the profile prior to copy the new version to the profile. This part of the code has a rollback capability as Kent already mentioned above, though there is no fine graned logging to proof evidence in the log that it has been completely removed prior to install the new version.

> 2) Comment 28 made me think about the following workflow, maybe it is
> related to the error or not. Thunderbird 38.2 is running. In the background
> Lightning 4.0.2.1 was downloaded from AMO and is waiting for a Thunderbird
> restart to install. User gets Thunderbird 38.3 update and restarts
> Thunderbird. What Lightning version will be installed? 4.0.2.1 from AMO or
> 4.0.3 from distribution folder?

It looks like staged addons are installed prior to distro addons[2], but that would be worth a test. 4.0.2.1 was released only a week before TB38.3, so it's probably fair to assume that such a constallation has occurred for a significant number of users but not all users though - which is exactly what we're seeing.

[2] https://mxr.mozilla.org/mozilla-esr38/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm#2700
[2] https://mxr.mozilla.org/mozilla-esr38/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm#3506
Great that we have some many looking at this and offering multiple perspectives.
My concern about going with kent's proposed patch is it may return us to the problems of the past here calendar sometimes does not automatically update when Thunderbird updates. (however, the patch may be the lesser of two evils)
(In reply to Stefan Sitter from comment #29)
> 2) Comment 28 made me think about the following workflow, maybe it is
> related to the error or not. Thunderbird 38.2 is running. In the background
> Lightning 4.0.2.1 was downloaded from AMO and is waiting for a Thunderbird
> restart to install. User gets Thunderbird 38.3 update and restarts
> Thunderbird. What Lightning version will be installed? 4.0.2.1 from AMO or
> 4.0.3 from distribution folder?

I did some more testing to cover also this scenario, but unfortunately, I wasn't able to reproduce the issue that way (using a new profile), too. As expected, Lightning 4.0.3 got installed properly.
Mike, do you have any experience, suspicions in this area of failed updates/addon behavior?
Note also the examples in bug 1211358.
Flags: needinfo?(mozilla)
FWIW, last week I started triaging/examining a selection of addon update/upgrade bug reports with hopes of coming across something related to this calendar update issue. Two query examples, with distribution extensions in bug comment:
http://mzl.la/1OwpBdw
http://mzl.la/1OwpNJK
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #33)
> Mike, do you have any experience, suspicions in this area of failed
> updates/addon behavior?
> Note also the examples in bug 1211358.

I do not, but I've always wondered what would happen in this scenario. I'm really surprised the distribution/extensions code isn't smart enough to not install if it is downleveling the extension. That seems like the core problem here.
Flags: needinfo?(mozilla)
Comment on attachment 8672124 [details] [diff] [review]
part 1, Add preference to only install certain addons once.

Review of attachment 8672124 [details] [diff] [review]:
-----------------------------------------------------------------

Aside from a minor nit to put a period at the end of the pref, wouldn't this pref completely disable installing the distribution add-on? I don't see a check for it existing in the profile. Even if that were there, this wouldn't install the distro add-on for people upgrading from 31 -> 38.next, leaving it to AMO, which has failed in the past and one of the major reasons we wanted to integrate Lightning.
Attachment #8672124 - Flags: feedback?(philipp) → feedback-
Attachment #8672127 - Flags: feedback?(philipp)
Oops, you are correct. In one of my (In reply to Philipp Kewisch [:Fallen] from comment #36)
> Comment on attachment 8672124 [details] [diff] [review]
> part 1, Add preference to only install certain addons once.
> 
> Review of attachment 8672124 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Aside from a minor nit to put a period at the end of the pref, wouldn't this
> pref completely disable installing the distribution add-on? I don't see a
> check for it existing in the profile.

Oops, you are correct. In one of my optimizations I moved this too far up in the code. This code needs to be right before "// Install the add-on"

> Even if that were there, this wouldn't
> install the distro add-on for people upgrading from 31 -> 38.next, leaving
> it to AMO, which has failed in the past and one of the major reasons we
> wanted to integrate Lightning.

Yes, this is really the key question. Unfortunately we are now failing, and failing very badly, trying to install from distro. We need a plan to allow us to unblock Thunderbird 38.3 updates which are needed for many reasons. I'm open to other suggestions, but we need to do something.
(In reply to Kent James (:rkent) from comment #37)
> Oops, you are correct. In one of my (In reply to Philipp Kewisch [:Fallen]
> from comment #36)
> > Comment on attachment 8672124 [details] [diff] [review]
> > part 1, Add preference to only install certain addons once.
> > 
> > Review of attachment 8672124 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > Aside from a minor nit to put a period at the end of the pref, wouldn't this
> > pref completely disable installing the distribution add-on? I don't see a
> > check for it existing in the profile.
> 
> Oops, you are correct. In one of my optimizations I moved this too far up in
> the code. This code needs to be right before "// Install the add-on

This should live within the

 if (existingEntry) {...}

section I guess to not cause a regressions for the use case of resetting an extensions.installedDistroAddon.* pref for stopping to prevent an distro addon installation.
May I please have some suggestions on how to move forward with this? Nobody really liked my approach, but I don't understand what the alternative is. We are blocked on this, and we need a plan.
See bug 1214440. If Lightning is getting flagged by AVs, this might cause the problems as I suggested in comment 25.
(In reply to Kent James (:rkent) from comment #25)
> ...
> So I am left with something external to Gecko that did not allow the file to
> be copied, yet did not correctly report an error - maybe an AntiVirus
> program of some sort?

I do not have information to refute the possible involvement of antivirus. However, I'm fairly certain I've encountered reporters that have not installed an AV program.


> [full disable] Without STR and a clear cause, how can we respond in the sort run? (In the
> 45 time frame, hopefully system addon capability will be implemented and we
> can use that See "Bug 1207287: Move the app-shipped system add-ons somewhere
> where they will be included in up date MARs." for both recent work there,
> and possible issues related to this bug).
 
> [partial disable] A related option would be to add a preference to only install from the
> distro if the addon is missing, but not if the addon is already installed.
> That would only be a few lines of code, and I think that we could do that
> safely.

Fallen, with kent's additional option above, how do you rate our current choices (acknowledging that we lack hard data to rate them):
1. full disable until TB45 (and can we effectively reenable or do system addon?)
2. partial disable (and can we effectively reenable...?)
3. re-enable 38.3.0 updates as is (according to crash-stats 84% of users are updated [1] - but presumably we'd have the same problem when 38.4.0 is released) 
4. wait a few more days for potential STR/more attractive options
5. any option I've missed

I feel we can wait longer another week or so for STR because 38.3.0 fixes no security issues. But failing that, choice 2 seems most attractive to me, assuming it doesn't kill our long term options.


[1] https://crash-stats.mozilla.com/daily?p=Thunderbird&v=&v=38.1.0&v=38.2.0&v=38.3.0&hang_type=any&os=Windows&os=Mac+OS+X&os=Linux&date_range_type=report&date_start=2015-09-30&date_end=2015-10-20&submit=Generate


(In reply to Kent James (:rkent) from comment #40)
> See bug 1214440. If Lightning is getting flagged by AVs, this might cause
> the problems as I suggested in comment 25.

If antivirus is involved, our more techincally astute users should be able to tell us they found something their AV logs.  As for bug 1214440, IMO it is at worst a minor false positive.
Flags: needinfo?(philipp)
Would a re-install of lighting from AMO help? If so, could we detect the broken state and get the update if we're broken?
There is a new antivirus related report in bug 1214434 for 4.0.3.1.
I am not sure what AMO means but having fixed my problems by reinstalling Thunderbird 38.2 and Lightning 4.0.2 I tried updating to Lightning 4.0.3.  You cannot do that because it is incompatible with 38.2.  So I thought I would try updating to 38.3 with Lightning disabled and then installing it.

Thunderbird 38.3 opened in working order with of course no Calendar Tab but there was a switch to Calendar tab button visible.  The only defect was that for the first time none of my previous tabs and emails were open but that is not a major problem - at least not for me, it may be for others.

In Add-Ons Manager Lightning 4.0.3 was enabled even though before I installed Thunderbird 38.3 it was disabled.  I wonder if this is because I had tried but failed to install it in 38.2 or if it would automatically be enabled in any new installation of 38.3.  I had not uninstalled 38.2 just closed it and installed 38.3 manually over the top.

When I clicked on switch to the Calendar Tab it appeared and everything else seemed to carry on working.

I am not sure what happened next.  I closed Thunderbird but there still seemed to be a version open with all the tabs I had previously.  Maybe I had not closed it before reinstalling though I am sure that I had.  Any way it was also 38.3.  I then closed that and reopened and still had all my old settings in 38.3 with Lightning 4.0.3 installed but the Calendar tab not yet opened.  ie everything looked the way it looked before I reinstalled 38.3 except there was a button for switch to Calendar Tab.  The calendar tab opened fine and everything carried on working..

Conclusion.

You may have already reached this conclusion.  If so apologies for the duplication.  

Version 38.3 updates fine if you disable Lightning first.  Does that help decide on the best fix.  

I will now reset my options to allow automatic updates.
Anyone seen this happen for version 38.4.0 who has NOT reported it already in a bug report?
No longer blocks: 1211291
Flags: needinfo?(philipp)
Hit again by this bug after auto-updating to thunderbird 38.5.0. Solution was to remove and reinstall Lighthning. Again this did happen only after a restart after updating.
(In reply to Christoph Nelles from comment #46)
> Hit again by this bug after auto-updating to thunderbird 38.5.0. Solution
> was to remove and reinstall Lighthning. Again this did happen only after a
> restart after updating.

Could you describe a little your environment, particularly if there is anything unusual about your file system? This bug only seems to occur on a small minority of systems, we cannot reproduce, and we need to know what might be unique about those systems seeing the failure.
We are seeing this again with the 38.5 update as well. The computers are Windows 7 in a Samba 3.6 domain.  Other than that, things are pretty generic.

FWIW, these are the recovery steps.

1. Start Task Manager and end any running Thunderbird instance.
2. Open up a cmd window
3. cd "C:\Program Files (x86)\Mozilla Thunderbird"
4. Start tbird in safe mode:  thunderbird -safe-mode
5. Select "Disable all add-ons"
6. Press "Make Changes and Restart"
7. Tbird should start with inbox viewable
8. From the tbird menu, "Tools", "Add-ons", From Left menu "Extensions"
9. Lightning: Remove, restart
10. Install Lightning.
My current best guess on this issue is that it is a synchronization issue with server side Windows profiles (most of the reports are from organization and mostly limited to Windows; we haven't found any obvious reason in our code so far, so it's more likely to have a reason in the user's environment). Also other options for file synchronization are candidates.

MAR updates don't update the chrome.manifest in distribution/extensions (which is correct as it has not changed from perspective of automated builds), so when Lightning is installed into the user profile, the chrome.manifest then has an older timestamp than that of a previously installed AMO version.

From file sync perspective, this would mean the updated chrome.manifest is the outdated version that needs to be replaced by the serverside version (in fact the AMO version), as at both locations the file exists and usually the newer one is persisted.

If this is it, we could easily e.g. add a comment line referring to the current version into the chrome.manifest which we update during the build process automaically to assure we have always a up-to-date version of the chrome.manifest.


As I cannot reproduce the issue myself, I would need an affected user to do a simple simple test to confirm or rebut the above on that very system (maybe you will need assistance from your IT support):

1. Assuming you have a working Lightning, shut down TB if running.

2. Copy your current chrome.manifest located in the local copy of <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} to a place elsewhere

3. Copy the chrome.manifest from <tb-app-dir>/distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} to the local copy of <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} and cross check the copied file still has the same timestamp as in the source folder (this step would break Lightning from running at this point in time, so don't restart TB until the test is completed)

4. Trigger file synchronization (depending on implementation, this may be a Windows logoff/logon or a serverside triggered background operation) in an appropriate way.

5. Check the chrome.manifest file in <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} - is this still the copied version or replaced by the previous version, you have backed up before (check timestamp and content)?

6. In the former case, replace the chrome.manifest in <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} with the backed up copy again before starting TB again. (This step would not be required, if the theory is correct; now it is safe to start TB again).

7. Report your result in this bug.
I am able to reproduce the problem.  Notable clues are multiple extensions installed and a reboot.  It is not an issue of sync failing.

I started with 38.4 installed with lightning-4.0.4.1-sm+tb-windows.xpi and lookout-1.2.13-sm+tb.xpi extensions. chrome.manifest is 12662 bytes, Dec 30, 17:27 timestamp.

I upgraded to 38.5.  Thunderbird worked fine.  chrome.manifest is 5725 bytes, Dec 21, 15:10 timestamp. 

I rebooted.  No inbox. chrome.manifest is 12662 bytes, Dec 30 17:27 timestamp. It is identical to the before upgrade chrome.manifest.  

Also, when upgrading with only the calendar extension installed, which worked correctly, I noticed during the upgrade process, the {e2fda1a4-762b-4020-b5ad-a41df1933103} is completely removed.  With multiple plugins, {e2fda1a4-762b-4020-b5ad-a41df1933103} is left in place with only a few files.  

My question right now is: Is the lookout extension toxic or is it any combination of multiple plugins?
John, thanks for testing. Some further questions:

(In reply to John Kaiser from comment #50)
> I am able to reproduce the problem.  Notable clues are multiple extensions
> installed and a reboot.  It is not an issue of sync failing.

Did you come to this conclusion because your test system doesn't sync with another system at all or based on the described results? 

> I started with 38.4 installed with lightning-4.0.4.1-sm+tb-windows.xpi and
> lookout-1.2.13-sm+tb.xpi extensions. chrome.manifest is 12662 bytes, Dec 30,
> 17:27 timestamp.
> 
> I upgraded to 38.5.  Thunderbird worked fine.  chrome.manifest is 5725
> bytes, Dec 21, 15:10 timestamp. 
> 
> I rebooted.  No inbox. chrome.manifest is 12662 bytes, Dec 30 17:27
> timestamp. It is identical to the before upgrade chrome.manifest.

Have you only checked after reboot or before also after shutdown/restart TB? It would be a valuable information if TB behaves the same after an application restart only or the reboot is required to make the issue to show up.

> Also, when upgrading with only the calendar extension installed, which
> worked correctly, I noticed during the upgrade process, the
> {e2fda1a4-762b-4020-b5ad-a41df1933103} is completely removed.  With multiple
> plugins, {e2fda1a4-762b-4020-b5ad-a41df1933103} is left in place with only a
> few files.  
>
> My question right now is: Is the lookout extension toxic or is it any
> combination of multiple plugins?

You have only installed lookout as additional extension? Are the other addon(s) installed from amo oder also as distributed addons? I assume lookout has been already up-to-date for your 38.4.0 installation?

Additional note: if you have open the Lightning folder in the TB profile using a file explorer while doing the update test that may disturbe the complete removal of the existing folder and therefore invalidate your test results. Are you sure this did not happen?
(In reply to MakeMyDay from comment #49)
> My current best guess on this issue is that it is a synchronization issue
> with server side Windows profiles (most of the reports are from organization
> and mostly limited to Windows; we haven't found any obvious reason in our
> code so far, so it's more likely to have a reason in the user's
> environment). Also other options for file synchronization are candidates.
> 
> MAR updates don't update the chrome.manifest in distribution/extensions
> (which is correct as it has not changed from perspective of automated
> builds), so when Lightning is installed into the user profile, the
> chrome.manifest then has an older timestamp than that of a previously
> installed AMO version.
> 
> From file sync perspective, this would mean the updated chrome.manifest is
> the outdated version that needs to be replaced by the serverside version (in
> fact the AMO version), as at both locations the file exists and usually the
> newer one is persisted.
> 
> If this is it, we could easily e.g. add a comment line referring to the
> current version into the chrome.manifest which we update during the build
> process automaically to assure we have always a up-to-date version of the
> chrome.manifest.
> 
> 
> As I cannot reproduce the issue myself, I would need an affected user to do
> a simple simple test to confirm or rebut the above on that very system
> (maybe you will need assistance from your IT support):
> 
> 1. Assuming you have a working Lightning, shut down TB if running.
> 
> 2. Copy your current chrome.manifest located in the local copy of
> <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} to a place
> elsewhere
> 
> 3. Copy the chrome.manifest from
> <tb-app-dir>/distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}
> to the local copy of
> <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} and cross
> check the copied file still has the same timestamp as in the source folder
> (this step would break Lightning from running at this point in time, so
> don't restart TB until the test is completed)
> 
> 4. Trigger file synchronization (depending on implementation, this may be a
> Windows logoff/logon or a serverside triggered background operation) in an
> appropriate way.
> 
> 5. Check the chrome.manifest file in
> <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} - is this
> still the copied version or replaced by the previous version, you have
> backed up before (check timestamp and content)?
> 
> 6. In the former case, replace the chrome.manifest in
> <tb-profile>/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} with the
> backed up copy again before starting TB again. (This step would not be
> required, if the theory is correct; now it is safe to start TB again).
> 
> 7. Report your result in this bug.

I haven't tried your suggested test yet but I will when occasion will present to me (most probably next week).

At the meantime here are some additional inputs https://bugzilla.mozilla.org/show_bug.cgi?id=1211358#c23 that may be of interest.
As indicated in my comment https://bugzilla.mozilla.org/show_bug.cgi?id=1211358#c24, it seems issue may well be linked to a combination of Thunderbird upgrade (with Lightening extension) and usage of roaming profile indeed. Issue is occurring only after first logoff/logon following the upgrade. Keep you posted about further test, hopefully next week.
A lot of Danish users have reported this lately at MozillaDanmarks support site.
I just opened Thunderbird with a profile I use for testing. Lightning version is 4.0.5.1. Lightning at AMO is 4.0.4.1. Why the difference in versions?
(In reply to Jørgen Rasmussen from comment #54)
> I just opened Thunderbird with a profile I use for testing. Lightning
> version is 4.0.5.1. Lightning at AMO is 4.0.4.1. Why the difference in
> versions?

The version change is a fudge to stop a second update from AMO.  The version bundled with Thunderbird gets incremented point zero one in the version.  It is however an identical build.

Are those affected using roaming profiles?  Streaming backups or anything that might see part of the profile arrive late to the party and replace already updated files.
(In reply to Matt from comment #55)
> (In reply to Jørgen Rasmussen from comment #54)
> Are those affected using roaming profiles?  Streaming backups or anything
> that might see part of the profile arrive late to the party and replace
> already updated files.

I don't have much information from the affected users. I think at least one of the users profile is located at the default location (Windows 7).
Other users have reported, that deleting {e2fda1a4-762b-4020-b5ad-a41df1933103} form the extensions folder in the the profile solved their problems.
One user's problem disappeared when Thunderbird updated to version 38.5.1
This bug occurs on all the computers, all the users profiles after all of the updates.

For me it seems really linked to the chrome.manifest update management in a roaming profiles environnement.

On my personnal computer with a local profile, I don't have the bug. 

So I use my working version of chrome.manifest in a script and I replace all the "wrong" chrome.manifest on the computers and the Domain Controller profiles.

And then it works.
Whiteboard: DUPETOME
The working chrome.manifest is a 6 ko file
All the disfunctional chrome.manifest are 12.5 ko.

(In reply to François Alix from comment #57)
> This bug occurs on all the computers, all the users profiles after all of
> the updates.
> 
> For me it seems really linked to the chrome.manifest update management in a
> roaming profiles environnement.
> 
> On my personnal computer with a local profile, I don't have the bug. 
> 
> So I use my working version of chrome.manifest in a script and I replace all
> the "wrong" chrome.manifest on the computers and the Domain Controller
> profiles.
> 
> And then it works.
See attached test result as per request in Comment 49.
(Had to attach as a .zip file as .pdf not accepted)
Richard, thanks for testing. At a first glance, this seems to confirm the expected root cause I'll have a closer look later on.
I've setup an AD roaming profile environment on AWS with a Domain Controller and a client. I can now reliably reproduce this.

The underlying issue is that AMO updates and distribution updates handle the filedates differently. AMO updates give each file in the profile the date/time when the update occurs. Distribution updates preserve the modification time of the original files, so that files that are not changed between versions are left with old dates.

Roaming profile sync is not copying files with older modify dates into the synced profile storage location.

So this sequence results in a state where I see the mail folders, but clicking on them does not display their messages:

1) Install Thunderbird 38.3.0 (giving a 6K chrome.manifest with 9/2015 date)
2) Logoff (which copies the 9/2015 chrome.manifest to the file server)
3) Login, and update extensions in Thunderbird. Newer Lightning installs (4.0.4.1), chrome.manifest is 13K with today's date as well as all other Lightning files.
4) Logoff, all of the Lightning extension files get copied to the file server with today's date.
5) Login, and update Thunderbird (which finds 38.5.0). Lightning files in the profile directory are now mostly older dates, with just a few newer dates of modified files. Chrome.manifest is 6K with a 9/2015 date.
6) Logoff. The older files are not copied to the file server, so the file server shows mostly the 4.0.4.1 files, but a few files are updated with the changed 4.0.5 versions.
7) Login. The Lightning profile now gets updated with the "newer" 4.0.4.1 files from the file server, except for a few changed 4.0.5 files.

At this point, if I run Thunderbird, I can see all of the message folders, but if I click on those folders I do not get the thread pane display, so I cannot actually view any message files.

I think that solution that has already been suggested, to update the dates of the all of the files in the profile when copying from distribution/extensions, should solve the problem.

Both the server and the client on AWS can be accessed as remote profiles if one of the Lightning developers wants to play with this, or I can be the tester if you prefer. I have reproduced this multiple times using the same client bu uninstalling Thunderbird and profiles, and starting over from 38.3.0.
Better, for Win7... more finally, way?
Create under "C:\Users\...\AppData\Local\Mozilla\Thunderbird\" a new folder - the current profile (with an unequivocal name) moves there. Assign with command "thunderbird.exe -ProfileManager" this to Thunderbird.
mossop, does the proposed solution in comment 62 raise any red flags from your perspective?  Is there something we should change in addons manager?
Flags: needinfo?(dtownsend)
Attached patch FixUpdateIssue-V1.diff (obsolete) — Splinter Review
Possible patch to get files and directories timestamps updated regradless of the source of installation in the xpi provider.

If such a change is not appropriate for mozilla-central/mozilla-esr38, we maybe can wrap this with #ifdef MOZ_THUNDERBIRD.

It's probably tricky to get a fix for this issue tested for the usual mar update scenario.
Attachment #8708571 - Flags: feedback?(rkent)
Attached patch FixUpdateIssue-V2.diff (obsolete) — Splinter Review
Updated using the correct object now.
Attachment #8708571 - Attachment is obsolete: true
Attachment #8708571 - Flags: feedback?(rkent)
Attachment #8708580 - Flags: feedback?(rkent)
Update to reproducing: I can reproduce using standard Thunderbird build downloads, without having to do Thunderbird updates.

Basic plan:

1) Install Thunderbird 38.3.0 and Thunderbird 39.5.0, change channel prefs to default to prevent updates.
2) Run 38.3.0, creating a new profile.
3) Update Lighting on that profile (which updates to Lightning 4.0.4.1)
4) Quit and logout (which adds Lightning files with current date to roaming profile store).
5) Login, run Thunderbird 35.5.0. Now Lightning 4.0.5 is installed, chrome.manifest date is 12/21/2015
6) Quit and logout
7) Login, run Thunderbird 38.5.0  As before, the Lightning extension is now borked, clicking on a folder does not show any messages in the thread pane.

So I should be able to test MakemyDays' patch without having to concern myself with MAR files and Mozilla updates.
Comment on attachment 8708580 [details] [diff] [review]
FixUpdateIssue-V2.diff

Review of attachment 8708580 [details] [diff] [review]:
-----------------------------------------------------------------

Nice try, but this does not work. I'll add my change that fixes the issues.

::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
@@ +406,4 @@
>    _installFile: function(aFile, aTargetDirectory, aCopy) {
>      let oldFile = aCopy ? null : aFile.clone();
>      let newFile = aFile.clone();
> +    newFile.lastModifiedTime = Date.now();

At this point,newFile is just oldFile, so you are trying to update the time of a file in the installation location, not the profile. This fails if the installation location is not writable by the user, which is the typical case. Plus, that is not what we want, we want the profile location to be updated.

@@ +424,4 @@
>    _installDirectory: function(aDirectory, aTargetDirectory, aCopy) {
>      let newDir = aTargetDirectory.clone();
>      newDir.append(aDirectory.leafName);
> +    newDir.lastModifiedTime = Date.now();

This is being done in a place where the directory is always being created, so this always fails because newDir does not exist.
Attachment #8708580 - Flags: feedback?(rkent) → feedback-
Here's my version. At this point I think I'll take the assignment.

I think that we should ask for this on mozilla-central, and there is no reason that should not accept it. But if that is delayed, I'll ask for a review from :Fallen prior to landing on our esr38 branch. I don't think we should do another esr38 release without fixing this.
Assignee: nobody → rkent
Status: NEW → ASSIGNED
Attachment #8708707 - Flags: feedback?(philipp)
Attachment #8708707 - Flags: feedback?(makemyday)
Debug hints:

1) Set extensions.logging.enabled true to see debug output from addon install
2) To force an attempt at updating the distribution extensions, set extensions.lastAppVersion to an older version, say 38.4.0
Comment on attachment 8708707 [details] [diff] [review]
V3, properly generate newFile before we update its time.

Review of attachment 8708707 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for updating the patch.

> Here's my version. At this point I think I'll take the assignment.

You're welcome. You should also update the patch header, then.

> I think that we should ask for this on mozilla-central, and there is no
> reason that should not accept it. But if that is delayed, I'll ask for a
> review from :Fallen prior to landing on our esr38 branch. I don't think we
> should do another esr38 release without fixing this.

I fully agree, we definitely should have that in the upcoming esr release.

::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
@@ +419,5 @@
> +      try {
> +      // Windows roaming profiles won't properly sync directories if a new file
> +      // has an older lastModifiedTime than a previous file, so update. But
> +      // don't abort if this fails.
> +        newFile.lastModifiedTime = Date.now();

Shouldn't this inner try/catch block to update the timestamp better go into the copy branch of the previous if/else block? Installing a distro addon from a directory uses the copy method and this should be the only affected case this fix is about. Furthermore, the moveTo is also used for rollback operations and we probably don't want to update the timestamp in that case.
Attachment #8708707 - Flags: feedback?(makemyday) → feedback+
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #64)
> mossop, does the proposed solution in comment 62 raise any red flags from
> your perspective?  Is there something we should change in addons manager?

It's probably more correct to set the last modified times of add-on's files when extracting the updates from AMO but that still wouldn't fix the issue here in some cases so I guess that's what we have to do.
Flags: needinfo?(dtownsend)
Comment on attachment 8708707 [details] [diff] [review]
V3, properly generate newFile before we update its time.

Review of attachment 8708707 [details] [diff] [review]:
-----------------------------------------------------------------

Other than the comments this is looking ok but I'd really like to see some tests here. Particularly add a couple of checks in test_install.js, test_update.js and test_distribution.js that an installed add-on's file has a recent file modification time.

::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
@@ +419,5 @@
> +      try {
> +      // Windows roaming profiles won't properly sync directories if a new file
> +      // has an older lastModifiedTime than a previous file, so update. But
> +      // don't abort if this fails.
> +        newFile.lastModifiedTime = Date.now();

We definitely don't want to update the modification times for stuff that might get rolled back, other than that I'd be fine to do this for both moves and copies for consistency but maybe only for copies is the easier way to deal with that.
Attachment #8708707 - Flags: feedback?(philipp) → feedback+
Attached patch V4, adding test (obsolete) — Splinter Review
I added a test, but only in a single place. This issue is really only relevant in distribution installs, so I added it there.

I don't really understand "We definitely don't want to update the modification times for stuff that might get rolled back, other than that I'd be fine to do this for both moves and copies for consistency but maybe only for copies is the easier way to deal with that." It sounds like you are saying the patch is fine?

There were also two drive-by fixes here: typo using Date.now (missing parenthesis), and correcting newFile for the issue that it does not get properly updated on copy. Both of those might have other effects beyond this bug.
Attachment #8708580 - Attachment is obsolete: true
Attachment #8708707 - Attachment is obsolete: true
Attachment #8710786 - Flags: review?(dtownsend)
Comment on attachment 8710786 [details] [diff] [review]
V4, adding test

Review of attachment 8710786 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Kent James (:rkent) from comment #74)
> Created attachment 8710786 [details] [diff] [review]
> V4, adding test
> 
> I added a test, but only in a single place. This issue is really only
> relevant in distribution installs, so I added it there.

Right but for things to keep working in the future we need to always install add-ons with recent file modification times so I want tests in the other places to verify that that doesn't ever break. Had I not read this bug for instance I would have expected to fix ZipUtils to set the correct file modification time on extraction. If anyone does that we want a test to fail.

> I don't really understand "We definitely don't want to update the
> modification times for stuff that might get rolled back, other than that I'd
> be fine to do this for both moves and copies for consistency but maybe only
> for copies is the easier way to deal with that." It sounds like you are
> saying the patch is fine?

No, at the moment the file modification time gets updating on add-on's files that are moved aside in preparation for installing an updated version. If that update fails the files will be moved back with their new modification time. This can have ramifications for sideloading detection as well as presumably causing roaming profiles to get updated needlessly.
Attachment #8710786 - Flags: review?(dtownsend)
Is the approach taken in the existing test changes appropriate?
Flags: needinfo?(dtownsend)
Comment on attachment 8710786 [details] [diff] [review]
V4, adding test

Review of attachment 8710786 [details] [diff] [review]:
-----------------------------------------------------------------

f+ for the approach in the test, just a couple of notes.

::: toolkit/mozapps/extensions/test/xpcshell/test_distribution.js
@@ +107,3 @@
>  function run_test_1() {
>    writeInstallRDFForExtension(addon1_1, distroDir);
> +  let oldTime = Date.now() - 100000;

Please define this as a constant at the top so it's next to MAX_TIME_DIFFERENCE

@@ +107,4 @@
>  function run_test_1() {
>    writeInstallRDFForExtension(addon1_1, distroDir);
> +  let oldTime = Date.now() - 100000;
> +  setDistributionModificationTime(addon1_1, distroDir, oldTime);

writeInstallRDFForExtension returns the nsIFile for the extension so you can just pass that to setExtensionModified time rather than writing your own function here.
Attachment #8710786 - Flags: feedback+
Flags: needinfo?(dtownsend)
I tried to add some testing in install and update, though those areas AFAICT were not affected by the current issue.

On my Windows 10 system, I seem to be getting test failures due to problems with file.remove(). I fixed that in the one test that I modified, but there are a few other tests where I also see this (at least test_dictionary.js and test_bootstrap.js) I don't want to expand the scope of this extremely urgent bug, that has already been delayed several days by the request to expand testing, by investigating that further.

Try server run https://treeherder.mozilla.org/#/jobs?repo=try&revision=6793c70d7070 is not showing any new failures AFAICT.
Attachment #8672124 - Attachment is obsolete: true
Attachment #8710786 - Attachment is obsolete: true
Attachment #8714475 - Flags: review?(dtownsend)
Comment on attachment 8714475 [details] [diff] [review]
Add tests in install and update

Philipp, if there is any way that you can look at this for a review+ so that I can land this on a branch of mozilla-beta in preparation for our first Thunderbird 45 beta, that would be great. I hope that we can get the official m-c review and landing done prior to pushing to comm-esr38 for a release.
Attachment #8714475 - Flags: review?(philipp)
Comment on attachment 8714475 [details] [diff] [review]
Add tests in install and update

Review of attachment 8714475 [details] [diff] [review]:
-----------------------------------------------------------------

As far as I understand the extension manager code this looks fine to me. I'm going to give you f+ for this issue since I am not a reviewer in that area, but I nevertheless approve of pushing this to a release branch like THUNDERBIRD450b1_2016020122_RELBRANCH. It would be nice to get this one fixed and officially approved for backporting so we don't have to carry this around for another 7 releases as we did for Thunderbird 38.

Also, given the patch is now in toolkit maybe we should move and rename this bug.
Attachment #8714475 - Flags: review?(philipp) → feedback+
Comment on attachment 8714475 [details] [diff] [review]
Add tests in install and update

Review of attachment 8714475 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
@@ +414,5 @@
> +        newFile = aTargetDirectory.clone();
> +        newFile.append(aFile.leafName);
> +        // Windows roaming profiles won't properly sync directories if a new file
> +        // has an older lastModifiedTime than a previous file, so update. But
> +        // don't abort if this fails.

You say don't abort if it fails but it will. You need another try...catch around this.
Attachment #8714475 - Flags: review?(dtownsend) → review+
(In reply to Dave Townsend [:mossop] from comment #81)
> Comment on attachment 8714475 [details] [diff] [review]
> Add tests in install and update
> 
> Review of attachment 8714475 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
> @@ +414,5 @@
> > +        newFile = aTargetDirectory.clone();
> > +        newFile.append(aFile.leafName);
> > +        // Windows roaming profiles won't properly sync directories if a new file
> > +        // has an older lastModifiedTime than a previous file, so update. But
> > +        // don't abort if this fails.
> 
> You say don't abort if it fails but it will. You need another try...catch
> around this.

Hmmm, I decided not to bother with the "don't abort" but did not remove the comment. Is it OK if I just get rid of the comment?
(In reply to Kent James (:rkent) from comment #82)
> (In reply to Dave Townsend [:mossop] from comment #81)
> > Comment on attachment 8714475 [details] [diff] [review]
> > Add tests in install and update
> > 
> > Review of attachment 8714475 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: toolkit/mozapps/extensions/internal/XPIProvider.jsm
> > @@ +414,5 @@
> > > +        newFile = aTargetDirectory.clone();
> > > +        newFile.append(aFile.leafName);
> > > +        // Windows roaming profiles won't properly sync directories if a new file
> > > +        // has an older lastModifiedTime than a previous file, so update. But
> > > +        // don't abort if this fails.
> > 
> > You say don't abort if it fails but it will. You need another try...catch
> > around this.
> 
> Hmmm, I decided not to bother with the "don't abort" but did not remove the
> comment. Is it OK if I just get rid of the comment?

Yes, that's fine.
Component: General → Add-ons Manager
Product: Calendar → Toolkit
Version: Lightning 4.0.3 → Trunk
Flags: in-testsuite+
Keywords: regression
OS: Unspecified → All
Hardware: Unspecified → All
https://hg.mozilla.org/mozilla-central/rev/7068a4265c8e
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
On behalf of sys admins and end-users, thank you so much for your time and efforts to sort this issue.
Your work is much appreciated.
I confirm that 38.6.0 has fixed this problem.  Thanks for all your diligent work Kent !!
Depends on: 1249894
Pushed to http://hg.mozilla.org/releases/mozilla-esr45/rev/740989d8336f branch THUNDERBIRD_45_VERBRANCH so this will be in Thunderbird 45.
You need to log in before you can comment on or make changes to this bug.