If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

popstate-N.dat folders appear in folder pane

NEW
Unassigned

Status

MailNews Core
Networking: POP
3 years ago
3 hours ago

People

(Reporter: maybe-the-one, Unassigned)

Tracking

({regression, regressionwindow-wanted})

x86_64
Windows 8.1
regression, regressionwindow-wanted

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [regression:TB26])

User Story

citations:
https://support.mozilla.org/en-US/questions/1013503 (first report found, 8/1/2014)
https://support.mozilla.org/en-US/questions/1016811
https://support.mozilla.org/en-US/questions/1037008
https://support.mozilla.org/en-US/questions/1037795
https://support.mozilla.org/en-US/questions/1038406
https://support.mozilla.org/en-US/questions/1040289
http://forums.mozillazine.org/viewtopic.php?f=39&t=2908757
http://forums.mozillazine.org/viewtopic.php?f=39&t=2957159

user OK/solved - perhaps
https://support.mozilla.org/en-US/questions/1026288

user has profile in dropbox
https://support.mozilla.org/en-US/questions/1038967

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
Build ID: 20140923175406

Steps to reproduce:

Cannot reproduce at will.  Often, popstate.dat folders appear in the left folder pane under accounts--SEEMINGLY (but not certainly) under accounts that have messages retained on the server when downloaded to Thunderbird (POP).


Actual results:

Randomly, popstate,dat files appear under accounts in the left folder pane.


Expected results:

No popstate.dat folders should be create by Thunderbird in the left folder pane.
Why do you state explicitly for accounts with "messages retained on the server"?
Flags: needinfo?(fm_mozbugs)
Also, please check tools | error console
(Reporter)

Comment 3

3 years ago
'Why do you state explicitly for accounts with "messages retained on the server"? '

Because my recollection is that the popstste.dat folders have only appeared under accounts that are set that way--but I am not 100% certain of that.  I cannot look and see because I delete them when they appear as the drive folders I use all the time "off the bottom of the list" and deleting them means I do not have to scroll every time to get to them.

"Also, please check tools | error console"

I looked for a way to export them to provide the list, but there is no such facility...and you cannot highlight the whole list to copy and paste; only one at a time can be hilited.
(Reporter)

Comment 4

3 years ago
Morning of 11-30-14, more than ever appeared in one account.  Before, two was the most I had seen; this time, six appeared under one account.  Sometimes they are in more than one account.  See arch "pic-to-show."
Flags: needinfo?(fm_mozbugs)
(Reporter)

Comment 5

3 years ago
Created attachment 8530633 [details]
pic-to-show.png
(Reporter)

Comment 6

3 years ago
Ah, there were also three under another account further down; 
I had not scrolled down.  
(Wish we could edit posts.)
What is the size of these postate-N.dat files on disk?
Does problem go away if you change back to version 24?
 download from https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/24.6.0/win32/en-US/
(Reporter)

Comment 8

3 years ago
Created attachment 8530654 [details]
popstate sizes pic
See https://support.mozilla.org/en-US/questions/1016811 

Does it help?
(Reporter)

Comment 10

3 years ago
(In reply to Wayne Mery (:wsmwk) from comment #9)
> See https://support.mozilla.org/en-US/questions/1016811 
> 
> Does it help?

No.   I had seen that previously and had renamed those seven days ago.
I added "(dot)old" to the foldertree.json and panacea.dat files, 
i.e. foldertree.json.old and panacea.dat.old.

The popstate.dat files have reappeared several times since then.
Correct one is "popstate-N.dat folder" instead. of "popstate.dat folder". Tb never shows "popstate.dat" as folder.

FYI.
Such phenomenon is easily observed by : Remove write permission of popstate.dat file.
Upon updating popstate.dat file, Tb does (1) "write data to popstate-N.dat" and (2) "delete popstate.dat+rename popstate-N.dat to popstate.dat".
If you interfere (2) by removing Tb's write permission on popstate.dat, (2) fails, so popstate-N.dat remains.
If Tb is restarted when popstate-N.dat remains, it's shown as mail folder, because it's not "popstate.dat".

Comment 12

3 years ago
So if the rename fails, we should remove the temporary popstate-N.dat file immediately so it does not show up after restart. Yes, that would loose the changes to it. So we should not attempt a download of messages in the first place if popstate.dat is not writable. That should be doable.

But first see if some antivirus is not blocking the file.
(Reporter)

Comment 13

3 years ago
That describes how the phenomenon MIGHT POSSIBLY be happening, but I checked all of the popstate.dat files, and none are read-only, plus they all have all permissions authorized (in Security tab) for System, the logged in user, and administrators, except "Special permissions."
(Reporter)

Comment 14

3 years ago
Gosh, I wish we could edit posts...

Forgot to say that the antivirus has not quarantined popstate.dat (nor any files for that matter) which is what it would do if it decided the file was suspicious.
It would not be necessary for the file to be quarantined for the AV to have interferred.

Since it is easy for you to reproduce, I suggest...

Start *Windows'* safe mode with networking enabled
- win8 http://windows.microsoft.com/en-us/windows-8/windows-startup-settings-safe-mode

Still In Windows safe mode, start thunderbird in safe mode
- http://support.mozillamessaging.com/en-US/kb/safe-mode

Does problem go away?
(Reporter)

Comment 16

3 years ago
It is not "easy" to reproduce.
It happens randomly.  I cannot tell when they are going to appear.
I would estimate that they show up about four days of the week.
averaging 4 days out of 7 is frequent enough, even though you can't reproduce on demand.
SO hopefully running safe mode for 1-2 days is possibel for you.
(In reply to maybe-the-one from comment #16)
> It is not "easy" to reproduce. It happens randomly.  I cannot tell when they are going to appear.

Who accesses popstate.dat when?
Watching popstate.dat file access by Process Monitor is usually helpful.
> http://technet.microsoft.com/en-us/sysinternals/bb896645
> Filter : If path ends with \popstate.dat, Include. If path contains \popstate-, Include.
Virus scan by AV software, Auto-backup software, are two major culprit of "interfere Tb's file access".
"Meaningless backup merely by archive bit of file with short interval" can easily interfere Tb's file access, because Tb frequetly updates file content while Tb is running.

If popstaate-N.dat folder appeared as message folder after restart of Tb, delete such folder manually. Because it's garbage, there is no need to keep it.
(In reply to Wayne Mery (:wsmwk) from comment #17)
> averaging 4 days out of 7 is frequent enough, even though you can't
> reproduce on demand.
> SO hopefully running safe mode for 1-2 days is possibel for you.

needs test in windows safe mode
Flags: needinfo?(fm_mozbugs)
Whiteboard: [needs test in Windows Safe Mode]
(Reporter)

Comment 20

3 years ago
(In reply to Wayne Mery (:wsmwk) from comment #19)
> (In reply to Wayne Mery (:wsmwk) from comment #17)
> > averaging 4 days out of 7 is frequent enough, even though you can't
> > reproduce on demand.
> > SO hopefully running safe mode for 1-2 days is possible for you.
> 
> needs test in windows safe mode

That would be hard to do...to keep the computer running in safe mode until it happened (or long enough to say it isn't going to).  << especially true since the pattern seems to have changed.  

It has been (guess) abut five days since the last occurrence.  I will try to keep a log of times it happens to get a better handle on frequency.
maybe-the-one,

does the suggestion at https://support.mozilla.org/de/questions/1016811#answer-619413 work for you??

help menu (alt+H) > troubleshooting information  click on the show folder button
Close Thunderbird.
Delete the files foldertree.json and panacea.dat (or rename them for added backout security)
Restart Thunderbird.
(Reporter)

Comment 22

3 years ago
See my previous comment about this same thing...

"I had seen that previously and had renamed those seven days ago.
I added "(dot)old" to the foldertree.json and panacea.dat files, 
i.e. foldertree.json.old and panacea.dat.old.

The popstate.dat files have reappeared several times since then."
Still true.

I think that other referenced thread describing exactly the same thing confirms that my case is not unique.
Flags: needinfo?(fm_mozbugs)
ah yes. Too bad, I thought I had found something new.
Yes, not unique. But still, "cause unknown".

citations:
https://support.mozilla.org/en-US/questions/1013503 (first report found, 8/1/2014)
https://support.mozilla.org/en-US/questions/1016811
https://support.mozilla.org/en-US/questions/1037008
https://support.mozilla.org/en-US/questions/1037795
https://support.mozilla.org/en-US/questions/1038406
https://support.mozilla.org/en-US/questions/1040289
http://forums.mozillazine.org/viewtopic.php?f=39&t=2908757

user OK/solved - perhaps
https://support.mozilla.org/en-US/questions/1026288

user has profile in dropbox
https://support.mozilla.org/en-US/questions/1038967
Component: Untriaged → Networking: POP
Keywords: regression, regressionwindow-wanted
Product: Thunderbird → MailNews Core
Whiteboard: [needs test in Windows Safe Mode] → [regression??][needs test in Windows Safe Mode]
What do you make of this?  Perhaps a regression from bug 239455?

Not pervasive. But certainly seems like a change in behavior for version 31.
Flags: needinfo?(buchner.johannes)
Why still no information about file access of popstate.dat/popstate-N.dat?
If process monitor like one is used, (a) "who accessed popstate.dat while getting log is Tb only" or (b) "someone else periodiclly accessed popstate.dat while getting log" can pretty easily be known.
If (a), problem like following is supected.
   Contention among :  (in popstate.dat, K: Keep, P: Partial, D: Delere, is held for each UIDL)
       (i)   pop3 download
       (ii)   update by filter(for "delete from POP3 server" just before next pop3 access)
       (iii)  "delete from POP3 server" just before pop3 access for new mail check
       (iv) download of entire mail data for fetch heder only mail
   IIUC, event occurs in next order
       (iii) by previous download/filtering -> (i) -> (ii) by filter for this downlod
       (iv) occurs independently, and can occur at same time, at any time, because it's invoked by user's click of "Click Here".
       If manual filter run is executed, (ii) also can occur at same time, at any time.
And, if non-deleted popstate-N.dat fortuntely happens while getting log, "when/by whoom the non-deleted popstate-N.dat was generated" can be known.

Another concern.
  This bug was reported for Tb 31.
  It seems somehow changed recently.
  popstte-N.dat and timestamp
    On 11/29, many garbage waere generated.
      popstate-1.dat           11/29/14  3:10 PM
      popstate-2.dat           11/29/14  4:10 PM
      popstate-3.dat           11/29/14  4:44 PM
      popstate-4.dat           11/29/14  7:32 PM
      popstate-5.dat           11/29/14 10:20 PM
      popstate-6.dat           11/29/14 10:52 PM
  Next day, popstte.dat was normally written.
      popstate.dat              11/30/14  1:17 PM
  Some one kept popstate.dat file open from 3:00 PM to 23:00 PM, and closed at 24:00 PM?
It may be following, 
   once popstate-1.dat is used(for example, due to read open by someone),
  "write popstate-N.dat" + "rename popstate-N.dat popstate.dat(with replace mode)" won't work as expected.
because "create unique file name" seems had bug in Tb 31, and it seems to have been resolved recently,
and because "creation of popstte-N.dat" uses the "create unique file name"...
Summary: popstate.dat folders appear in folder pane → popstate-N.dat folders appear in folder pane
FYI.
Suspected CreateUnique issue.
   When two tasks tries to update popstate.dat, two tasks requests unique temporry file name.
        Example of two tasks : pop3 download, and full download of fetch header only mail.
   To task #1, popstate-1.dat is returned.
   Because popstate-1.dat is not created yet on HDD, popstate-1.dat is returned to Task #2.
   Two tasks tries to write popstate-1.dat, and final "rename popstate-1.dat popstte.dat(replace mode)" fails.
If this kind of issue, "Write permission", "Interfere by other software" are absolutely  irrelevantto problem.

Comment 27

3 years ago
(In reply to Wayne Mery (:wsmwk) from comment #24)
> What do you make of this?  Perhaps a regression from bug 239455?
> 
> Not pervasive. But certainly seems like a change in behavior for version 31.

Yes, perhaps; but I don't understand why *folders* are being produced -- should be only files. Is that a Windows-specific implementation of NS_NewSafeLocalFileOutputStream (nsSafeFileOutputStreamConstructor)?
The next thing I don't understand is why folders would show up automatically in the folders pane -- is that a normal Thunderbird feature?
I also thought the IDs in the filename would be generated randomly, not in order. I suspect this is a odd Windows implementation problem of the SafeFileOutputStream, which is now triggered by using it more frequently (e.g. for popstate.dat).
Hope that helps tracking down the problem.
Flags: needinfo?(buchner.johannes)
(In reply to Johannes Buchner from comment #27)
> The next thing I don't understand is why folders would show up automatically in the folders pane -- is that a normal Thunderbird feature?

This is design/spec of Tb.
If you create file named ABC while Tb is not running, upon restart of Tb, Tb treats it as "file for folder named ABC", and creates "ABC.msf" for it.
Reserved name is xxx.sbd etc. only. So, "file named popstate-N.dat file is treated as "file for fiolder named nstmp-N.dat" upon restart of Tb.

> I also thought the IDs in the filename would be generated randomly, not in order.
> I suspect this is a odd Windows implementation problem of the SafeFileOutputStream, which is now triggered by using it more frequently
> (e.g. for popstate.dat).

As I wrote in coment #25, popstate-N.dat is creaated in order of N in bug opener's case.
Text dat of File naame,s ize, timestamp etc. is pretty easily obtained.
   At Commnt prompt,
      CD:C:\...\Mail\serername
      DIR popstate*.* > C:WORK\DIR.TXT
      notepad.exe  C:\WORK\DIR.TXT
Data like following is held in popstate.dat(and  popstate-N.dat too).
   UIDL state : where stat = K(already downloaded), P8partial, fetch header only), D(should be deleted from POP3 server upon next login)
So, "Which UIDL is added/removed" can be known merely by "comparison of popstate.dat content and popstate-N.dat" content".

Actully created in what order in your environment?

> Yes, perhaps; but I don't understand why *folders* are being produced --

Phenoenon itself is simple.
By safe file writing, following is executed.
1. Obtain unique file name for popstate.dat update:  logic like next.
    for(N=0;N<10000;N++){if(N==0) FN="popstate.dat";else FN="popstate-"+N+".dat; if FN is free return FN; else N++; }
2. Write new version of popstate.dat to obtained popstate-N.dat
3. Rename popstate-N.dat to popstate.dat with replace mode.
In above steps, if something bad happens in step 2/step3, popstate.dat is not updated and popstate-N.dat is not removed.
Problem is : data to analyze "What is the something bad" is not provided from anyone yet.
To test if bug 239455's checkin is involved the builds to be tested are...
https://archive.mozilla.org/pub/thunderbird/nightly/2013/09/2013-09-05-04-59-33-comm-central/ (should work) and https://archive.mozilla.org/pub/thunderbird/nightly/2013/09/2013-09-11-03-03-41-comm-central/ (should fail).  (the other nightly builds adjacent to 2013-09-09 are unfortunately broken)

xref http://forums.mozillazine.org/viewtopic.php?f=39&t=2961371
Blocks: 239455
Flags: needinfo?(vseerror)
Whiteboard: [regression??][needs test in Windows Safe Mode] → [regression:TB26??][needs test in Windows Safe Mode]
(User in mozillazine (comment 29) tested the nightly builds and confirms that 2013-09-11 is broken - so sort of confirm this is a regression of bug 239445.)

Aceman, If your patch in bug 789679 doesn't help this bug, would you be able to take a look at this? Or do we need to consulte Johannes?  

And does bug 1144020 really block doing  bug 789679?   
Or is bug 1144020 merely an imperative for maildir users?

Then perhaps we move on to bug 1054232.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(vseerror) → needinfo?(acelists)
Whiteboard: [regression:TB26??][needs test in Windows Safe Mode] → [regression:TB26]
another example - http://forums.mozillazine.org/viewtopic.php?f=39&t=2957159
User Story: (updated)

Comment 32

2 years ago
Could someone point out where the Windows implementation of SafeFileOutputStream lives?
(In reply to Johannes Buchner from comment #32)
> Could someone point out where the Windows implementation of
> SafeFileOutputStream lives?

ishikawa probably knows
Flags: needinfo?(ishikawa)

Comment 34

2 years ago
Hi,
I know more about the linux(i.e., POSIX) implementation.
But since I am about to investigate the issue of read(fid, buf, size) returnning 0 for end-of-file condition under MacOS (which seems fishy to me), 
let me check the window implementation as well and come back to you.
That the implementation of PR_Read() under linux and Windows did not cause the
loop due to read() returning 0 at EOF condition looks very suspicious to me (maybe there is a bug in MacOS under a strange condition or the PR_Read() implementation under MacOS does not handles this subtle issue of strange |read| behavior and 

I agree the low-level implementation is very hard to track. I have not figured out where the mapping of |PR_Read| to |read| is actually handled very clearly even :-(
Flags: needinfo?(ishikawa)
(In reply to ISHIKAWA, Chiaki from comment #34)
> 
> I agree the low-level implementation is very hard to track. I have not
> figured out where the mapping of |PR_Read| to |read| is actually handled
> very clearly even :-(

Chiaki do you have good news?
Flags: needinfo?(ishikawa)

Comment 36

a year ago
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #35)
> (In reply to ISHIKAWA, Chiaki from comment #34)
> > 
> > I agree the low-level implementation is very hard to track. I have not
> > figured out where the mapping of |PR_Read| to |read| is actually handled
> > very clearly even :-(
> 
> Chiaki do you have good news?

No, not yet sorry. At the same time, I am refreshing my patches for removing seek and enabling buffering for pop3 download, etc., I may  be able to look into this further.

But I have a feeling that given the problem was reported on Windows, is there any change that TB was not terminating fully when another invocation of it started (the famous termination issue/bug, etc.). But that is a wild guess.

MacOS issue seems to be my incorrect reading of the spec: I failed to note that PR_Read() returns 0 when EOF condition is hit (as opposed to read under POSIX returns EOF.)

TIA
Flags: needinfo?(ishikawa)
Flags: needinfo?(acelists)
I have an instance which may help.  I set up TB with the latest version 45.2.0.  I created a bare profile, and wanted to have it on a server rather than the local machine.

Local machine is a vmware 6.0 hosted Windows 10 desktop with the above TB installed.

NAS is a Readynas Pro 4 8tb server.  I added a CIFS share for the drive storage.  The share software is Samba, and the Readynas is the Intel version (not slower sparc).

The share which I moved the profile to is mapped as a drive to be remounted at startup by windows as a drive letter.  

The ini file was modified to point at the NAS hosted profile.  I copied the profile (which as then empty) to the nas directory, and updated the profile.ini file

<profile.ini>
[General]
StartWithLastProfile=1

[Profile0]
Name=shared
IsRelative=0
Path=Z:\users\thunderbird-email\2016-0710\2016-0710
Default=1

<Configuration file>
C:\Users\jimst\AppData\Roaming\Thunderbird\pfofile.ini

The server in this case is a remote linux system running qpopper and is remote from my TB instance over the internet with appropriate latency.

I have not had time to investigate, but my previous setup, which was accessing the same qpopper server with the profile hosted on the system with TB installed (also a vmware instance, but of XP64) never had this occur.

I've had one TB crash, but I don't know how to find and attach that crash report to this bug entry as yet.  I didn't notice the files until after the crash.

Also this installation is all new enough, I can state there have been no shutdowns of the system killing TB.  I have had two disconnects from the network (the cable failed from the switch to the hosting vmware esxi server) that may have caused something, since the profile is on a remote network drive (separate from the esxi hardware).

Comment 38

11 months ago
I don't understand English very well but I have linux Mageia 5 and I have the latest Thunderbird ESR. I have the same random folder in the left panel. It compare very rarely and today is the day. I have deleted it. I have none antivirus.

Comment 39

3 hours ago
(In reply to maybe-the-one from comment #3)
> 'Why do you state explicitly for accounts with "messages retained on the
> server"? '
> 
> Because my recollection is that the popstste.dat folders have only appeared
> under accounts that are set that way--but I am not 100% certain of that.  I
> cannot look and see because I delete them when they appear as the drive
> folders I use all the time "off the bottom of the list" and deleting them
> means I do not have to scroll every time to get to them.

Yes, it is probable that if the account is set to not keep messages on the server, then we do not write popstate.dat and the problem of replacing it (as described by WADA) does not happen so no duplication of files happens.
You need to log in before you can comment on or make changes to this bug.