Closed Bug 541197 Opened 14 years ago Closed 14 years ago

Repeated attempts to create huge temporary file during message synchronization due to Eset NOD32 causes slow gloda indexing

Categories

(Thunderbird :: General, defect)

x86
Windows XP
defect
Not set
major

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: sub, Unassigned)

Details

(Keywords: perf)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.64 Safari/532.5
Build Identifier: 

I tried to install TB 3 because TB 2.0.0.23 would get stuck repeatedly downloading or indexing new messages. I ran Procmon and it shows TONS of the following problems. Look at the size of the file it is trying to allocate????

124792	3:44:05.6094246 PM	thunderbird.exe	1404	CreateFile	C:\Documents and Settings\Greg\Local Settings\Temp\~ar10F6.tmp	NAME COLLISION	Access: Generic Read, Disposition: Create, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: , AllocationSize: 35,606,850,841,870,336


Reproducible: Always

Steps to Reproduce:
1.Clear out all old profiles, remove TB
2.Reboot.
3.Reinstall TB with secure IMAP parameters
4.During message sync, it CRAWLS...
Actual Results:  
Indexing proceeds at rate of 1 message every 3-4 seconds on a 3ghz dual core machine. Procmon is reporting these errors at the rate of 17,600 PER SECOND.  Yup...  PER SECOND.

Expected Results:  
Indexed at a MUCH faster pace.  It should eat up 20-23% of the CPU during this phase...

I installed TB 3 plain vanilla.  Nothing special. I do have 20,000+ messages on my IMAP server.  I'd suspect this size as an issue, but my laptop runs TB 2.0.0.23 and it works fine.

Something happened 6 months ago, or so, and after that TB never worked quite right on the PC that is causing the problems now.  Could some residual stuff be laying around that causes it to ask for such a GIGANTIC file in the temp folder?
Summary: Indexing SUPER slow, massive number of NAME COLLISION errors → Gloda Indexing SUPER slow, massive number of NAME COLLISION errors
Version: unspecified → 3.0
It sounds like the problem is the code trying to create the temporary file, and that gloda is a victim (as always! ;).

The file size you cite in hex is 0x7e 80 3e 00 00 00 00 which I could completely believe is a pointer overwriting or that overwrote something.

I know that when downloading a message to get along with the virus checkers we write it to a temporary file first to let the virus checkers check it out.  Perhaps we size the temporary file based on our understood size for the message and that got corrupted by a random memory write.  The fact that this seems to happen to you all the time could suggest that the value actually got written to an .msf file and so will haunt you forever until that .msf file gets blown away.

A very good way to gather data on what is happening here might be to do use the xperf tool to get stack traces of when this is happening.  The basics on using xperf with thunderbird can be found here:

https://developer.mozilla.org/En/Profiling_with_Xperf

Instead of invoking:
xperf -on latency -stackwalk profile

You would want to invoke something like this:
xperf -on base -stackwalk FileCreate

To process the BLAH.etl file you save the output to, you would run something like:
xperf -i BLAH.etl -symbols -o stacks.csv

(For the symbol resolution to properly work, you would want to make sure to set up the environment variables to use our symbol server like the wiki page says.)

And then upload stacks.csv as an attachment to this bug or such.  Might want to compress it first, it can get big.


Other alternatives include grepping the .msf files in your profile for the string 7e803e.
Summary: Gloda Indexing SUPER slow, massive number of NAME COLLISION errors → Repeated attempts to create huge temporary file during message synchronization
The temp file stuff is only for pop3 download, not imap, even imap offline download. We don't preflight the temp file size either...
Seeing David's reply, I am hesitant to go the xperf route. Besides, MS site is estimating 4.5 hours to download 2.6gb to install it!  Yikes...

I am running Eset's NOD32. I was running 4.0, but I downloaded the beta 4.2 as it claims support for TB v3.  The problems happen with either version.

Is there something else I can try to decipher why the software keeps trying to create these temp files?  Would it be helpful if I uninstall TB v3, again, trash the profiles, and try to go back to the latest v2 release and see if it does the same thing?  Would that rule things out?
Does NOD32 install something into Thunderbird?  Does it show up in extensions/plugins?
Nothing shows up in Extensions.  I count 16 plugins, however.  Silverlight, Google update, DivX, Java, etc.  Do these come with TB v3 by default? I didn't load any plugins.  BTW, no Eset plug in appears.  However, there is an option in the Message menu for NOD32 with a grayed choice of 'Rescan messages'.

I disabled Real-time file system protection and Real-time Anti-virus and antispyware protection in NOD32 and then started Thundebird and it is indexing MUCH faster and I don't see the messages about the CreateFile error.  I tried this before and it didn't help, but I suspect I didn't disabled both and started TB in the right sequence.

Of course now, I am running without AV protection, which is unacceptable.  I then found the place in NOD32 to disable Thunderbird integration and I turned the real-time protection back on and then I started TB and it is zooming along indexing.  So it appears to be the attempt to integrate NOD32's AV into TB v3 that is causing this serious problem.

I'll go to the Eset support forums and refer them to this bug so they can read up on our findings.

Thanks everyone!
Greg, thx, marking this works for me since the problem seems to be in the AV code.
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
more accurately, this is invalid - caused by external forces :)
Keywords: perf
Resolution: WORKSFORME → INVALID
Summary: Repeated attempts to create huge temporary file during message synchronization → Repeated attempts to create huge temporary file during message synchronization due to Eset NOD32 causes slow gloda indexing
You need to log in before you can comment on or make changes to this bug.