428550 - Crashed after typing in an URL and the page opened [@ syncJournal]

Status: RESOLVED WORKSFORME

(Firefox :: Bookmarks & History, defect)

Description

[Miguel Ventura]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5

Although crash reporter didn't appear, windbg attached to the process and I created a dump of this exception. The exception occurs within sqlite module.

Reproducible: Couldn't Reproduce




Stack trace from windbg

0:000> kb n
  *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr  Args to Child              
00 0012f7b8 6021a7ed 014b2358 00fa8448 0012f834 sqlite3!syncJournal+0xc5 [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 23118]
01 0012f7d8 6021d82e 00000000 00000000 003f5f08 sqlite3!sqlite3PagerCommitPhaseOne+0xdd [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 24832]
02 0012f7f4 60225d07 00000000 00000000 03c14488 sqlite3!sqlite3BtreeCommitPhaseOne+0x7e [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 28730]
03 0012f834 60225eff 003f5f08 03c14488 003f5f08 sqlite3!vdbeCommit+0x607 [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 35585]
04 0012f860 6022cb21 03c14488 003f5f08 003f5f08 sqlite3!sqlite3VdbeHalt+0x13f [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 35902]
05 0012fb28 6022750a 03c14488 0012fb6c 00290040 sqlite3!sqlite3VdbeExec+0x4201 [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 40272]
06 0012fb48 60227628 00000000 00000001 03c14488 sqlite3!sqlite3Step+0x13a [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 37032]
07 0012fb6c 60241d46 03c14488 0012fbec 0012fc30 sqlite3!sqlite3_step+0x48 [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 37099]
08 0012fba8 605c0b8b 003f5f08 60cda414 00000000 sqlite3!sqlite3_exec+0x116 [e:\fx19rel\winnt_5.2_depend\mozilla\db\sqlite3\src\sqlite3.c @ 55877]
09 0012fbd8 605e433c 01467e80 0012fbec 00fb2ce8 xul!mozStorageConnection::ExecuteSimpleSQL+0x5b [e:\fx19rel\winnt_5.2_depend\mozilla\storage\src\mozstorageconnection.cpp @ 300]
0a 0012fbf8 605ace41 01467e80 6061b020 60637241 xul!mozStorageConnection::CommitTransaction+0x33 [e:\fx19rel\winnt_5.2_depend\mozilla\storage\src\mozstorageconnection.cpp @ 427]
0b 0012fc00 6061b020 60637241 03d9f490 00000002 xul!mozStorageTransaction::~mozStorageTransaction+0x24 [e:\fx19rel\winnt_5.2_depend\mozilla\obj-fx-trunk\dist\include\storage\mozstoragehelper.h @ 84]
0c 0012fc40 605668fc 03d9f490 00fb2c00 0031b3c0 xul!nsNavHistory::CommitLazyMessages+0xc8 [e:\fx19rel\winnt_5.2_depend\mozilla\toolkit\components\places\src\nsnavhistory.cpp @ 4651]
0d 0012fc58 60566873 00000000 00000001 604ddd5d xul!nsTimerImpl::Fire+0x7c [e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\threads\nstimerimpl.cpp @ 400]
0e 0012fc64 604ddd5d 0532fd30 00f4e600 00319470 xul!nsTimerEvent::Run+0x1f [e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\threads\nstimerimpl.cpp @ 492]
0f 0012fc88 604af56a 00000001 00000001 0012fca8 xul!nsThread::ProcessNextEvent+0x29d [e:\fx19rel\winnt_5.2_depend\mozilla\xpcom\threads\nsthread.cpp @ 510]
10 0012fca0 6063e990 00000001 80000000 6057a06a xul!nsBaseAppShell::Run+0x4a [e:\fx19rel\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsbaseappshell.cpp @ 169]
11 0012fcac 6057a06a 00f4d970 0031c0b0 00000000 xul!nsAppStartup::Run+0x1e [e:\fx19rel\winnt_5.2_depend\mozilla\toolkit\components\startup\src\nsappstartup.cpp @ 182]
12 0012fcb4 0031c0b0 00000000 0031c0a8 00300518 xul!XRE_main+0xdac [e:\fx19rel\winnt_5.2_depend\mozilla\toolkit\xre\nsapprunner.cpp @ 3158]
WARNING: Frame IP not in any known module. Following frames may be wrong.

Comment 1

[Miguel Ventura]

Attachment: Output from WinDbg session

Commands:
!analyze -v
.excr
kb n
lm
!heap -v
dv -v (for both the top frame and the top frame outside sqlite)

Comment 2

[timeless]

 23117                   for(pPg=pPager->pAll; pPg; pPg=pPg->pNextAll){
 23118                     pPg->needSync = 0;

this is actually a topcrasher. which i've wanted to catch because it doesn't make sense.

the log shows:
0:000> dv -v
@esi          pPager = 0x0142e808

please try something like:

?? pPager->pAll
?? pPager->pAll->pNextAll
?? pPager->pAll->pNextAll->pNextAll
...

(you can also just use the watches or locals view and follow those paths)

Comment 3

[Miguel Ventura]

Attachment: Contents for pPager->pAll linked list

The list goes on like forever, which is, until I hit some windbg limit that doesn't let me see any further... So I go through the memory address of the last one I could see (0x07f5a008) but I get bored real soon. What are we supposed to be looking for in this list anyway?

Comment 4

[timeless]

well, it somehow crashed in this code.
ideally you'd eventually reach the @eax=676d6361 value.
it occurs to me that @eax is odd. which is well odd :)

((PgHdr*)0x07f5a008)->...
struct PgHdr * 0x07f4a008

you should be able to do:
((PgHdr*)0x07f4a008)->pNextAll

i think that windbg actually has a macro or primitive for linked list walking, sadly i haven't learned it.

Comment 5

[Miguel Ventura]

Attachment: Contents for pAll->pNextAll until EAX

There's a macro after all... !list,
which saved the day because I only stopped on the previous output because i was really tired of endless iterations looking for something that I didn't know what it was :P

this attached output contains the start and end for !list and the last 2 elements of the linked list

Comment 6

[timeless]

out of curiosity, do all the other pointers end in 8 (the last one ends in 0, the bogus one in 1, and all the others both in your previous list and this one end in 8.

this is odd:
   +0x000 pPager           : (null) 
in the previous view, everyone shared a common value for pPager.

is pPager ever supposed to be null along the list?

i don't think this is important as the previous view shows lots of elements with this value:
   +0x022 nRef             : 0

Comment 7

[Miguel Ventura]

Yes, 0x079bb008 seems to be the last coherent structure. All others before have a reference to pPager @ 0142e808 followed by a small (< 0x00000fff) pgno. pData also seems to never be null. The list obviously broke there (the moment the pNextAll was set to 0x07988008). If you think it may be useful I can provide you with a full dump of the list but I think it'll be a very tedious reading.

Comment 8

[timeless]

I think it's probably time for drh to take a look.

someone else of course could try to figure out how that item became null. but i don't have the energy to chase it. if i were chasing it, i'd probably turn to the sources and look for ways for it to happen.

Comment 9

[D. Richard Hipp]

(In reply to comment #8)
> I think it's probably time for drh to take a look.
> 

I've been looking.  I don't have any ideas.  If you can find a way to reproduce the problem, that would be helpful.  

Comment 10

[Miguel Ventura]

Sorry, but I can't seem to reproduce it...

Comment 12

[Miguel Ventura]

Attachment: Output from new hang (firefox-15Mai000.dmp) windbg session

I hit this one once again! Now with the RC1 build (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0) instead of the 3Beta5 one. Again it was just shortly after entering the address and the page loaded. Again I don't know a deterministic way of reproducing it. Here are all the previous commands I did on windbg but with this new dump.

Comment 13

[D. Richard Hipp]

We (the SQLite developers) have looked and look and we still do not see any way that this could be happening.  If anybody manages to find a way to reproduce the problem, or if you can catch the crash in a debugger and let one of us attach to your desktop remotely, please let us know.  The SQLite hotline is (US area code 704) 948.4565.

Comment 14

[Miguel Ventura]

@drh:
I don't know how to hang the application (although it happened twice already) but I still have the crash dumps which I believe will tell you as much as attaching a debugger at the time of the crash. I would rather not send you over the dumps as I'm not aware of confidential information they may contain. However I can try to provide you with a remote desktop or vnc session so you can examine the dump. If that's enough for you, I'd also rather arrange that session by e-mail (free) than phone calling the US (a bit expensive).

Comment 15

[Shawn Wilsher :sdwilsh]

Try server builds with a slightly modified sqlite can be found here:
https://build.mozilla.org/tryserver-builds/2008-05-19_15:[e-mail]-bug428550test/

Getting the memory dump like attachment 315541 [details] would be helpful with one of these builds.

Comment 16

[Miguel Ventura]

@Shawn: What differences do those builds have from the official branch? Are there available sources? I also can't access the debugging symbols at tryserver (I don't have a user for it) which renders crash dumps quit cryptic...

I've been talking with drh and he's already looked into the two dumps. I believe he has interest in using code as close as possible with sqlite's vanilla flavor. As of now I'm waiting for either news from him or for ffx to crash again...

Comment 17

[Shawn Wilsher :sdwilsh]

Attachment: code change to sqlite

This is just a patch to sqlite that was given to me by drh that made up those try-server builds.

Comment 18

[D. Richard Hipp]

The new sqlite3.c writes some sentinel values into various places of the PgHdr structure.  It changes those values when the structure is allocated and deallocated, etc., so that after a crash if the sentinels have not been overwritten we can perhaps identify memory allocation errors.  This is a long shot, but we do not have any better clues at the moment.

Comment 19

[Miguel Ventura]

Ok, I'll install it then.

Still, is there any way I can access the symbol files? I found a symbol server URL (http://build.mozilla.org/tryserver-symbols) at http://wiki.mozilla.org/Build:TryServer but it asks me for http authentication and it's stated that the builds are deleted after 30 days. If the debug symbols are also deleted then I'd rather download them and have them locally.

As I can't reproduce the crash when I want, this may take a while...

Comment 20

[Shawn Wilsher :sdwilsh]

Apparently it needs LDAP authentication, which sorta sucks...

Do you have a local build environment setup where you could produce a build yourself?

Comment 21

[Miguel Ventura]

I installed the build and managed to download the debug symbols (I was afraid 30 days would pass and the try server deleted them). However I haven't hit this crash ever since. Meanwhile FF has self-updated into RC2 so I guess I kinda lost the patch again.

Comment 22

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv