466385 - no Thunderbird symbols for Mac crashers for nightlies

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Wayne Mery (:wsmwk)]

no Thunderbird symbols for Mac crashers for 3.0b1pre per crash-stats query, since at least 10-31 http://crash-stats.mozilla.com/report/index/233fed91-5adc-47f7-8d40-44be72081122  -- crash-stats query is broken, so it won't let me query earlier than that to determine the regression range

http://crash-stats.mozilla.com/?do_query=1&product=Thunderbird&version=Thunderbird%3A3.0b1pre&platform=mac&query_search=stack&query_type=contains&query=&date=&range_value=30&range_unit=days

Comment 1

[Philippe M. Chiasson (:gozer)]

This is most likely related to bug 464247 and the corresponding fix. This should have resolved itself in the nightlies on/after Nov 19th 2008.

Looking at: http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1227438000.1227443665.3461.gz&fulltext=1, make buildsymbols is now getting passed the right arguments.

Of course, it still sometimes fails (bug 411171) while dumping certain binaries, so there might still not be complete symbols for every single nightiles:

dump_syms(3362) malloc: *** vm_allocate(size=1496895488) failed (error code=3)
dump_syms(3362) malloc: *** error: can't allocate region
dump_syms(3362) malloc: *** set a breakpoint in szone_error to debug
2008-11-23 04:30:33.137 dump_syms[3362] *** Uncaught exception: <NSInvalidArgumentException> *** NSCopyMemoryPages(0x2008000, 0x0, 1197510656) failed

Comment 2

[Wayne Mery (:wsmwk)]

clarifying comment 0, I was careful to check *builds newer* than 10-31, so I'm not talking all builds *since* 10-31

unfortuantely crash stats query by date still "broken, so it won't let me query earlier than that to determine the regression range"

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to comment #2)
> clarifying comment 0, I was careful to check *builds newer* than 10-31, so I'm
> not talking all builds *since* 10-31
> 
> unfortuantely crash stats query by date still "broken, so it won't let me query
> earlier than that to determine the regression range"

gah - c/not talking all builds *since* 10-31/not talking *all crashes* since 10-31/

Comment 4

[Philippe M. Chiasson (:gozer)]

Looking at the buildbot logs, I can see that the nightly on Oct 29 was the last time symbols for thunderbird-bin were successfully extracted. Before that, it was intermittend, now, they fail _every_ time, it would seem.

SeaMonkey has the same problem. Something has changed this from an occasional occurrence...

This might also mean that come beta 1 buld time, generating symbols might just *not* work at all, no matter how many times I run and re-run make buildsymbols.

Comment 5

[David :Bienvenu]

is this still broken?

Comment 6

[Philippe M. Chiasson (:gozer)]

Need to look at the recent buildbot logs, but I suspect it is. Strangely, when it came time to roll Beta 1 Build 1, symbols generated okay the first time around.

I am working on getting a reproductible crash build setup, and plan on digging deep into this problem and finding the root cause, if not a fix.

Comment 7

[(not currently active) Ted Mielczarek]

We should be able to land bug 421534 on the 1.9.1 branch after ~1 week of baking on trunk. I wouldn't bother with deep investigation, personally.

Comment 8

[Philippe M. Chiasson (:gozer)]

Attachment: more verbose reporting of failing dump_syms

When dump_syms fails, it spews some interesting information, but, omits the most critical piece of information. What file it was trying to dump when it failed.

This small patch adds it, and would assist in looking for failure patterns in buildbot logs.

Comment 9

[Philippe M. Chiasson (:gozer)]

Uploaded a new symbols package, with the missing symbols in there.

Before: http://crash-stats.mozilla.com/report/index/233fed91-5adc-47f7-8d40-44be72081122

After:
http://crash-stats.mozilla.com/report/index/c53262c5-90bf-4470-88cb-bdcf32081210

Comment 10

[Philippe M. Chiasson (:gozer)]

Apologies, previous comment was about similar, but unrelated TB 3 Beta 1 symbols.

Comment 11

[Philippe M. Chiasson (:gozer)]

Oh, and for reference, this is basically caused by bug 411171, which is still hapenning most of the time.

Comment 12

[(not currently active) Ted Mielczarek]

Comment on attachment 352358 [details] [diff] [review]
more verbose reporting of failing dump_syms

+                    # dump_syms can soemtimes fail for various reasons
"sometimes" :)

Comment 13

[(not currently active) Ted Mielczarek]

Hopefully should be fixed when you get bug 468622 sorted out.

Comment 14

[Philippe M. Chiasson (:gozer)]

Closing bug 468622 seems like it will resolve this bug indeed.