990676 - No crash reporter on Windows after Thunderbird crash

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Wayne Mery (:wsmwk)]

Followup from IRC with rhelmer.  This is perhaps multiple bugs. I'm not showing all possibilities below, just some examples. rhelmer has recent fixes on stage, so all examples are using stage

1. beta and alpha versions don't show in overview https://crash-stats.allizom.org/home/products/Thunderbird 

2. windows crash counts missing for beta topcrash list https://crash-stats.allizom.org/topcrasher/products/Thunderbird/versions/28.0b1 (Mac and linux look OK.  ANd query for release version is good https://crash-stats.allizom.org/topcrasher/products/Thunderbird/versions/24.4.0 )

digging deeper with query - 

3. beta and alpha queries bad -- version = all  and  release channel = "beta" lists windows crashes. HOWEVER, the beta crashes seen are all version 26 and older. We should be seeing version 27.0b1 and 28.0b1   https://crash-stats.allizom.org/query/?product=Thunderbird&version=ALL%3AALL&range_value=1&range_unit=weeks&date=04%2F01%2F2014+17%3A00%3A00&query_search=signature&query_type=contains&query=&reason=&release_channels=beta&build_id=&process_type=any&hang_type=any

Comment 1

[Wayne Mery (:wsmwk)]

Not critical yet. But to state the obvious, we now lack insight into crash volume leading up to channel releases and public release.

Comment 2

[Robert Kaiser]

(In reply to Wayne Mery (:wsmwk) from comment #0)
> 1. beta and alpha versions don't show in overview
> https://crash-stats.allizom.org/home/products/Thunderbird 

That's because nobody has set them to "featured". I once did in the past and was shouted down because "Thunderbird doesn't want them to be shown there" as I was told.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Robert Kaiser (:[e-mail]) from comment #2)
> (In reply to Wayne Mery (:wsmwk) from comment #0)
> > 1. beta and alpha versions don't show in overview
> > https://crash-stats.allizom.org/home/products/Thunderbird 
> 
> That's because nobody has set them to "featured". I once did in the past and
> was shouted down because "Thunderbird doesn't want them to be shown there"
> as I was told.

Thanks Robert. That's excellent. Forget I event mentioned it here.  #1 is not my main issue. In fact, it doesn't concern me at all: a) I never use overview b) I didn't remember that TB policy is show only released versions there

Comment 4

[Wayne Mery (:wsmwk)]

only crash reports showing up for non-release queries are non-english locale builds

as example 

- zero hits for this query for english builds 
https://crash-stats.mozilla.com/search/?useragent_locale=en&product=Thunderbird&release_channel=beta&date=%3E2014-03-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform

- OTOH, hits on german builds
 https://crash-stats.mozilla.com/search/?useragent_locale=de&product=Thunderbird&release_channel=beta&date=%3E2014-02-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #4)
> only crash reports showing up for non-release queries are non-english locale
> builds
> 
> as example 
> 
> - zero hits for this query for english builds 
> https://crash-stats.mozilla.com/search/
> ?useragent_locale=en&product=Thunderbird&release_channel=beta&date=%3E2014-
> 03-
> 01&_facets=signature&_columns=date&_columns=signature&_columns=product&_colum
> ns=version&_columns=build_id&_columns=platform

my skills are off, but results are same. The query above should not be "en" but "en-US", as in https://crash-stats.mozilla.com/search/?useragent_locale=en-US&product=Thunderbird&release_channel=beta&date=%3E2014-03-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform

regarding nightly channel crash reports - https://crash-stats.mozilla.com/search/?product=Thunderbird&release_channel=nightly&date=%3E2014-03-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform

no en-US crash reports in the query since 2014-03-08, with one exception bp-9355b15b-adfe-4c56-b557-41b6e2140331 dated 2014-03-31 with TB27.0a1

Comment 6

[Wayne Mery (:wsmwk)]

no WINDOWS crash reports in the query since 2014-03-08, with one exception bp-9355b15b-adfe-4c56-b557-41b6e2140331 dated 2014-03-31 with TB27.0a1

Comment 7

[Wayne Mery (:wsmwk)]

From testing a couple builds, using crashme I get no crash reporter with current daily build, daily 2014-03-20, nor 28.0b1. Perhaps a Thunderbird build issue.

I've not yet determined regression range, nor can I test mac

Comment 8

[Wayne Mery (:wsmwk)]

stacktrace from running tools | nightly tester | crashme 

0:061> g
ModLoad: 733b0000 733be000   C:\Users\Wayne\AppData\Roaming\Thunderbird\Profiles\9a159fc8.default\extensions\{8620c15f-30dc-4dba-a131-7c5d20cf4a29}\platform\WINNT_x86-msvc\crashme.dll
(12ec.10a8): Integer divide-by-zero - code c0000094 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000005 ebx=049c2e20 ecx=00000000 edx=00000000 esi=122cfa00 edi=600d07b5
eip=733b1101 esp=0018e59c ebp=0018e59c iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00210246
*** WARNING: Unable to verify checksum for C:\Users\Wayne\AppData\Roaming\Thunderbird\Profiles\9a159fc8.default\extensions\{8620c15f-30dc-4dba-a131-7c5d20cf4a29}\platform\WINNT_x86-msvc\crashme.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Users\Wayne\AppData\Roaming\Thunderbird\Profiles\9a159fc8.default\extensions\{8620c15f-30dc-4dba-a131-7c5d20cf4a29}\platform\WINNT_x86-msvc\crashme.dll - 
crashme!Crash+0x4d:
733b1101 f7f9            idiv    eax,ecx

Comment 9

[rsx11m]

(In reply to Wayne Mery (:wsmwk) from comment #7)
> I've not yet determined regression range, nor can I test mac

I can't test OSX either, but today's nightly 31.0a1 build for Linux x86_64 opens the crash reporter as desired when a crash is induced. Thus, it may be a Windows-only issue.

Comment 10

[Wayne Mery (:wsmwk)]

rx11m, thanks for testing. 

Using crashme via nightly tester tools on windows, as best as I can determine without putting in multiple man hours, is we have crash reporter for 28.0a1 2013-11-12 but no crash reporter starting 2013-11-13.  It hangs.

I did not test many intervening daily builds, but I know these fail: TB28 2013-11-15 and 2013-11-19, TB 31 2014-04-08 and 2014-08-09, 27.0b1 (dec 18).  I did not test 27.0a2.  N.B. we had no windows builds for 2013-11-07 through -11 - coincidence??

approximate pushlogs
http://hg.mozilla.org/comm-central/pushloghtml?startdate=2013-11-12+03%3A00&enddate=2013-11-13+05%3A00
http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2013-11-12+03%3A00&enddate=2013-11-13+05%3A00

If we have no windows daily crashes since 2013-11-13 nor for any alpha or beta since 27.0b1 (28, 29, 30) then we are pretty blind. This is a blocker.

Comment 11

[Wayne Mery (:wsmwk)]

irving, is this perhaps dependent on bug 982077 (to port bug 932100)? 
I ask this because bug 932100 landed in the regression range.

Comment 12

[Wayne Mery (:wsmwk)]

I haven't checked if SM has suffered this problem.

Comment 13

[Mark Banner (:standard8)]

I'm pretty certain bug 982077 is going to fix this - there's now a patch up there.

Comment 14

[Mark Banner (:standard8)]

Bug 982077 has landed on trunk, I've spun up a new off-cycle windows nightly build so that we can hopefully get feedback in a bit earlier, if it works, then we can push it across to aurora & beta.

Comment 15

[Mark Banner (:standard8)]

Successfully reported a crash using that respun nightly (the previous nightly just hung):

https://crash-stats.mozilla.com/report/index/949a8085-ddce-4375-ad7c-267732140411

Still waiting for this to turn up on crash-stats summary pages. Guess it might take a few hours.

Comment 16

[Robert Helmer [:rhelmer]]

(In reply to Mark Banner (:standard8) from comment #15)
> Successfully reported a crash using that respun nightly (the previous
> nightly just hung):
> 
> https://crash-stats.mozilla.com/report/index/949a8085-ddce-4375-ad7c-
> 267732140411
> 
> Still waiting for this to turn up on crash-stats summary pages. Guess it
> might take a few hours.

Reports (like topcrashers, graphs, etc. - basically everything but search) are run once per day, after Socorro gets ADI data. ADI comes in at the end of each day (UTC).

Comment 17

[Mark Banner (:standard8)]

https://crash-stats.mozilla.com/query/?product=Thunderbird&version=Thunderbird%3A31.0a1&range_value=1&range_unit=weeks&date=04%2F11%2F2014+09%3A00%3A00&query_search=signature&query_type=contains&query=&reason=&release_channels=&build_id=&process_type=any&hang_type=any

now contains that windows crash. Hence  I'm going to call this fixed.

Comment 18

[rsx11m]

(In reply to Wayne Mery (:wsmwk) from comment #12)
> I haven't checked if SM has suffered this problem.

https://crash-stats.mozilla.com/query/?product=SeaMonkey&version=SeaMonkey:2.28a1&platform=win&range_value=1&range_unit=weeks&date=04/09/2014+13:00:00&query_search=signature&query_type=contains&query=&reason=&release_channels=&build_id=&process_type=any&hang_type=any returns matches for SM trunk during last week's period for Windows as well. This was fixed with bug 947599 on trunk and the aurora builds, but still may need to land on beta there.