Closed Bug 1607671 Opened 1 year ago Closed 1 year ago

Crash in [@ CallPropertyPageCallback]

Categories

(Core :: Printing: Setup, defect, P1)

73 Branch
All
Windows
defect

Tracking

()

VERIFIED FIXED
mozilla74
Tracking Status
firefox-esr68 --- unaffected
firefox72 --- unaffected
firefox73 blocking verified
firefox74 blocking verified

People

(Reporter: philipp, Assigned: jwatt)

References

(Regression)

Details

(Keywords: crash, regression, topcrash, Whiteboard: [print2020_v73])

Crash Data

Attachments

(2 files)

This bug is for crash report bp-92ff7673-5143-4886-aaa3-765180200108.

Top 10 frames of crashing thread:

0 comctl32.dll CallPropertyPageCallback 
1 comctl32.dll DestroyPropertySheetPage 
2 comctl32.dll RemovePropPageData 
3 comctl32.dll _alloca_probe_16 
4 comctl32.dll InitPropSheetDlg 
5 comctl32.dll Prsht_PrepareTemplate 
6 user32.dll InternalCallWinProc 
7 user32.dll VersionRegisterClass 
8 user32.dll DefDlgProcWorker 
9 user32.dll DefDlgProcW 

these printing related crash signatures are starting to show up from users on windows 7 at the start of the 73.0b cycle.

Crash Signature: [@ CallPropertyPageCallback] [@ FreePropertyPageStrings] → [@ CallPropertyPageCallback] [@ FreePropertyPageStrings] [@ DestroyPropertySheetPage] [@ InterlockedDecrement]

there are similar signatures from win10 as well.

Crash Signature: [@ CallPropertyPageCallback] [@ FreePropertyPageStrings] [@ DestroyPropertySheetPage] [@ InterlockedDecrement] → [@ CallPropertyPageCallback] [@ FreePropertyPageStrings] [@ DestroyPropertySheetPage] [@ InterlockedDecrement] [@ _purecall | QISearch] [@ purecall]
OS: Windows 7 → Windows

Hi Jonathan, can you please look at this new crash regression affecting Beta73? Thanks!

Flags: needinfo?(jwatt)

Assigning to jwatt for now to investigate.

Assignee: nobody → jwatt

It's taking me some time to get a Windows build environment working again but I will hopefully have that done soon.

Philip, where did the blame for bug 1602125 come from? It seems like a good candidate but patching the only obvious trigger for this crash in that bug doesn't seem to prevent it. Did you bisect with some specific repo steps?

Flags: needinfo?(jwatt) → needinfo?(madperson)
Priority: -- → P2

sorry, blaming bug 1602125 for the crash was more of a guestimate, as it was showing up a number of times around the crash stack - i had no specific way to reproduce the crash while filing the bug.

now looking at the data more closely, we started seeing those crashes fairly regularly on nightly since 73.0a1 build 20191222094212, which would perhaps make bug 1602410 a more likely candidate as the regressor in this range:

Flags: needinfo?(madperson)

Almost 1000 crashes/545 installs in one signature in b2.

Keywords: topcrash

Given the rate, we need to fix this ASAP.

Severity: normal → critical
Priority: P2 → P1

Jonathan, bug 1602410 backs out cleanly from Beta. Should we consider doing that for 73.0b3 to see if it solves the issue there?

Flags: needinfo?(jwatt)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)

Given the rate, we need to fix this ASAP.

Yes, I'm aware; I worked super late last night FWIW and have continued today. Unfortunately development on Windows absolutely sucks and I'm running into issue after issue getting builds to complete and work as I change between revisions.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)

Jonathan, bug 1602410 backs out cleanly from Beta. Should we consider doing that for 73.0b3 to see if it solves the issue there?

I have a way to reproduce this now (or at least one instance?), but it's a bit awkward. It requires you to sign into a firefox account, load about:preferences#sync, sign into a Firefox Account, click Manage Account, to the right of Account Recovery' clickEnable..., then to the right ofRecovery keyclickGenerate(clickingRevokefirst if you already have a key), then enter your Firefox Accounts password, then at "Step 2 of 2: Save Recovery key" click onPrint` a few times until it crashes.

Ideally I'd like to use that to identify which commit caused the issue before speculatively backing things out in the hope it will fix them, but Mozilla development on Windows is making me a very cross person right now. That said, I'll look at whether that patch is safe to back out speculatively if we're about to make b3.

Flags: needinfo?(jwatt)

(In reply to Jonathan Watt [:jwatt] from comment #10)

That said, I'll look at whether that patch is safe to back out speculatively if we're about to make b3.

I think it is.

Apologies if that above came across as a bit curt or anything like that. I was just rushing to give a reply without spending too much time distracted from getting a fix (and obviously being frustrated with the amount of friction I'm encountering obtaining multiple successful local rebuilds across the potential regression ranges). Thanks for making sure the importance of this bug isn't being missed, Ryan!

Got the builds I need finally. Before the patch for bug 1602410 things work fine with my STRs in comment 10, after the patch I get the broken behavior and crash. Looks like at a minimum that patch needs backed out.

Flags: needinfo?(ryanvm)

Bug 1602410 backed out for 73.0b3. I'll leave the Fx73 status set to affected until we have confirmation from crash-stats that it worked.
https://hg.mozilla.org/releases/mozilla-beta/rev/93585cda453f

Flags: needinfo?(ryanvm)
Regressed by: 1602410
No longer regressed by: 1602125

From the b3 stats, it seems the backout fixed things.

What I find weird is that this crash makes up only 0.27% of the crashes on Nightly, but 28% of the crashes in 73.0b2. I guess that's just down to people using Nightly for testing whereas people actually use Beta for daily browsing?

urls in crash reports show that the crash is mostly affecting users trying to print pages from indian government, banks and other institutions. if i remember correctly from past occasions we have an unproportionally high share of indian users on the beta channel, for reasons that nobody really knows. so this very difference in the user base might explain the dramatic spike once 73 hit the beta channel.

early crash data from 73.0b3 with the backout of bug 1602410 looks very promising btw.

73.0b3 very clearly shows this was fixed by the backout. Setting the status accordingly.

Attached file firefox.exe

This might be totally unrelated but... if you go into Print Preview of http://archive.ph/xTh3E and then toggle between Portrait and Landscape a few times it fails and loads about:blank instead. That's reproducible in Nightly on Linux. Just in case it's the same underlying issue and might enable you to debug it on Linux instead...

Do we know if this is likely to be fixed before 74 is in beta? Thanks

Flags: needinfo?(jwatt)

I guess kmag is busy, but probably anyone can review the attached patch to backout on m-c. RyanVM maybe?

Flags: needinfo?(jwatt) → needinfo?(ryanvm)
Pushed by rvandermeulen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d7e962f7ca16
Back out changeset edf3df2adebb (bug 1602410) for regressions. kmag r=RyanVM
Flags: needinfo?(ryanvm)
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla74
Status: RESOLVED → VERIFIED
Whiteboard: [print_v73]

Please specify a root cause for this bug. See :tmaity for more information.

Root Cause: --- → ?
Whiteboard: [print_v73] → [print20_v73]
Whiteboard: [print20_v73] → [print2020_v73]

Setting root cause to Testing Error due to general lack of test coverage for printing. We now have QA manually testing printing refactoring for Fission to hopefully catch this kind of thing earlier.

Root Cause: ? → Testing Error
You need to log in before you can comment on or make changes to this bug.