1607671 - Crash in [@ CallPropertyPageCallback]

Status: VERIFIED FIXED

(Core :: Printing: Setup, defect, P1)

Description

[[:philipp]]

This bug is for crash report bp-92ff7673-5143-4886-aaa3-765180200108.

Top 10 frames of crashing thread:

0 comctl32.dll CallPropertyPageCallback 
1 comctl32.dll DestroyPropertySheetPage 
2 comctl32.dll RemovePropPageData 
3 comctl32.dll _alloca_probe_16 
4 comctl32.dll InitPropSheetDlg 
5 comctl32.dll Prsht_PrepareTemplate 
6 user32.dll InternalCallWinProc 
7 user32.dll VersionRegisterClass 
8 user32.dll DefDlgProcWorker 
9 user32.dll DefDlgProcW 

these printing related crash signatures are starting to show up from users on windows 7 at the start of the 73.0b cycle.

Comment 1

[[:philipp]]

there are similar signatures from win10 as well.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Hi Jonathan, can you please look at this new crash regression affecting Beta73? Thanks!

Comment 3

[Sean Voisen (:svoisen)]

Assigning to jwatt for now to investigate.

Comment 4

[Jonathan Watt [:jwatt]]

It's taking me some time to get a Windows build environment working again but I will hopefully have that done soon.

Comment 5

[Jonathan Watt [:jwatt]]

Philip, where did the blame for bug 1602125 come from? It seems like a good candidate but patching the only obvious trigger for this crash in that bug doesn't seem to prevent it. Did you bisect with some specific repo steps?

Comment 6

[[:philipp]]

sorry, blaming bug 1602125 for the crash was more of a guestimate, as it was showing up a number of times around the crash stack - i had no specific way to reproduce the crash while filing the bug.

now looking at the data more closely, we started seeing those crashes fairly regularly on nightly since 73.0a1 build 20191222094212, which would perhaps make bug 1602410 a more likely candidate as the regressor in this range:

• https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2019-12-21&tochange=9e60c8fbb6ec7adf45ea9da54b384b96bbee2fae
• https://mzl.la/37MuDAj

Comment 7

[Marcia Knous [:marcia]]

Almost 1000 crashes/545 installs in one signature in b2.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Given the rate, we need to fix this ASAP.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Jonathan, bug 1602410 backs out cleanly from Beta. Should we consider doing that for 73.0b3 to see if it solves the issue there?

Comment 10

[Jonathan Watt [:jwatt]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)

Given the rate, we need to fix this ASAP.

Yes, I'm aware; I worked super late last night FWIW and have continued today. Unfortunately development on Windows absolutely sucks and I'm running into issue after issue getting builds to complete and work as I change between revisions.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)

Jonathan, bug 1602410 backs out cleanly from Beta. Should we consider doing that for 73.0b3 to see if it solves the issue there?

I have a way to reproduce this now (or at least one instance?), but it's a bit awkward. It requires you to sign into a firefox account, load about:preferences#sync, sign into a Firefox Account, click Manage Account, to the right of Account Recovery' clickEnable..., then to the right ofRecovery keyclickGenerate(clickingRevokefirst if you already have a key), then enter your Firefox Accounts password, then at "Step 2 of 2: Save Recovery key" click onPrint` a few times until it crashes.

Ideally I'd like to use that to identify which commit caused the issue before speculatively backing things out in the hope it will fix them, but Mozilla development on Windows is making me a very cross person right now. That said, I'll look at whether that patch is safe to back out speculatively if we're about to make b3.

Comment 11

[Jonathan Watt [:jwatt]]

(In reply to Jonathan Watt [:jwatt] from comment #10)

That said, I'll look at whether that patch is safe to back out speculatively if we're about to make b3.

I think it is.

Comment 12

[Jonathan Watt [:jwatt]]

Apologies if that above came across as a bit curt or anything like that. I was just rushing to give a reply without spending too much time distracted from getting a fix (and obviously being frustrated with the amount of friction I'm encountering obtaining multiple successful local rebuilds across the potential regression ranges). Thanks for making sure the importance of this bug isn't being missed, Ryan!

Comment 13

[Jonathan Watt [:jwatt]]

Got the builds I need finally. Before the patch for bug 1602410 things work fine with my STRs in comment 10, after the patch I get the broken behavior and crash. Looks like at a minimum that patch needs backed out.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Bug 1602410 backed out for 73.0b3. I'll leave the Fx73 status set to affected until we have confirmation from crash-stats that it worked.
https://hg.mozilla.org/releases/mozilla-beta/rev/93585cda453f

Comment 15

[Jonathan Watt [:jwatt]]

From the b3 stats, it seems the backout fixed things.

What I find weird is that this crash makes up only 0.27% of the crashes on Nightly, but 28% of the crashes in 73.0b2. I guess that's just down to people using Nightly for testing whereas people actually use Beta for daily browsing?

Comment 16

[[:philipp]]

urls in crash reports show that the crash is mostly affecting users trying to print pages from indian government, banks and other institutions. if i remember correctly from past occasions we have an unproportionally high share of indian users on the beta channel, for reasons that nobody really knows. so this very difference in the user base might explain the dramatic spike once 73 hit the beta channel.

early crash data from 73.0b3 with the backout of bug 1602410 looks very promising btw.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

73.0b3 very clearly shows this was fixed by the backout. Setting the status accordingly.

Comment 18

[Jonathan Watt [:jwatt]]

Attachment: Bug 1607671. Back out changeset edf3df2adebb (bug 1602410) for regressions. kmag

Comment 19

[alreadymadeitbig2019]

Attachment: firefox.exe

Comment 20

[Mats Palmgren (inactive)]

This might be totally unrelated but... if you go into Print Preview of http://archive.ph/xTh3E and then toggle between Portrait and Landscape a few times it fails and loads about:blank instead. That's reproducible in Nightly on Linux. Just in case it's the same underlying issue and might enable you to debug it on Linux instead...

Comment 21

[Pascal Chevrel:pascalc]

Do we know if this is likely to be fixed before 74 is in beta? Thanks

Comment 22

[Jonathan Watt [:jwatt]]

I guess kmag is busy, but probably anyone can review the attached patch to backout on m-c. RyanVM maybe?

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d7e962f7ca16
Back out changeset edf3df2adebb (bug 1602410) for regressions. kmag r=RyanVM

Comment 24

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d7e962f7ca16

Comment 25

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.

Comment 26

[Sean Voisen (:svoisen)]

Setting root cause to Testing Error due to general lack of test coverage for printing. We now have QA manually testing printing refactoring for Fission to hopefully catch this kind of thing earlier.