Closed Bug 1302343 Opened 8 years ago Closed 8 years ago

Startup crash in nsFileStreamBase::Write with Symantec/Norton

Categories

(External Software Affecting Firefox :: Other, defect)

x86
All
defect
Not set
blocker

Tracking

(firefox48blocking fixed, firefox49blocking fixed)

RESOLVED FIXED
Tracking Status
firefox48 blocking fixed
firefox49 blocking fixed

People

(Reporter: philipp, Unassigned)

References

Details

(Keywords: crash, Whiteboard: [platform-rel-Symantec][platform-rel-Norton])

Crash Data

This bug was filed from the Socorro interface and is 
report bp-67bf282c-8809-434a-b680-d1d762160913.
=============================================================
Crashing Thread (0)
Frame 	Module 	Signature 	Source
0 		@0xfe02ba 	
1 	xul.dll 	nsFileStreamBase::Write(char const*, unsigned int, unsigned int*) 	netwerk/base/nsFileStreams.cpp:275
2 	xul.dll 	nsAtomicFileOutputStream::Write(char const*, unsigned int, unsigned int*) 	netwerk/base/nsFileStreams.cpp:1021
3 	xul.dll 	WriteLine(nsIOutputStream*, nsACString_internal const&) 	security/manager/ssl/CertBlocklist.cpp:399
4 	xul.dll 	CertBlocklist::SaveEntries() 	security/manager/ssl/CertBlocklist.cpp:455

in the last hours we have seen a dramatic rise in startup crashes of firefox in crash stats data and user reports on sumo about it.

at first sight it looks like it's correlated to some modules related to norton security products. can we try some quick outreach to symantec about this?
Flags: needinfo?(sledru)
Flags: needinfo?(gchang)
user reports about this issue on sumo: https://support.mozilla.org/en-US/questions/firefox?tagged=bug1302343&show=all
Crash Signature: [@ nsStreamCopierIB::DoCopy] [@ nsFileStreamBase::Write] [@ nsBufferedOutputStream::Finish] [@ nsBufferedOutputStream::Write] [@ nsSocketOutputStream::Write] → [@ nsStreamCopierIB::DoCopy] [@ nsFileStreamBase::Write] [@ nsBufferedOutputStream::Finish] [@ nsBufferedOutputStream::Write] [@ nsSocketOutputStream::Write] [@ PR_vfprintf | PR_fprintf | nsPluginHost::WritePluginInfo ]
Crash Signature: [@ nsStreamCopierIB::DoCopy] [@ nsFileStreamBase::Write] [@ nsBufferedOutputStream::Finish] [@ nsBufferedOutputStream::Write] [@ nsSocketOutputStream::Write] [@ PR_vfprintf | PR_fprintf | nsPluginHost::WritePluginInfo ] → [@ nsStreamCopierIB::DoCopy] [@ nsFileStreamBase::Write] [@ nsBufferedOutputStream::Finish] [@ nsBufferedOutputStream::Write] [@ nsSocketOutputStream::Write] [@ nsAtomicFileOutputStream::Write] [@ PR_vfprintf | PR_fprintf | nsPluginHost::WritePlugin…
I sent them an email.
Flags: needinfo?(sledru)
it looks like mainly older firefox versions are affected by this: http://bit.ly/2cnxR1X
See Also: → 597260
Some new bug reports with the same problem:
(Firefox is 45.3.0 ESR and Symantec AV als is also installed)
https://crash-stats.mozilla.com/report/index/b6101bfa-1a56-434b-96cb-1862c2160913
https://crash-stats.mozilla.com/report/index/e08787b1-bc23-40d5-a3a0-85d0c2160913
https://crash-stats.mozilla.com/report/index/b6ac1af5-b78e-4692-bd90-5bf572160913

It also seams, that the reports of this message is increasing extreme today (0 to 25k in last 24h):
https://crash-stats.mozilla.com/signature/?product=Firefox&signature=nsFileStreamBase%3A%3AWrite&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1#graphs

We have at the moment about 4k clients with the start problem.
A reinstallation of firefox on some test systems seams to solve the problem.
Sorry the version wasn't correct.

Rechecked Versions:
Version ESR 45.3.0 ist working after fresh reinstall.
Version ESR 45.1.1 doesn't start since today.
Yes, all versions are affected by the Norton change... :/
(In reply to Gerald Prock from comment #4)
> We have at the moment about 4k clients with the start problem.
> A reinstallation of firefox on some test systems seams to solve the problem.

most likely this was triggered by a botched signature update in the last couple of hours to the symantec security solution running on those systems. you might want to try rolling back the last signature updates and see if that helps or get in contact with norton about this as well: https://support.symantec.com/en_US/article.TECH102935.html
On the first try a rollback did not solve the problem.

It seams, that the first start with the new signatures break the CertBlocklist.
Rollback and restore from all files in the firefox install folder solves the problem.

We decided to start to install the 45.3.0 version on all clients.
judging by reports of other affected users, updating to the latest release or esr version seems to address the crashes on startup, as does rolling back the IPS signatures*.

on 49.0b this problem is apparently showing up as a non-startup crash with the signature [@ mozilla::net::PollableEvent::Signal] as well.

* https://support.mozilla.org/questions/1138694#answer-916178
Crash Signature: nsPluginHost::WritePluginInfo] [@ PR_vfprintf | _MD_WindowsGetSysInfo | PR_fprintf | nsPluginHost::WritePluginInfo] → nsPluginHost::WritePluginInfo] [@ PR_vfprintf | _MD_WindowsGetSysInfo | PR_fprintf | nsPluginHost::WritePluginInfo] [@ mozilla::net::PollableEvent::Signal ]
Flags: needinfo?(gchang)
Summary: Startup crash in nsFileStreamBase::Write with Norton → Startup crash in nsFileStreamBase::Write with Symantec/Norton
Crash Signature: nsPluginHost::WritePluginInfo] [@ PR_vfprintf | _MD_WindowsGetSysInfo | PR_fprintf | nsPluginHost::WritePluginInfo] [@ mozilla::net::PollableEvent::Signal ] → nsPluginHost::WritePluginInfo] [@ PR_vfprintf | _MD_WindowsGetSysInfo | PR_fprintf | nsPluginHost::WritePluginInfo] [@ mozilla::net::PollableEvent::Signal ] [@ mozilla::net::CacheFileIOManager::WriteInternal ]
They say that they are working on it.
Whiteboard: [platform-rel-Symantec][platform-rel-Norton]
Latest versions are affected, we have ~20000 crashes for 48.0.2 and ~50000 for 45.3.0esr.

Given that updating Firefox fixed the crash for the users that philipp contacted, it's possible that the cause of the crash is some file that Norton is removing and that a Firefox update restores.
See Also: → 1222933
(In reply to [:philipp] from comment #7)
> (In reply to Gerald Prock from comment #4)
> > We have at the moment about 4k clients with the start problem.
> > A reinstallation of firefox on some test systems seams to solve the problem.
> 
> most likely this was triggered by a botched signature update in the last
> couple of hours to the symantec security solution running on those systems.
> you might want to try rolling back the last signature updates and see if
> that helps or get in contact with norton about this as well:
> https://support.symantec.com/en_US/article.TECH102935.html

Do we understand how this manifests? For example, is the symantec product removing files from our install, or blocking files from loading from within our process, or..?
(In reply to Jim Mathies [:jimm] from comment #12)
> Do we understand how this manifests? For example, is the symantec product
> removing files from our install, or blocking files from loading from within
> our process, or..?

they might block firefox from writing to revocations.txt in the profile folder. an affected user reported that they ended up with dozens of those files: https://support.mozilla.org/en-US/questions/1138644
we created a support article on the subject: https://support.mozilla.org/en-US/kb/firefox-crashes-startup-symantec-norton-update

some users say that they are already receiving symantec signature updates which fix the crashing again.
(In reply to [:philipp] from comment #13)
> (In reply to Jim Mathies [:jimm] from comment #12)
> > Do we understand how this manifests? For example, is the symantec product
> > removing files from our install, or blocking files from loading from within
> > our process, or..?
> 
> they might block firefox from writing to revocations.txt in the profile
> folder. an affected user reported that they ended up with dozens of those
> files: https://support.mozilla.org/en-US/questions/1138644

Yes, looking at the crash stacks for nsFileStreamBase::Write we're crashing in https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/CertBlocklist.cpp#420, while trying to write the revocations.txt file.

The crashes in nsSocketOutputStream::Write look different, but maybe they are still caused by the lack of the same file?
i guess we can consider this particular bug as solved by symantec now, as they are rolling out signature updates fixing this kind of extraordinary crash spike: https://support.symantec.com/en_US/article.TECH235780.html
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.