1252041 - EXCEPTION_ILLEGAL_INSTRUCTION crashes [@ mp4parse_new] starting in nightly 2016-02-27 (enabling rust in nightly)

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

The #1 topcrash in nightly starting in 2016-02-27 builds is crashes in mp4parse_new.

https://crash-stats.mozilla.com/signature/?product=Firefox&release_channel=nightly&platform=Windows&date=%3E%3D2016-02-01&signature=mp4parse_new&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&page=1#reports

Regression window is:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=918df3a0bc1c&tochange=5e0140b6d118
presumably regression from bug 1248461.

Comment 1

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

The CPU breakdown of the crashes so far is:

Rank 	Cpu info 	Count 	%
1 	AuthenticAMD family 6 model 8 stepping 1 | 1 	85 	75.89 %
2 	AuthenticAMD family 6 model 10 stepping 0 | 1 	21 	18.75 %
3 	AuthenticAMD family 6 model 8 stepping 0 | 1 	3 	2.68 %
4 	GenuineIntel family 6 model 8 stepping 6 | 1 	2 	1.79 %
5 	CentaurHauls family 6 model 9 stepping 10 | 1 	1 	0.89 %

Comment 2

[(not currently active) Ted Mielczarek]

Disassembling the minidump from https://crash-stats.mozilla.com/report/index/1870d0d1-fe33-4d1e-91cb-fad322160228 indicates that this is crashing calling an SSE instruction:

5f53c2ac:	f2 0f 10 05 28 a1 39 	movsd  0x6139a128,%xmm0
^^^^^^^^^	^^^^^^^^^^^^^^^^^^^^^	^^^^^^^^^^^^^^^^^^^^^^^^

(In reply to David Baron [:dbaron] ⌚️UTC+8 from comment #1)
> The CPU breakdown of the crashes so far is:
> 
> Rank 	Cpu info 	Count 	%
> 1 	AuthenticAMD family 6 model 8 stepping 1 | 1 	85 	75.89 %

This is an "AMD Athlon XP2400" (released in 2002!)

> 4 	GenuineIntel family 6 model 8 stepping 6 | 1 	2 	1.79 %

This appears to be a Pentium 3 or Celeron of some variety.

I guess this was an easy way to answer the "how many of our Nightly users have SSE-capable processors" question! FTR, I don't actually think it's that many, the crash stats indicate ~20 unique users.

Comment 3

[Ralph Giles (:rillian)]

This is probably from llvm assuming sse2 is available. We can pass a flag to turn it off.

Comment 4

[Ralph Giles (:rillian)]

Attachment: Disable sse2 code generation on sse2

Not sure if this works; I don't see where RUSTFLAGS is exported.

Comment 5

[Ralph Giles (:rillian)]

Attachment: Disable sse2 code generation on sse2 v2

Clean up the patch.

Add an export statement for RUSTFLAGS which don't appear to propagate otherwise.

Builds: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5cc5d770cea5

Comment 6

[Ralph Giles (:rillian)]

Comment on attachment 8724987 [details] [diff] [review]
Disable sse2 code generation on sse2 v2

Whoever gets to it first. :)

Comment 7

[Ralph Giles (:rillian)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)

> Disassembling the minidump ... indicates that this is crashing calling an SSE instruction:
> 
> 5f53c2ac:	f2 0f 10 05 28 a1 39 	movsd  0x6139a128,%xmm0

Thanks for disassembling. That was very helpful!

Just for clarity, I believe movsd with an xmm register is sse2, and not part of the original sse extensions set? 
https://en.wikipedia.org/wiki/X86_instruction_listings#SSE2_instructions
> ^^^^^^^^^	^^^^^^^^^^^^^^^^^^^^^	^^^^^^^^^^^^^^^^^^^^^^^^
> 
> (In reply to David Baron [:dbaron] ⌚️UTC+8 from comment #1)
> > The CPU breakdown of the crashes so far is:
> > 
> > Rank 	Cpu info 	Count 	%
> > 1 	AuthenticAMD family 6 model 8 stepping 1 | 1 	85 	75.89 %
> 
> This is an "AMD Athlon XP2400" (released in 2002!)
> 
> > 4 	GenuineIntel family 6 model 8 stepping 6 | 1 	2 	1.79 %
> 
> This appears to be a Pentium 3 or Celeron of some variety.
> 
> I guess this was an easy way to answer the "how many of our Nightly users
> have SSE-capable processors" question! FTR, I don't actually think it's that
> many, the crash stats indicate ~20 unique users.

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8724987 [details] [diff] [review]
Disable sse2 code generation on sse2 v2

As discussed in IRC, we need an AC_SUBST as well.

Comment 9

[Ralph Giles (:rillian)]

Attachment: Disable sse2 code generation on sse2 v3

Add and AC_SUBST line, per feedback. Thanks for the quick attention!

Pushed to try as https://treeherder.mozilla.org/#/jobs?repo=try&revision=972f4e35226e

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8725000 [details] [diff] [review]
Disable sse2 code generation on sse2 v3

LGTM. Please double-check the try build when it finishes to make sure that your flag shows up on the command-line before landing.

Comment 11

[Ralph Giles (:rillian)]

Will do. Thanks again.

Comment 12

[Ralph Giles (:rillian)]

Looks like the flag is propagating now.

16:06:42     INFO -  c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/rustc/bin/rustc --target=i686-pc-windows-msvc -C target-feature=-sse2 -O --crate-type staticlib --emit dep-info=.deps/liblib.lib.pp,link=liblib.lib c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/media/libstagefright/binding/mp4parse/lib.rs

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a5afedbecaad

Comment 14

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> I guess this was an easy way to answer the "how many of our Nightly users
> have SSE-capable processors" question! FTR, I don't actually think it's that
> many, the crash stats indicate ~20 unique users.

How does crash stats end up making a crash that affects ~20 unique users the top #1 crasher?

Comment 15

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

(In reply to Mike Hommey [:glandium] from comment #14)
> How does crash stats end up making a crash that affects ~20 unique users the
> top #1 crasher?

When builds tend towards being more stable, it only requires 10-20 crashes per day to be the top crash in nightly.

Comment 16

[Xidorn Quan [:xidorn] UTC+10]

Comment on attachment 8725000 [details] [diff] [review]
Disable sse2 code generation on sse2 v3

Review of attachment 8725000 [details] [diff] [review]:
-----------------------------------------------------------------

> By default llvm (and rustc) generate sse2 instructions on **x84**, but

What is "x84"? I suppose it's x86 :)

Comment 17

[Ralph Giles (:rillian)]

(In reply to David Baron [:dbaron] ⌚️UTC+8 from comment #15)

> When builds tend towards being more stable, it only requires 10-20 crashes
> per day to be the top crash in nightly.

Seems like good news, really.

> By default llvm (and rustc) generate sse2 instructions on **x84**, but

Oops! Yes, x86 (i686 by the target triple).

Comment 18

[(not currently active) Ted Mielczarek]

(In reply to David Baron [:dbaron] ⌚️UTC+8 from comment #15)
> When builds tend towards being more stable, it only requires 10-20 crashes
> per day to be the top crash in nightly.

FWIW, we have currently have 210 crashes with this signature over 3 days of builds, but yeah, it's likely the same few people hitting this repeatedly.

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/a5afedbecaad

Comment 20

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

This crash has not occurred since the fix landed. It stopped at 2016-03-01 build which was the last build without the fix. With that data, I consider this verified.