Created attachment 584386 [details] testcase ###!!! ASSERTION: aWhichFlags should only be displaystyle or compression flag: 'NS_MATHML_IS_DISPLAYSTYLE(aWhichFlags) || NS_MATHML_IS_COMPRESSED(aWhichFlags)', file layout/mathml/nsMathMLFrame.cpp, line 160
This assertion was added in bug 534963.
Here are another way to get the assertion: <math> <msubsup> <mtext>X</mtext> <mtable displaystyle="true"/> </mtext>Y</mtext> </msubsup> </math> (or same with mtable replaced by mstyle). nsMathMLmtableOuterFrame::UpdatePresentationData and nsMathMLmstyleFrame::UpdatePresentationData intentionally disable the flag for displaystyle update and then call the their ancestor UpdatePresentationData, possibly to update the compression flag. So I'm not sure adding the assertion was quite correct or otherwise we would have to verify in each UpdatePresentationData that disable the displaystyle flag that the compression flag is set before calling the ancestor UpdatePresentationData... I suspect fixing bug 838506 would also solve that issue.
Fixed per bug 838506.