1223198 - Large regression in intermittent failures in Linux tests involving form controls after enabling GTK+ 3

Status: RESOLVED WONTFIX

(Core :: Widget: Gtk, defect, P1)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=asan&tochange=d7844fe50165&fromchange=92a73b885c87&group_state=expanded

Dunno whether we're slow about drawing form controls, and thus our slowest platform is sometimes too slow, or we now sometimes just don't draw them at all, but switching to GTK+ 3 has produced a ton of very frequent failures in ASan tests, which at a glance all seem to have form controls in common.

Comment 1

[Karl Tomlinson (:karlt)]

Most of these are very subtle differences in native rendering.

Perhaps whether alpha-recovery gets used depends on clips or what the native theme draws depends on offsets somehow.  Maybe Lee has some ideas.

bug 1210135, bug 1222402, and 
css-required-radio.html in
https://treeherder.mozilla.org/logviewer.html#?job_id=17028401&repo=mozilla-inbound
are exceptions with larger differences in native rendering.

Bug 1203867, with clipped text seems unrelated to native rendering, unless dimensions are changing.

roc was seeing some odd behaviour with buttons.

Don't know why these would be asan-only.
I wonder whether asan explicitly initializes memory to different values.

Comment 2

[Karl Tomlinson (:karlt)]

I tried disabling various different paths of
nsNativeThemeGTK::DrawWidgetBackground().
Although they didn't obviously clear up these kind of failures, they did
trigger some similar failures more reliably.

Disabling snapping [1][3] caused a new permanent fail in the scrollbar thumb:

> 551463-1.html | image comparison (==), max difference: 15, number of differing pixels: 117

Disabling the "defined(MOZ_X11) && defined(CAIRO_HAS_XLIB_SURFACE)" fast-path
[2][4][5] in DrawThemeWithCairo() caused a permanent fail in textarea
background:

> wordwrap-03.html | image comparison (==), max difference: 1, number of differing pixels: 44

and a new frequent failure with mostly missing tree header row background

> tree-row-outline-1.xul | image comparison (==), max difference: 61, number of differing pixels: 13248

[1] https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#1087
[2] https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#859

[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2cdfafd3275
[4] https://treeherder.mozilla.org/#/jobs?repo=try&revision=095143f3b3f5
[5] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a8d9c74da361

Comment 3

[Milan Sreckovic [:milan] (needinfo for best results)]

Andrew, is there something you can help here? Karl do you want/need help?

Comment 4

[Andrew Comminos [:acomminos]]

Sure, I can give it a shot if Karl doesn't mind me taking this.

I looked at this issue a while back and it seemed nontrivial, but I'm sure some digging in rr will turn up something.

Comment 5

[Karl Tomlinson (:karlt)]

Thanks Andrew and Milan.  Yes, any help would be much appreciated.

The various code paths in DrawWidgetBackground() may lead to different
tessellation algorithms, so it is reasonable to fuzz tests with small
differences.  Some look similar, but differences up to 16/255 sound quite large
for just different tessellation algorithms.  It's a bit disturbing that this
only happens with ASAN builds, but fuzzing the test is still reasonable if the
differences are such that they are not visible enough to be worth investigating.

The failures with larger differences, such as the native widget not being draw
at all, are more concerning.  It may be worth logging whether the X11 error
path is hit.
https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#1178

But requesting access to a VM and installing rr on it is probably the best
strategy.

(In reply to Karl Tomlinson (ni?:karlt) from comment #1)
> roc was seeing some odd behaviour with buttons.

This may also be worth checking out to see whether it is easy to reproduce.
He was seeing buttons being mis-drawn on scrolling.  He probably had FC21 at
the time.

data:text/html,<div%20style="height:120%"><button%20style="height:100px;width:100px">

Comment 6

[Andrew Comminos [:acomminos]]

Attachment: gtk3-widget-factory using Ambiance without unico

It appears most of the issues here are caused by the unico theming engine intermittently not being used for rendering, falling back to gtk's default theme engine.

In the attached screenshot of gtk3-widget-factory using the fallback theme engine on Ubuntu 12.04, note the flat checkboxes, which similar to what we see when layout/reftests/forms/input/checkbox/indeterminate-selector.html fails. Other artifacts similar to what we're seeing on the test infrastructure are the black bordered scrollbar troughs and text fields inset differently.

I suspect that ASAN's memory poisoning is triggering this, particularly the malloc/free implementation's poisoning (the only effect ASAN should have on system libraries). I considered that ASAN's malloc/free implementation violated the expectations of glib's slice allocator (chromium recommends to disable it on ASAN builds), but setting G_SLICE=always-malloc didn't seem to have any impact on the intermittents (see https://treeherder.mozilla.org/#/jobs?repo=try&revision=54c1954fd6a0).

Comment 7

[Robert O'Callahan (:roc) (email my personal email if necessary)]

You could try running some of these tests under rr chaos mode. If they catch a failure, you'll be able to figure it out.

I haven't tested ASAN under rr so there might be some issues there, but if so they should not be hard to fix.

Comment 8

[Phil Ringnalda (:philor)]

I vote we just stop running reftests on ASan - I've lost all interest in their results, and I will star any number of failures, 2 or 10 or 100, as whatever happens to be the first failure which is already filed and thus suggested.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

This is blocking enabling e10s reftests on ASAN. The failure rate is ~50% on R1 and 15-20% on R2.

Comment 10

[Karl Tomlinson (:karlt)]

One option here may be to run these tests with GTK_THEME=Adwaita in the environment.
Maybe even Raleigh, as that was the default GTK theme for 3.4.
I'm guessing some reftests expected results may need tweaking.

If it turns out to be a bug in unico, then that may be our best option.

Comment 11

[Andrew Comminos [:acomminos]]

(In reply to Karl Tomlinson (ni?:karlt) from comment #10)
> One option here may be to run these tests with GTK_THEME=Adwaita in the
> environment.
> Maybe even Raleigh, as that was the default GTK theme for 3.4.
> I'm guessing some reftests expected results may need tweaking.
> 
> If it turns out to be a bug in unico, then that may be our best option.

GTK_THEME was introduced in GTK+ commit 1e70ad1cd126942b38e83c2d01712e8dc7ca675e, which didn't make it in until 3.11.2. I'd imagine the best way to do this would be by setting the "gtk-theme-name" setting at runtime. I don't think adding a custom settings.ini will work, as XSettings will override that.

https://developer.gnome.org/gtk3/stable/GtkSettings.html#GtkSettings--gtk-theme-name

Using the default theming engine sounds good to me though- I think it should resolve this issue. I'll see if I can put together a patch.

Comment 12

[Andrew Comminos [:acomminos]]

Forcing the fallback GTK3 theme (Raleigh) appears to clean up the cases of test failures caused by intermittent changes in the theming engine used, but new intermittents (such as layout/reftests/text-overflow/single-value.html) appear to be introduced by the change- mostly small compositing issues with scrollbar elements, from what I can tell. These may be fuzzed reasonably, unlike the theming engine variances we currently obtain which may remarkably vary.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=08c0609e9810

(the reftests consistently failing across R1 and R2 may be viably fuzzed, IMO- it would be interesting to see the cause now that we can deterministically reproduce it)

I'm not sure if it's worth it to switch to a much less commonly used fallback theming engine in this case just to trade one type of intermittent for another (albeit a much more manageable kind).

Comment 13

[Andrew Comminos [:acomminos]]

In addition, the Raleigh theme appears to not draw resizers, so we would have to disable tests for those were we to switch.

Comment 14

[Phil Ringnalda (:philor)]

Thanks for the investigation.

I've hidden ASan reftest on every tree other than esr38/esr45. Not only are their results useless, but they are starting to devalue the entire suite - I find myself ignoring a reftest failure because I expect this bustage, only to realize it was on Linux64 debug rather than ASan.

Comment 15

[Andrew McCreight [:mccr8]]

I wonder how hard it would be to just disable these image comparison failures on ASan builds. I'd think that we really only care about leaks and things like use-after-frees on Asan builds, not whether an image is off by a pixel or two.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

That's a really interesting idea (though IIRC, we have other tests outside of reftests that are also affected by this root problem - see |skip-if = asan| hits in dxr). What are your thoughts about that, David?

Comment 17

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Are the ASAN tests run on the same OS version / configuration / etc. as other tests?

Comment 18

[Ryan VanderMeulen [:RyanVM]]

Yes, they run on the same machine pool as our other Linux64 tests.

Comment 19

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #16)
> That's a really interesting idea (though IIRC, we have other tests outside
> of reftests that are also affected by this root problem - see |skip-if =
> asan| hits in dxr). What are your thoughts about that, David?

In some ways it's a reasonable idea, but I'm somewhat hesitant to add such a configuration to reftest for fear that it would be mistakenly used where not intended.

It also seems like this should be something that somebody should just be able to reproduce and debug the problem and fix it.

Comment 20

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Also, does somebody have a pointer to an example log showing one of these failures?  All the logs in this bug seem to be expired.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/logviewer.html#?job_id=155009&repo=ash

Comment 22

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #21)
> https://treeherder.mozilla.org/logviewer.html#?job_id=155009&repo=ash

That one looks like it's intermittently using an entirely different GTK theme from one test to the next!

Comment 23

[Karl Tomlinson (:karlt)]

> That one looks like it's intermittently using an entirely different GTK
> theme from one test to the next!

That's consistent with Andrew's observation that it looks like Unico is
intermittently not used.

Looking at the source code, I'm not clear what if anything ensures the engine
is resolved from the style before gtk_render_handle() is called.  This
function uses priv->theming_engine without doing anything to ensure it is
initiallize appropriately.

gtk_style_context_init sets up the default engine:

  priv->theming_engine =
    g_object_ref ((gpointer) gtk_theming_engine_load(/* name = */NULL));

File-static style_data_lookup() is where this can be set to a different
engine.  It is called from gtk_style_context_get_section(),
gtk_style_context_get_property(), _gtk_style_context_peek_property(),
_gtk_style_context_peek_style_property(), gtk_style_context_lookup_icon_set(),
gtk_style_context_notify_state_change(), gtk_style_context_get_font(), but I'm
don't see that any of these are necessarily called.

If this is really an issue, it is fixed in
https://git.gnome.org/browse/gtk+/commit/gtk/gtkstylecontext.c?id=6ac754cdd1b5a273b0c0930866211c9b4bc450f4
which is in
https://git.gnome.org/browse/gtk+/log/gtk/gtkstylecontext.c?h=gtk-3-6&qt=grep&q=engine

We can force resolution of the engine through calling a suitable function.
Tried this at
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1926fdba0fb3&exclusion_profile=false&filter-tier=1
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9e0918755c83&filter-tier=1&filter-tier=3&exclusion_profile=false

but didn't reproduce without the patch either
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5967042fab02&exclusion_profile=false&filter-tier=1

Comment 24

[Karl Tomlinson (:karlt)]

Attachment: flexbox-scrollbar-troughs.png

The much more common issue producing large-magnitude differences now seems to
be the ends of scrollbars.

This comparison of scrollbar troughs, just above the empty resizer space in
layout/reftests/w3c-css/submitted/flexbox/flexbox-overflow-horiz-001.html
recorded in
https://treeherder.mozilla.org/logviewer.html#?job_id=18239274&repo=try
"Reference" on the right is the outlier.

On vertical scrollbars the antialiasing can look very different.  From a
distance it gives the feel of a one pixel difference in position, but up close
it is clearly not a pixel shift and looks like much less smoothing than normal
in the aa.

gtk3drawing uses integer coordinates and sizes, so the only way I could
imagine non-integer offsets is through the cairo transformation matrix.

The scrollbars do not have the dark borders that would be expected if Unico
were not loaded.

Comment 25

[Karl Tomlinson (:karlt)]

Attachment: Scrollbar troughs with (L) and without (R) unico.png

> The scrollbars do not have the dark borders that would be expected if Unico
> were not loaded.

But this is a similar comparison with Ambience modified to use a background
like the -unico-border-gradient that unico uses.
data:text/html,<textarea style="overflow:scroll;height:100px;background:purple"></textarea>

I suspect the background is sometimes being drawn without Unico, but Unico is used for the borders.

Comment 26

[Karl Tomlinson (:karlt)]

(In reply to Karl Tomlinson (ni?:karlt) from comment #25)
> But this is a similar comparison with Ambience modified to use a background
> like the -unico-border-gradient that unico uses.

I mean s/background/border-color/

Comment 27

[Karl Tomlinson (:karlt)]

~/moz/rr/rr% git diff
diff --git a/src/record_syscall.cc b/src/record_syscall.cc
index e0eb6d5..8aae71b 100644
--- a/src/record_syscall.cc
+++ b/src/record_syscall.cc
@@ -1803,7 +1803,7 @@ static void prepare_exit(RecordTask* t, int exit_code) {
 
 static void prepare_mmap_register_params(RecordTask* t) {
   Registers r = t->regs();
-  if (t->session().enable_chaos() &&
+  if (false && t->session().enable_chaos() &&
       !(r.arg4_signed() & (MAP_FIXED | MAP_32BIT)) && r.arg1() == 0) {
     // No address hint was provided. Randomize the allocation address.
     size_t len = r.arg2();

% cat obj-asan/dist/bin/asan-options
#!/bin/sh
export ASAN_OPTIONS=handle_segv=false:detect_leaks=false
exec rr record --chaos /home/karl/moz/dev/obj-asan/dist/bin/firefox ${1+"$@"}

% while MOZCONFIG=../mozconfig-asan DISPLAY=:1 LD_LIBRARY_PATH=/home/karl/downloads/ubuntu/precise/usr/lib/x86_64-linux-gnu:/home/karl/downloads/ubuntu/precise/lib/x86_64-linux-gnu ./mach mochitest --e10s accessible/tests/mochitest/scroll/test_zoom_text.html --keep-open=false --appname=obj-asan/dist/bin/asan-options; do done

The random selection of themes is caused by the bugs fixed in
https://git.gnome.org/browse/gtk+/commit/gtk/gtkbitmaskprivateimpl.h?id=77912a65e2aa6eb9cfee994fa93bbfbc6b9f8538
https://git.gnome.org/browse/gtk+/commit/gtk/gtkbitmaskprivateimpl.h?id=e8f2eeac9221d91ba32447bbcbdc5ba057647f8e
which are in
https://git.gnome.org/browse/gtk+/log/gtk/gtkbitmaskprivateimpl.h?h=3.6.2
The bug exists in
https://git.gnome.org/browse/gtk+/log/gtk/gtkbitmaskprivateimpl.h?h=3.4.0

When allocated addresses of bit masks have bits of the low 32 bits in common,
the intersection returns true.  gtk_css_style_provider_lookup() uses
_gtk_bitmask_intersects() as an optimization, and so returning true incorrectly
causes no problems.  We only see the bugs when allocated addresses have none
of the low 32 bits in common.  This is more likely with ASAN as allocations
are larger.

Unico registers 26 custom properties.
If the unico engine is not loaded, with Raleigh, for example, there should be
only 51 properties.  Bit masks will not be allocated and so the buggy path will
not be taken.

I don't know why we are not seeing the resizer with Raleigh.
gtk3-widget-factory version 3.4.3 draws a resizer for the window, so I assume
we could get a resizer out of Raleigh.

That may be easier than using g_mem_set_vtable to over-allocate to ensure that
all sizeof(GtkBitmask) == 16 allocations have a single bit set in common.

Comment 28

[Karl Tomlinson (:karlt)]

To prevent Unico from loading, the theme needs to be selected before
nsLookAndFeel::Init() runs because that happens early

#0  nsLookAndFeel::Init (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/widget/gtk/nsLookAndFeel.cpp:1098
#1  0x00007f40025caf98 in nsXPLookAndFeel::GetInstance ()
    at /mnt/ssd1/karl/moz/dev/widget/nsXPLookAndFeel.cpp:264
#2  0x00007f40025cb5b3 in mozilla::LookAndFeel::GetInt (aID=<optimized out>, 
    aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/widget/nsXPLookAndFeel.cpp:911
#3  0x00007f4000fdaf86 in mozilla::LookAndFeel::GetInt (aDefault=0, 
    aID=mozilla::LookAndFeel::eIntID_UseAccessibilityTheme)
    at /mnt/sda11/karl/obj-opt/dist/include/mozilla/LookAndFeel.h:561
#4  nsChromeRegistryChrome::CheckForOSAccessibility (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/chrome/nsChromeRegistryChrome.cpp:160
#5  0x00007f4002c7bf60 in ScopedXPCOMStartup::SetWindowCreator (
    this=<optimized out>, native=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1597
#6  0x00007f4002c7f9f4 in XREMain::XRE_mainRun (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4086
#7  0x00007f4002c8063b in XREMain::XRE_main (this=<optimized out>, 
    argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4445
#8  0x00007f4002c80864 in XRE_main (argc=<optimized out>, 
    argv=<optimized out>, aAppData=<optimized out>, aFlags=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4551
#9  0x0000000000405262 in do_main (argc=argc@entry=3, 
    argv=argv@entry=0x7ffd86312128, envp=envp@entry=0x7ffd86312148, 
    xreDirectory=0x7f400ab586c0)
    at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:220
#10 0x00000000004049d5 in main (argc=3, argv=0x7ffd86312128, 
    envp=0x7ffd86312148)
    at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:360

before

#0  nsAppShell::Init (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/widget/gtk/nsAppShell.cpp:204
#1  0x00007f40025f1f8f in nsAppShellInit ()
    at /mnt/ssd1/karl/moz/dev/widget/nsAppShellSingleton.h:45
#2  0x00007f4000fc0851 in nsComponentManagerImpl::KnownModule::Load (
    this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:870
#3  0x00007f4000fc181f in nsFactoryEntry::GetFactory (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1896
#4  0x00007f4000fc1b16 in nsComponentManagerImpl::CreateInstance (
    this=<optimized out>, aClass=..., aDelegate=<optimized out>, aIID=..., 
    aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1118
#5  0x00007f4000fc3024 in nsComponentManagerImpl::GetService (
    this=<optimized out>, aClass=..., aIID=..., aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1363
#6  0x00007f4000fea415 in CallGetService (aCID=..., aIID=..., 
    aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:56
#7  0x00007f4000fea451 in nsGetServiceByCIDWithError::operator() (
    this=<optimized out>, aIID=..., aInstancePtr=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:265
#8  0x00007f4000fe1630 in nsCOMPtr_base::assign_from_gs_cid_with_error (
    this=<optimized out>, aGS=..., aIID=...)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsCOMPtr.cpp:92
#9  0x00007f4002c470a5 in nsCOMPtr<nsIAppShell>::operator= (aRhs=..., 
    this=<optimized out>)
    at /mnt/sda11/karl/obj-opt/dist/include/nsCOMPtr.h:631
#10 nsAppStartup::Init (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/components/startup/nsAppStartup.cpp:157
#11 0x00007f4002c7551a in nsAppStartupConstructor (aOuter=<optimized out>, 
    aIID=..., aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/components/build/nsToolkitCompsModule.cpp:67
#12 0x00007f4000fc1f32 in nsComponentManagerImpl::CreateInstanceByContractID (
    this=<optimized out>, aContractID=<optimized out>, 
    aDelegate=<optimized out>, aIID=..., aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1203
#13 0x00007f4000fc3246 in nsComponentManagerImpl::GetServiceByContractID (
    this=<optimized out>, aContractID=<optimized out>, aIID=..., 
    aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1560
#14 0x00007f4000fea48f in CallGetService (aContractID=<optimized out>, 
    aIID=..., aResult=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:69
#15 0x00007f4000fea4a7 in nsGetServiceByContractID::operator() (
    this=<optimized out>, aIID=..., aInstancePtr=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:280
#16 0x00007f4000fe1693 in nsCOMPtr_base::assign_from_gs_contractid (
    this=<optimized out>, aGS=..., aIID=...)
    at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsCOMPtr.cpp:103
#17 0x00007f4002c7bf87 in nsCOMPtr<nsIWindowCreator>::nsCOMPtr (aGS=..., 
    this=<optimized out>)
    at /mnt/sda11/karl/obj-opt/dist/include/nsCOMPtr.h:540
#18 ScopedXPCOMStartup::SetWindowCreator (this=<optimized out>, 
    native=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1599
#19 0x00007f4002c7f9f4 in XREMain::XRE_mainRun (this=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4086
#20 0x00007f4002c8063b in XREMain::XRE_main (this=<optimized out>, 
    argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4445
#21 0x00007f4002c80864 in XRE_main (argc=<optimized out>, 
    argv=<optimized out>, aAppData=<optimized out>, aFlags=<optimized out>)
    at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4551
#22 0x0000000000405262 in do_main (argc=argc@entry=3, 
    argv=argv@entry=0x7ffd86312128, envp=envp@entry=0x7ffd86312148, 
    xreDirectory=0x7f400ab586c0)
    at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:220
#23 0x00000000004049d5 in main (argc=3, argv=0x7ffd86312128, 
    envp=0x7ffd86312148)
    at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:360

With that, and bug 1261277 and bug 1264170 fixed, the reftests differences all
seem fuzzable, but there are some other tests failing:

docshell/test/browser/browser_timelineMarkers-02.js | markers includes Paint
dom/tests/mochitest/general/test_offsets.xul | scrollHeight - got 49, expected 51
toolkit/content/tests/chrome/test_scaledrag.xul | After mouseup and drag right on slider, scale has correct value - got 1, expected 2
https://treeherder.mozilla.org/#/jobs?repo=try&revision=16cc938b9356&exclusion_profile=false

Comment 29

[Karl Tomlinson (:karlt)]

Moving to ubuntu 16.04 images is another option here.

This can be configured on try using the last two changesets at
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6afe6747bb1

Comment 30

[Chris Peterson [:cpeterson]]

Intermittent e10s test failure

Comment 31

[Karl Tomlinson (:karlt)]

The whole testsuite has been hidden.

Comment 32

[Phil Ringnalda (:philor)]

(In reply to Phil Ringnalda (:philor) from comment #8)
> I vote we just stop running reftests on ASan - I've lost all interest in
> their results, and I will star any number of failures, 2 or 10 or 100, as
> whatever happens to be the first failure which is already filed and thus
> suggested.

This isn't going to be popular, since unlike ASan people actually expect to see results for reftests on opt and debug Linux32 and Linux64, but I'm ready to hide them too, for the exact same reason.

Comment 33

[Phil Ringnalda (:philor)]

P4, eh?

Okay, which do we want, of the two available choices:

* no visible reftests run on any flavor of linux, opt/debug/asan, linux32/linux64, or

* a bug1223198 condition in the sandbox, true for any flavor of linux we currently run, and a random-if(bug1223198) added to every single reftest which displays a form control or a scrollbar, so that we will have visible Linux reftests which are a lie, looking like reftests have run when in fact only a small percentage have run?

Comment 34

[Phil Ringnalda (:philor)]

Clear enough.

Comment 35

[Phil Ringnalda (:philor)]

All Linux reftests hidden on all trees except mozilla-esr45.

Comment 36

[Xidorn Quan [:xidorn] UTC+11]

This is pretty unfortunate. Linux is usually the only platform we run tests on before landing patches, because it is faster than other platforms. Hopefully this can be resolved as soon as possible.

Comment 37

[Joel Maher ( :jmaher ) (UTC -8)]

I ran across this bug, if we have all the reftests hidden for linux32 opt/debug linux64 opt/pgo, we are wasting a lot of effort. I have been looking to get linux32 tests running on taskcluster (building a docker image) and with the tests perma failing, I am considering ignoring these which is sad and only makes the problem worse.

Can we get a decision here as to if we care about these tests and either fix them this year or turn them off on our branches and leave them available on try in case somebody has time in the future?

as a note, asan reftests (on taskcluster/docker/ubuntu 16.04) seem to be pretty green.

Comment 38

[Joel Maher ( :jmaher ) (UTC -8)]

:karlt, I see you are the module owner, can you help address my comments in comment 37 ?

Comment 39

[Ryan VanderMeulen [:RyanVM]]

FWIW, I'm turning off all buildbot-based Linux reftests (except ESR45) over in bug 1317568 :)

Comment 40

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, with that information, I will not plan to run reftests on linux32 taskcluster and let someone else figure out how to get them running.  To be honest there are about 8 failures, so it isn't much, when there is interest to get them running again.

Comment 41

[Ryan VanderMeulen [:RyanVM]]

This bug should either be called fixed or wontfix depending on your perspective. Buildbot jobs are never going to be fixed because Taskcluster has since gotten them green and unhidden (bug 1311916) on Linux64, and that's obviously the only path forward we care about. Assuming we still care about Linux32 enough to support getting jobs ported over from Buildbot to TC on it, my suggestion would be to open a new bug similar to bug 1309086 for Linux32 and track getting them enabled there.

Comment 42

[Karl Tomlinson (:karlt)]

This bug still affects a few mochitests, but bug 1290183 is the solution for
them also, and so, yes, there is no need to keep this bug open.

Buildbot linux64 reftests are obsolete and so can be disabled.
They were still useful while hidden for finding regressions holding back bug
1311916, but are know longer useful.

I don't know the issues preventing making linux64 builds and reftests tier 1 (bug
1243025), or whether it matters if the tc reftests are tier 1 or 2.

I don't know the long term direction with bug 1287099 versus bug 1209932.  Having
buildbot linux32 reftests running on m-c-merged branches may be useful for
finding causes of regressions before bug 1287099 allows them to be unhidden,
but I expect there are few regressions affecting only linux32 reftests.  Bug
1319109 may prove me wrong.