Intermittent Assertion failure: [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000), at z:\build\build\src\obj-firefox\dist\include\mozilla/gfx/Logging.h:520

RESOLVED FIXED in Firefox 57

Status

()

Core
Layout
P3
normal
RESOLVED FIXED
4 months ago
10 days ago

People

(Reporter: aryx, Assigned: jmaher)

Tracking

(Blocks: 1 bug, {assertion, intermittent-failure})

Trunk
mozilla57
assertion, intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr52 wontfix, firefox55 wontfix, firefox56 wontfix, firefox57 fixed)

Details

(Whiteboard: [stockwell disabled])

Attachments

(2 attachments)

https://treeherder.mozilla.org/logviewer.html#?job_id=107575203&repo=autoland

23:40:06     INFO - REFTEST TEST-START | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018.html == file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018-ref.html
23:40:06     INFO - REFTEST INFO | RESTORE PREFERENCE pref(layout.css.shape-outside.enabled,false)
23:40:06     INFO - REFTEST INFO | SET PREFERENCE pref(layout.css.shape-outside.enabled,true)
23:40:06     INFO - REFTEST TEST-LOAD | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018.html | 1631 / 7783 (20%)
23:40:06     INFO - ++DOMWINDOW == 287 (12A05C00) [pid = 3940] [serial = 7065] [outer = 1B2A5400]
23:40:06     INFO - [GFX1-]: Failed 2 buffer db=00000000 dw=00000000 for 0, 0, 800, 1000
23:40:06     INFO - [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000)
23:40:06     INFO - Assertion failure: [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000), at c:\builds\moz2_slave\autoland-w32-d-000000000000000\build\src\obj-firefox\dist\include\mozilla/gfx/Logging.h:519
23:40:24     INFO - #01: mozilla::gfx::Log<1,mozilla::gfx::CriticalLogger>::Flush() [obj-firefox/dist/include/mozilla/gfx/Logging.h:283]
23:40:24     INFO - 
23:40:24     INFO - #02: mozilla::gfx::Factory::CreateDrawTarget(mozilla::gfx::BackendType,mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gfx::SurfaceFormat) [gfx/2d/Factory.cpp:397]
23:40:24     INFO - 
23:40:24     INFO - #03: gfxPlatform::CreateDrawTargetForBackend(mozilla::gfx::BackendType,mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gfx::SurfaceFormat) [gfx/thebes/gfxPlatform.cpp:1454]
23:40:24     INFO - 
23:40:24     INFO - #04: mozilla::layers::PersistentBufferProviderBasic::Create(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>,mozilla::gfx::SurfaceFormat,mozilla::gfx::BackendType) [gfx/layers/PersistentBufferProvider.cpp:73]
23:40:24     INFO - 
23:40:24     INFO - #05: mozilla::layers::LayerManager::CreatePersistentBufferProvider(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gfx::SurfaceFormat) [gfx/layers/Layers.cpp:149]
23:40:24     INFO - 
23:40:24     INFO - #06: mozilla::layers::ClientLayerManager::CreatePersistentBufferProvider(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gfx::SurfaceFormat) [gfx/layers/client/ClientLayerManager.cpp:920]
23:40:24     INFO - 
23:40:24     INFO - #07: mozilla::dom::CanvasRenderingContext2D::TrySharedTarget(RefPtr<mozilla::gfx::DrawTarget> &,RefPtr<mozilla::layers::PersistentBufferProvider> &) [dom/canvas/CanvasRenderingContext2D.cpp:1892]
23:40:24     INFO - 
23:40:24     INFO - #08: mozilla::dom::CanvasRenderingContext2D::EnsureTarget(mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits,float> const *,mozilla::dom::CanvasRenderingContext2D::RenderingMode) [dom/canvas/CanvasRenderingContext2D.cpp:1697]
23:40:24     INFO - 
23:40:24     INFO - #09: mozilla::dom::CanvasRenderingContext2D::DrawWindow(nsGlobalWindow &,double,double,double,double,nsAString const &,unsigned int,mozilla::ErrorResult &) [dom/canvas/CanvasRenderingContext2D.cpp:5601]
23:40:24     INFO - 
23:40:24     INFO - #10: mozilla::dom::CanvasRenderingContext2DBinding::drawWindow [obj-firefox/dom/bindings/CanvasRenderingContext2DBinding.cpp:2311]
23:40:24     INFO - 
23:40:24     INFO - #11: mozilla::dom::GenericBindingMethod(JSContext *,unsigned int,JS::Value *) [dom/bindings/BindingUtils.cpp:2960]
23:40:24     INFO - 

....

23:40:24    ERROR - TEST-UNEXPECTED-FAIL | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018.html | application terminated with exit code 1
23:40:24     INFO - REFTEST INFO | Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld\appdata\local\temp\tmpubwyc8.mozrunner\minidumps\48e261cc-30b6-4ed0-aa5d-ceb0bed311a0.dmp C:\slave\test\build\symbols
23:40:26     INFO - REFTEST INFO | Saved minidump as C:\slave\test\build\blobber_upload_dir\48e261cc-30b6-4ed0-aa5d-ceb0bed311a0.dmp
23:40:26     INFO - REFTEST INFO | Saved app info as C:\slave\test\build\blobber_upload_dir\48e261cc-30b6-4ed0-aa5d-ceb0bed311a0.extra
23:40:26     INFO - REFTEST PROCESS-CRASH | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018.html | application crashed [@ xul.dll + 0x8f2fe3]
23:40:26     INFO - Crash dump filename: c:\users\cltbld\appdata\local\temp\tmpubwyc8.mozrunner\minidumps\48e261cc-30b6-4ed0-aa5d-ceb0bed311a0.dmp
23:40:26     INFO - Operating system: Windows NT
23:40:26     INFO -                   6.1.7601 Service Pack 1
23:40:26     INFO - CPU: x86
23:40:26     INFO -      GenuineIntel family 6 model 45 stepping 7
23:40:26     INFO -      8 CPUs
23:40:26     INFO - 
23:40:26     INFO - GPU: UNKNOWN
23:40:26     INFO - 
23:40:26     INFO - Crash reason:  EXCEPTION_BREAKPOINT
23:40:26     INFO - Crash address: 0x5d922fe3
23:40:26     INFO - Process uptime: 767 seconds
23:40:26     INFO - 
23:40:26     INFO - Thread 0 (crashed)
23:40:26     INFO -  0  xul.dll + 0x8f2fe3
23:40:26     INFO -     eip = 0x5d922fe3   esp = 0x001da418   ebp = 0x001da420   ebx = 0x001da598
23:40:26     INFO -     esi = 0x6098f950   edi = 0x00000208   eax = 0x00000000   ecx = 0x70e906ef
23:40:26     INFO -     edx = 0x00000060   efl = 0x00000206
23:40:26     INFO -     Found by: given as instruction pointer in context

Comment 1

4 months ago
10 failures in 814 pushes (0.012 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 5
* windows7-32-vm: 2
* linux64: 2
* linux64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 2

3 months ago
16 failures in 720 pushes (0.022 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6

Platform breakdown:
* windows7-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-10&endday=2017-07-16&tree=all

Updated

3 months ago
See Also: → bug 1373532
(Assignee)

Comment 3

3 months ago
This really started up on July 13th and was failing quite often by July 15th.  I think this is on track for a high frequency failure (24 failures since July 13th).  This is the same failure as posted above.

:jet, could you help find someone to look at fixing this crash in the next 2 weeks?
Flags: needinfo?(bugs)
Whiteboard: [stockwell needswork]
Summary: Intermittent layout/reftests/w3c-css/submitted/shapes1/shape-outside-polygon-018.html | application crashed [@ xul.dll + 0x8f2fe3] → Intermittent Assertion failure: [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000), at z:\build\build\src\obj-firefox\dist\include\mozilla/gfx/Logging.h:519

Comment 4

3 months ago
25 failures in 822 pushes (0.03 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 13
* autoland: 12

Platform breakdown:
* windows7-32: 24
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 5

3 months ago
16 failures in 181 pushes (0.088 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 5
* mozilla-central: 1

Platform breakdown:
* windows7-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-25&endday=2017-07-25&tree=all

Comment 6

3 months ago
26 failures in 218 pushes (0.119 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 15
* mozilla-inbound: 10
* try: 1

Platform breakdown:
* windows7-32: 26

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-26&endday=2017-07-26&tree=all

Comment 7

3 months ago
17 failures in 186 pushes (0.091 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6
* cedar: 1

Platform breakdown:
* windows7-32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-27&endday=2017-07-27&tree=all

Comment 8

3 months ago
92 failures in 1008 pushes (0.091 failures/push) were associated with this bug in the last 7 days. 

This is the #21 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 54
* mozilla-inbound: 30
* try: 5
* mozilla-central: 2
* oak: 1

Platform breakdown:
* windows7-32: 92

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-24&endday=2017-07-30&tree=all

Updated

3 months ago
See Also: → bug 1378954

Updated

3 months ago
See Also: → bug 1378534

Updated

3 months ago
See Also: → bug 1378621

Comment 9

3 months ago
This is happening often enough that we should disable tests, but there are a lot of different tests failing this way currently. (Many, but not all in css-break.)
(In reply to Joel Maher ( :jmaher) (UTC-9) (PTO: back August 2nd) from comment #3)
> This really started up on July 13th and was failing quite often by July
> 15th.  I think this is on track for a high frequency failure (24 failures
> since July 13th).  This is the same failure as posted above.

This failure was seen much earlier, but reported in other bugs. For instance, July 3, https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=56cb20e086a2d67a467809d406d5fed673267aca&filter-searchStr=windows+debug+reftest.


AFAIK, all of these failures are Windows 7, non-e10s, so perhaps this is another case where we can wait for non-e10s testing to end.

Comment 11

3 months ago
(In reply to Geoff Brown [:gbrown] from comment #10)
> AFAIK, all of these failures are Windows 7, non-e10s, so perhaps this is
> another case where we can wait for non-e10s testing to end.

I'm not sure that all the "Failed to create DrawTarget" errors really are constrained to non-e10s Win32. +cc: Milan who may know more about that.
Flags: needinfo?(bugs) → needinfo?(milan)
This is a failure with backend type Skia - which means non-accelerated Windows.  I think the rest of the Windows testing has acceleration, right?  So, it's a coincidence that we're only seeing it on those configurations, although I guess e10s could play part of this if it turns out it is "just" running out of memory.

Mason, was there any OMTP that landed mid July that could explain some of these problems increasing in frequency?  I guess we don't usually have a recorded, so there shouldn't be, but just checking.
Flags: needinfo?(milan) → needinfo?(mchang)
(In reply to Milan Sreckovic [:milan] from comment #12)
> This is a failure with backend type Skia - which means non-accelerated
> Windows.  I think the rest of the Windows testing has acceleration, right? 
> So, it's a coincidence that we're only seeing it on those configurations,
> although I guess e10s could play part of this if it turns out it is "just"
> running out of memory.
> 
> Mason, was there any OMTP that landed mid July that could explain some of
> these problems increasing in frequency?  I guess we don't usually have a
> recorded, so there shouldn't be, but just checking.

I scanned a couple of the crash signatures and all of them are coming from Canvas. All the OMTP stuff only changed in ClientPaintedLayer, which isn't in the callstack at all here. From comment 3, this started on July 13th. I did a bugzilla search to see what OMTP patches landed between July 13-15 and came up with bug 1380493, and bug 1380483. Bug 1380483 might be the most suspicious in that we start checking that content client exists before recording, but this doesn't actually create a DrawTarget and would've happened by default since we don't enable OMTP on inbound yet.
Flags: needinfo?(mchang)

Comment 14

3 months ago
20 failures in 131 pushes (0.153 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 10
* autoland: 10

Platform breakdown:
* windows7-32: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-01&endday=2017-08-01&tree=all

Comment 15

3 months ago
23 failures in 160 pushes (0.144 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 17
* mozilla-inbound: 6

Platform breakdown:
* windows7-32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-02&endday=2017-08-02&tree=all

Comment 16

2 months ago
61 failures in 888 pushes (0.069 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 34
* mozilla-inbound: 22
* try: 5

Platform breakdown:
* windows7-32: 61

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 17

2 months ago
1 failures in 901 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-07&endday=2017-08-13&tree=all
Week over week, 61/888 -> 1/901, beta only.
This was happening on non-e10s. We are only running e10s windows reftests now. This might return if we start running non-e10s windows reftests again.
Whiteboard: [stockwell needswork] → [stockwell disabled]

Comment 20

2 months ago
19 failures in 949 pushes (0.02 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 6
* autoland: 6
* mozilla-central: 5
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 21

2 months ago
28 failures in 129 pushes (0.217 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 15
* mozilla-inbound: 8
* mozilla-central: 3
* try: 2

Platform breakdown:
* windows7-32: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-21&endday=2017-08-21&tree=all

Comment 22

2 months ago
38 failures in 143 pushes (0.266 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 20
* mozilla-inbound: 17
* try: 1

Platform breakdown:
* windows7-32: 38

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-22&endday=2017-08-22&tree=all

Comment 23

2 months ago
35 failures in 173 pushes (0.202 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 16
* mozilla-inbound: 9
* try: 6
* mozilla-central: 4

Platform breakdown:
* windows7-32: 35

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-23&endday=2017-08-23&tree=all

Updated

2 months ago
Summary: Intermittent Assertion failure: [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000), at z:\build\build\src\obj-firefox\dist\include\mozilla/gfx/Logging.h:519 → Intermittent Assertion failure: [GFX1]: Failed to create DrawTarget, Type: 3 Size: Size(800,1000), at z:\build\build\src\obj-firefox\dist\include\mozilla/gfx/Logging.h:520

Comment 24

2 months ago
18 failures in 194 pushes (0.093 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-24&endday=2017-08-24&tree=all

Comment 25

2 months ago
33 failures in 196 pushes (0.168 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 18
* mozilla-inbound: 13
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 33

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-25&endday=2017-08-25&tree=all
(Assignee)

Comment 26

2 months ago
we are looking at 160+ failures in the last week, all on windows 7 debug non-e10s:
https://brasstacks.mozilla.com/orangefactor/index.html?display=Bug&bugid=1373578

there is no test to turn off as this seems to happen inside of reftests (as pointed out earlier css-break)

Is this a graphics specific crash? I see comments regarding non accelerated graphics from :milan and :mchang.


:jet, I see you are the triage owner- can you help find the right person to fix this in the next week?
Flags: needinfo?(bugs)
Whiteboard: [stockwell disabled] → [stockwell needswork]

Comment 27

2 months ago
177 failures in 908 pushes (0.195 failures/push) were associated with this bug in the last 7 days. 

This is the #3 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 83
* mozilla-inbound: 58
* mozilla-central: 17
* try: 13
* pine: 4
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 176
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 28

2 months ago
18 failures in 145 pushes (0.124 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 12
* mozilla-inbound: 4
* try: 2

Platform breakdown:
* windows7-32: 17
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-28&endday=2017-08-28&tree=all

Comment 29

2 months ago
28 failures in 175 pushes (0.16 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 14
* mozilla-central: 7
* mozilla-inbound: 4
* try: 3

Platform breakdown:
* windows7-32: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-29&endday=2017-08-29&tree=all

Comment 30

2 months ago
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #26)
> :jet, I see you are the triage owner- can you help find the right person to
> fix this in the next week?

As noted above, we're running into resource contention on the Win32 Debug variants. This will only get worse on those platforms as more reftests get added. I'm willing to take a cheap fix here (e.g., restart the browser after n tests, or shrink the test bucket sizes) but it seems unlikely that we'll see a fix in Skia for a use case (thousands of sequential large canvas bitmap snapshots in debug builds) that only our test harness has.
Flags: needinfo?(bugs)
(Assignee)

Comment 31

2 months ago
bug 1302203 is to run reftests per manifest that should fix this- we could disable this on non-e10s win7-debug
Depends on: 1302203

Comment 32

2 months ago
15 failures in 113 pushes (0.133 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 7
* autoland: 7
* mozilla-central: 1

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-30&endday=2017-08-30&tree=all
(Assignee)

Comment 33

2 months ago
Created attachment 8903174 [details] [diff] [review]
disable_win7_reftest.patch

I don't think there is a test to disable, this leaves us with either disabling all the tests on win7/debug (non-e10s), or possibly running in more chunks.

this patch would disable the reftests on win7-debug for non-e10s- as a note, this is the configuration that we specifically turned back on for non-e10s; our coverage of non-e10s reftests will be android only.  

I am happy to do more chunks instead or other ideas.
Attachment #8903174 - Flags: review?(bugs)
I tried running with 16 chunks, but still hit this failure: I don't think more chunks is feasible.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=326923e740a99e12e9280d3798ed6cdd77d85b35

Comment 35

2 months ago
37 failures in 233 pushes (0.159 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 17
* autoland: 16
* try: 4

Platform breakdown:
* windows7-32: 37

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-31&endday=2017-08-31&tree=all
(Assignee)

Comment 36

2 months ago
it seems that just disabling reftests wholesale is our only option right now.  We have been working on making the harness restart between manifests and that is a lot harder than it would seem.  Possibly we could run a really small subset of the tests if we find there is important value on non-e10s for a specific feature or two?

Comment 37

2 months ago
23 failures in 180 pushes (0.128 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 12
* mozilla-inbound: 6
* try: 3
* mozilla-central: 2

Platform breakdown:
* windows7-32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-01&endday=2017-09-01&tree=all

Comment 38

2 months ago
15 failures in 62 pushes (0.242 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 8
* autoland: 5
* mozilla-central: 2

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-02&endday=2017-09-02&tree=all
https://treeherder.mozilla.org/logviewer.html#?job_id=128115082&repo=mozilla-inbound - non-e10s Win7 debug devtools, so good luck with just not running reftests.
(Assignee)

Comment 40

2 months ago
if 98% of the failures go away with reftest, then we will win.

Comment 41

2 months ago
182 failures in 939 pushes (0.194 failures/push) were associated with this bug in the last 7 days. 

This is the #4 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 84
* mozilla-inbound: 61
* mozilla-central: 23
* try: 14

Platform breakdown:
* windows7-32: 181
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 42

a month ago
27 failures in 118 pushes (0.229 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 12
* mozilla-inbound: 11
* mozilla-central: 3
* try: 1

Platform breakdown:
* windows7-32: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-04&endday=2017-09-04&tree=all

Comment 43

a month ago
77 failures in 155 pushes (0.497 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 34
* autoland: 30
* mozilla-central: 9
* try: 4

Platform breakdown:
* windows7-32: 77

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-05&endday=2017-09-05&tree=all

Updated

a month ago
Attachment #8903174 - Flags: review?(bugs) → review+

Comment 44

a month ago
This patch disables reftests on win32/debug/non-e10s. I gave it an r+ given the frequency of this failure on this build config, and after receiving a release-drivers notice today that e10s-multi is shipping to 100% of eligible users on the 55 Release.

mrbkap: please comment if you're concerned that we're losing too much test coverage too soon here. Thx!
Flags: needinfo?(mrbkap)
What is the definition of "eligible users"?  What fraction of our Windows user base is _not_ getting e10s-multi?

Also, do we have non-debug reftest coverage on win32/e10s?
Flags: needinfo?(bugs)
(Assignee)

Comment 46

a month ago
this would only be disable don win7-non-e10s (which is only run on debug for win7).  We have opt/debug/pgo reftests on windows for e10s coverage still.
Sorry, I misspoke.  Do we still have non-debug reftest coverage on win32/non-e10s?
(Assignee)

Comment 48

a month ago
no, this is it for win32.  We do have android reftest non-e10s coverage, and some on linux when profiling with jsdcov.
OK.  Then my main question remains: are we still shipping win32/non-e10s to users?
I just had the same conversation with Jet. Unfortunately, "eligible users" is still not 100% of all users. In particular, we have users on old-style extensions that disable e10s on 55 and 56 (release and beta currently) as well as users with a11y enabled that disable e10s everywhere.

I don't know what the breakdown of those users is wrt win32 vs win64. Jim, would you know?
Flags: needinfo?(mrbkap) → needinfo?(jmathies)

Updated

a month ago
Flags: needinfo?(bugs)
Attachment #8903174 - Flags: review+ → review?(bugs)

Comment 51

a month ago
I spoke with dbolter on irc about this. A11y users on win32 still get non-e10s and will continue to do so at least through FF56. A go/no-go decision for A11y+e10s on FF57 is expected this Friday. Per dbolter: "Basically we'd like a week before maybe saying yeah turn off single process tests. I'm pushing hard to ship."

Let's revisit this one next week. Resetting the r? on the patch.
(Assignee)

Comment 52

a month ago
sounds good, thanks for the discussion here

Comment 53

a month ago
69 failures in 205 pushes (0.337 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 36
* mozilla-inbound: 27
* mozilla-central: 3
* try: 2
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 69

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-06&endday=2017-09-06&tree=all

Comment 54

a month ago
(In reply to Boris Zbarsky [:bz] (still digging out from vacation mail) from comment #49)
> OK.  Then my main question remains: are we still shipping win32/non-e10s to
> users?

Sure, accessibility users and incompat add-on users, plus anyone who disables e10s. Our 64-bit distro #s are still really small (2% of release I think?) but this is changing (starting on beta this week) as we've started migrating 32-bit users on 64-bit machines over to 64-bit builds.
Flags: needinfo?(jmathies)

Comment 55

a month ago
93 failures in 173 pushes (0.538 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 55
* autoland: 31
* mozilla-central: 4
* try: 3

Platform breakdown:
* windows7-32: 93

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-07&endday=2017-09-07&tree=all

Updated

a month ago
Priority: -- → P3

Comment 56

a month ago
78 failures in 173 pushes (0.451 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 51
* mozilla-inbound: 14
* mozilla-central: 8
* try: 5

Platform breakdown:
* windows7-32: 77
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-08&endday=2017-09-08&tree=all

Comment 57

a month ago
396 failures in 924 pushes (0.429 failures/push) were associated with this bug in the last 7 days. 

This is the #1 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 189
* mozilla-inbound: 151
* mozilla-central: 34
* try: 20
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 395
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-04&endday=2017-09-10&tree=all
(Assignee)

Comment 58

a month ago
this continues to be our #1 intermittent, who is working on fixing this?
Flags: needinfo?(bugs)

Comment 59

a month ago
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #58)
> this continues to be our #1 intermittent, who is working on fixing this?

@dbolter asked to have until Friday (9/15) for the decision status on shipping a11y+e10s which will reduce the need for these tests on this configuration.
Flags: needinfo?(bugs) → needinfo?(dbolter)
Correct, agreed. This NI should make me come back Friday, but if I don't please feel free to ping. Note while there is a chance we'll ride 57 to Beta, it doesn't mean we'll stick.
Flags: needinfo?(dbolter)
Flags: needinfo?(dbolter)

Updated

a month ago
Blocks: 1393934

Comment 61

a month ago
18 failures in 247 pushes (0.073 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* windows7-32: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-13&endday=2017-09-13&tree=all

Comment 62

a month ago
6 failures in 191 pushes (0.031 failures/push) were associated with this bug yesterday.    

** This test has failed more than 200 times in the last 30 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 2
* autoland: 2

Platform breakdown:
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-14&endday=2017-09-14&tree=all
Whiteboard: [stockwell needswork] → [stockwell disable-recommended]

Comment 63

a month ago
51 failures in 1032 pushes (0.049 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 21
* mozilla-inbound: 13
* mozilla-central: 11
* try: 5
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 51

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-11&endday=2017-09-17&tree=all
(Assignee)

Comment 64

a month ago
:davidb, checking in with you here- I assume all is well?
(Assignee)

Updated

a month ago
See Also: → bug 1393934
(Assignee)

Comment 65

a month ago
chatted with :davidb online, he would like to see us keep reftests in non-e10s mode around at until we ship with a11y+e10s (ideally 57 release in November).  

I thought of solving this problem another way, what if we run the tests in many chunks- this has the browser session doing a lot less, and in the end I have ~1000 reftest jobs and 1 instance of this:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1ce7c0b8206e16566c625d03279dd2e276dbd22b

We typically see 400-500 jobs/week for reftest-non-e10s, so this appears to be a significant win for this bug and bug 1393934.

:gbrown, what do you think of this approach?
Flags: needinfo?(gbrown)
That result is slightly inconsistent with my experience in comment 34...but you used more chunks, so maybe it will work? Let's try it!
Flags: needinfo?(gbrown)
(Assignee)

Comment 67

a month ago
yeah, I looked back at that try push, 16 chunks with 5 data points/chunk (80 jobs total) and I see 2 instances of this error.

Now to figure out how to chunk 32 times on non-e10s only!
(Assignee)

Comment 68

a month ago
Created attachment 8909459 [details] [diff] [review]
win7/debug non-e10s reftests at 32 chunks

This is the simplest way I could think of solving this bug- I am open to other ideas if you have them.
Attachment #8909459 - Flags: review?(gbrown)
Comment on attachment 8909459 [details] [diff] [review]
win7/debug non-e10s reftests at 32 chunks

Review of attachment 8909459 [details] [diff] [review]:
-----------------------------------------------------------------

I don't have a better suggestion. Thanks for calling it out as a hack.

How can we help ensure this gets cleaned up when we stop running non-e10s? Maybe a note in tests.yml?
Attachment #8909459 - Flags: review?(gbrown) → review+

Comment 70

a month ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4ac60362e1cc
split reftest non-e10s into 32 chunks. r=gbrown

Comment 71

a month ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/4ac60362e1cc
Status: NEW → RESOLVED
Last Resolved: a month ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Blocks: 1401228
Assignee: nobody → jmaher
status-firefox55: --- → wontfix
status-firefox56: --- → wontfix
status-firefox-esr52: --- → wontfix
Flags: needinfo?(dbolter)

Comment 72

24 days ago
6 failures in 943 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 4
* try: 1
* mozilla-release: 1

Platform breakdown:
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 73

17 days ago
1 failures in 885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1373578&startday=2017-09-25&endday=2017-10-01&tree=all
(Assignee)

Updated

10 days ago
Whiteboard: [stockwell disable-recommended] → [stockwell disabled]
You need to log in before you can comment on or make changes to this bug.