Closed Bug 1510328 Opened 3 years ago Closed 2 years ago

Intermittent marionette.py | application crashed [@ xul.dll + 0x3b066a | arena_t::MallocSmall(unsigned __int64,bool)]

Categories

(Core :: Graphics: WebRender, defect, P5)

defect

Tracking

()

RESOLVED FIXED
mozilla66
Tracking Status
firefox-esr60 --- unaffected
firefox64 --- unaffected
firefox65 --- fixed
firefox66 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: bholley)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [stockwell unknown])

Crash Data

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=214148465&repo=autoland

https://queue.taskcluster.net/v1/task/AHryWODGSz618DNAoHgHdA/runs/1/artifacts/public/logs/live_backing.log

16:52:41     INFO - closing window: 49392123905
16:52:41     INFO - closing window: 51539607553
16:52:42     INFO - closing window: 6442451084
16:52:42     INFO - closing window: 38654705668
16:52:42     INFO - closing window: 40802189316
16:52:42     INFO - closing window: 42949672964
16:52:42     INFO - closing window: 45097156613
16:52:42     INFO - closing window: 47244640260
16:52:42     INFO - closing window: 49392123908
16:52:42     INFO - closing window: 51539607556
16:52:42     INFO - closing window: 6442451087
16:52:42     INFO - closing window: 38654705671
16:52:42     INFO - closing window: 40802189333
16:52:42     INFO - closing window: 42949672967
16:52:42     INFO - closing window: 45097156618
16:52:42     INFO - closing window: 47244640309
16:52:42     INFO - closing window: 49392123912
16:52:42     INFO - closing window: 51539607559
16:52:42     INFO - closing window: 6442451090
16:52:43     INFO - closing window: 38654705688
16:52:43     INFO - closing window: 40802189336
16:52:43     INFO - closing window: 42949672970
16:52:43     INFO - closing window: 45097156637
16:52:43     INFO - closing window: 47244640316
16:52:43     INFO - navigating to about:blank
16:52:43     INFO - navigated to about:blank
16:52:43     INFO - starting checkpoint TabsClosedExtraProcesses...
16:52:44     INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedExtraProcesses-2.json.gz
16:52:44     INFO - closing preloaded browser
16:52:44     INFO - starting checkpoint TabsClosed...
16:52:44     INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosed-2.json.gz
16:53:14     INFO - starting checkpoint TabsClosedSettled...
16:53:14     INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedSettled-2.json.gz
16:53:14     INFO - starting gc...
16:53:15     INFO - gc done!
16:53:15     INFO - starting checkpoint TabsClosedForceGC...
16:53:15     INFO - checkpoint created, stored in Z:\\task_1543333681\\build\\tests\\results\\memory-report-TabsClosedForceGC-2.json.gz
16:53:15     INFO - setting results
16:53:15     INFO - tearing down!
16:53:15     INFO - tearing down webservers!
16:53:40     INFO - processing data in Z:\task_1543333681\build\tests\results!
16:53:45     INFO - PERFHERDER_DATA: {"framework": {"name": "awsy"}, "suites": [{"extraOptions": ["stylo"], "name": "Resident Memory", "lowerIsBetter": true, "value": 588044763.2862101, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 334770176}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 293023744}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 1466859520}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 1456590848}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 990027776}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 472899584}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 472961024}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 439386112}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 412315648}]}, {"extraOptions": ["stylo"], "name": "Explicit Memory", "lowerIsBetter": true, "value": 373957347.8124979, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 217794071}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 166168087}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 1136501248}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 1114325504}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 754332160}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 277144064}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 277500416}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 244990464}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 219554304}]}, {"extraOptions": ["stylo"], "name": "Heap Unclassified", "lowerIsBetter": true, "value": 77751137.45692353, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 46842352}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 32905520}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 242183872}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 238676768}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 121183200}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 58635184}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 58318976}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 54671568}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 51442944}]}, {"extraOptions": ["stylo"], "name": "JS", "lowerIsBetter": true, "value": 116079035.9332693, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 104560400}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 62133840}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 255522128}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 242813408}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 209321888}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 92966432}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 93576464}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 75565376}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 68985408}]}, {"extraOptions": ["stylo"], "name": "Images", "lowerIsBetter": true, "value": 7259779.581239595, "units": "bytes", "subtests": [{"lowerIsBetter": true, "units": "bytes", "name": "Fresh start", "value": 1034192}, {"lowerIsBetter": true, "units": "bytes", "name": "Fresh start [+30s]", "value": 1034192}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open", "value": 78198272}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s]", "value": 67746528}, {"lowerIsBetter": true, "units": "bytes", "name": "After tabs open [+30s, forced GC]", "value": 32901056}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed extra processes", "value": 6656944}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed", "value": 6656944}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s]", "value": 3179200}, {"lowerIsBetter": true, "units": "bytes", "name": "Tabs closed [+30s, forced GC]", "value": 2132784}]}]}
16:53:45     INFO - Perfherder data written to Z:\task_1543333681\build\tests\results\perfherder_data.json
16:53:45     INFO - done tearing down!
16:53:45     INFO - TEST-PASS | awsy\test_memory_usage.py TestMemoryUsage.test_open_tabs | took 3569640ms
16:53:45     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cBAzpjbJQouxYUy_fpQ6iQ/artifacts/public/build/target.crashreporter-symbols.zip
16:53:50     INFO - mozcrash Copy/paste: Z:\task_1543333681\build\win32-minidump_stackwalk.exe c:\users\task_1543333681\appdata\local\temp\tmpbq0yvh.profile\minidumps\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp c:\users\task_1543333681\appdata\local\temp\tmpb2olby
16:53:52     INFO - mozcrash Saved minidump as Z:\task_1543333681\build\blobber_upload_dir\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp
16:53:52     INFO - mozcrash Saved app info as Z:\task_1543333681\build\blobber_upload_dir\9667aee6-f2da-4e97-8a41-de5a3d6f851d.extra
16:53:52     INFO - PROCESS-CRASH | marionette.py | application crashed [@ xul.dll + 0x3b066a]
16:53:52     INFO - Crash dump filename: c:\users\task_1543333681\appdata\local\temp\tmpbq0yvh.profile\minidumps\9667aee6-f2da-4e97-8a41-de5a3d6f851d.dmp
16:53:52     INFO - Operating system: Windows NT
16:53:52     INFO -                   10.0.15063 
16:53:52     INFO - CPU: amd64
16:53:52     INFO -      family 6 model 79 stepping 1
16:53:52     INFO -      16 CPUs
16:53:52     INFO - 
16:53:52     INFO - GPU: UNKNOWN
16:53:52     INFO - 
16:53:52     INFO - Crash reason:  EXCEPTION_BREAKPOINT
16:53:52     INFO - Crash address: 0x7ff81fb1066a
16:53:52     INFO - Assertion: Unknown assertion type 0x00000000
16:53:52     INFO - Process uptime: 623 seconds
16:53:52     INFO - 
16:53:52     INFO - Thread 33 (crashed)
16:53:52     INFO -  0  xul.dll + 0x3b066a
16:53:52     INFO -     rax = 0x00007ff83e97fa88   rdx = 0x0000007013b38e92
16:53:52     INFO -     rcx = 0x00000000000003e8   rbx = 0x00007ff824228e6d
16:53:52     INFO -     rsi = 0x0000000000000011   rdi = 0x00000000000003e8
16:53:52     INFO -     rbp = 0x0000007013b39438   rsp = 0x0000007013b38e10
16:53:52     INFO -      r8 = 0x0000007013b38e92    r9 = 0x0000000000000000
16:53:52     INFO -     r10 = 0x0000007013b38850   r11 = 0x0000000000000011
16:53:52     INFO -     r12 = 0x00007ff82422a690   r13 = 0x00007ff824223d30
16:53:52     INFO -     r14 = 0x0000007013b396a0   r15 = 0x0000000000000001
16:53:52     INFO -     rip = 0x00007ff81fb1066a
16:53:52     INFO -     Found by: given as instruction pointer in context
16:53:52     INFO -  1  0x21c4b96b5a0
16:53:52     INFO -     rbp = 0x0000007013b39538   rsp = 0x0000007013b39448
16:53:52     INFO -     rip = 0x0000021c4b96b5a0
16:53:52     INFO -     Found by: previous frame's frame pointer
16:53:52     INFO -  2  0x1
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39548
16:53:52     INFO -     rip = 0x0000000000000001
16:53:52     INFO -     Found by: previous frame's frame pointer
16:53:52     INFO -  3  mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 2988 + 0x9]
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39580
16:53:52     INFO -     rip = 0x00007ff83e951b71
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  4  mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 4192 + 0x23]
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b395f0
16:53:52     INFO -     rip = 0x00007ff83e952886
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  5  mozglue.dll!Allocator<MozJemallocBase>::malloc(unsigned __int64) [malloc_decls.h:ad28397dfa43589ea5bd814391c26386c5bf751f : 37 + 0x18]
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39640
16:53:52     INFO -     rip = 0x00007ff83e94e28d
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  6  mozglue.dll!arena_t::RallocSmallOrLarge(void *,unsigned __int64,unsigned __int64) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 3660 + 0x17]
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39730
16:53:52     INFO -     rip = 0x00007ff83e94d850
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  7  mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:ad28397dfa43589ea5bd814391c26386c5bf751f : 2988 + 0x9]
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39740
16:53:52     INFO -     rip = 0x00007ff83e951b71
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  8  ntdll.dll!RtlWakeAllConditionVariable + 0x61
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39770
16:53:52     INFO -     rip = 0x00007ff8527960f1
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO -  9  0x21c48d12490
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b397a0
16:53:52     INFO -     rip = 0x0000021c48d12490
16:53:52     INFO -     Found by: call frame info
16:53:52     INFO - 10  KERNELBASE.dll!Normalization__LoadTables + 0x94
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39818
16:53:52     INFO -     rip = 0x00007ff84ed6e910
16:53:52     INFO -     Found by: stack scanning
16:53:52     INFO - 11  0x7ff800000000
16:53:52     INFO -     rbp = 0x00007ff824229800   rsp = 0x0000007013b39848
16:53:52     INFO -     rip = 0x00007ff800000000
16:53:52     INFO -     Found by: call frame info
Component: Marionette → AWSY
Crash Signature: [@ xul.dll + 0x3b066a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a]
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a]
Component: AWSY → Memory Allocator
Product: Testing → Core
Summary: Intermittent marionette.py | application crashed [@ xul.dll + 0x3b066a] → Intermittent marionette.py | application crashed [@ xul.dll + 0x3b066a | arena_t::MallocSmall(unsigned __int64,bool)]
Version: Version 3 → unspecified
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a]
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca]
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a]
I'm more suspicious of bug 1509974, which landed two commits earlier. I did some more retriggers to check.
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] [@ xul.dll + 0x3b112a]
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] [@ xul.dll + 0x3b112a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] [@ xul.dll + 0x3b112a] [@ xul.dll + 0x3b187a]
Confirmed with retriggers that this was bug 1509974.

I'm looking at this with glandium. Stack is garbage but with some minidump analysis he's confirmed it's a rust panic.
Blocks: 1509974
Flags: needinfo?(poirot.alex)
Assignee: nobody → bobbyholley
I've tried triggering the same job with debug builds to see if we do a better job of either walking the stack or printing the panic message.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=246bc97e2a5926e4bd0c43bde3ea562e93df262f
The panic message is "no entry found for key", which corresponds to a hashmap[foo] operations. Likely source file option.rs.
The crashing thread name is "WRSceneBuilder#2" on the crash I looked at manually.
Component: Memory Allocator → Graphics: WebRender
This might be the location that panics: xul.dll!static void webrender::prim_store::PrimitiveStore::prepare_interned_prim_for_render(struct webrender::prim_store::PrimitiveInstance *, struct webrender::frame_builder::PrimitiveContext *, struct webrender::frame_builder::PictureContext *, struct webrender::frame_builder::FrameBuildingContext *, struct webrender::frame_builder::FrameBuildingState *, struct webrender::render_backend::FrameResources *, struct webrender::prim_store::PrimitiveScratchBuffer *) [https://hg.mozilla.org/try/annotate/c77072391f95afd0c07fbb9b0c92028a6cf44f3d/gfx/wr/webrender/src/prim_store.rs#l3469 + 130]
So, the good news is that by running minidump_stackwalk manually on one of the crashes, I actually *do* a stack trace that looks good:

Thread 33 (crashed)
 0  xul.dll!static void MOZ_CrashOOL(const char *, int, const char *) [Assertions.h:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 311 + 0x0]
    rax = 0x00007ffb30fdfa88   rdx = 0x0000008e4d638c52
    rcx = 0x00000000000003e8   rbx = 0x00007ffb14da6e6d
    rsi = 0x0000000000000011   rdi = 0x00000000000003e8
    rbp = 0x0000008e4d6391f8   rsp = 0x0000008e4d638bd0
     r8 = 0x0000008e4d638c52    r9 = 0x0000000000000000
    r10 = 0x0000008e4d638610   r11 = 0x0000000000000011
    r12 = 0x00007ffb14da8690   r13 = 0x00007ffb14da1d30
    r14 = 0x0000008e4d639460   r15 = 0x0000000000000001
    rip = 0x00007ffb1069073a
    Found by: given as instruction pointer in context
 1  xul.dll!GeckoCrashOOL [nsAppRunner.cpp:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 5350 + 0xa]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d638c00   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb137405de
    Found by: call frame info
 2  xul.dll!static void gkrust_shared::panic_hook(struct core::panic::PanicInfo *) [lib.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 234 + 0x7]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d638c30   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13b5019b
    Found by: call frame info
 3  xul.dll!static void core::ops::function::Fn::call<fn(core::panic::PanicInfo*),(core::panic::PanicInfo*)>( * *, struct core::panic::PanicInfo *) [function.rs : 78 + 0xc]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639080   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13b500ec
    Found by: call frame info
 4  xul.dll!static void std::panicking::rust_panic_with_hook() [panicking.rs : 481 + 0x6]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d6390b0   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb14206df1
    Found by: call frame info
 5  xul.dll!static void std::panicking::continue_panic_fmt() [panicking.rs : 391 + 0x16]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d6391c0   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb142096e1
    Found by: call frame info
 6  xul.dll!static void std::panicking::rust_begin_panic() [panicking.rs : 326 + 0x5]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639220   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb14209689
    Found by: call frame info
 7  xul.dll!static void core::panicking::panic_fmt() [panicking.rs : 77 + 0x5]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639250   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb1422656c
    Found by: call frame info
 8  xul.dll!static void core::option::expect_failed() [option.rs : 1000 + 0x11]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d6392b0   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb1421c562
    Found by: call frame info
 9  xul.dll!static union core::option::Option<webrender_bindings::moz2d_renderer::Job> core::iter::{{impl}}::next<webrender_bindings::moz2d_renderer::Job,core::slice::Iter<webrender_api::image::BlobImageParams>,closure>(struct core::iter::Map<core::slice::Iter<webrender_api::image::BlobImageParams>, closure> *) [mod.rs : 1394 + 0xb8]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639330   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13cc49d9
    Found by: call frame info
10  xul.dll!static struct alloc::vec::Vec<(webrender_api::image::BlobImageRequest, core::result::Result<webrender_api::image::RasterizedBlobImage, webrender_api::image::BlobImageError>)> webrender_bindings::moz2d_renderer::{{impl}}::rasterize(struct webrender_bindings::moz2d_renderer::Moz2dBlobRasterizer *, struct slice<webrender_api::image::BlobImageParams>*, bool) [moz2d_renderer.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 474 + 0x2f]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639440   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13cc43b5
    Found by: call frame info
11  xul.dll!static struct webrender::tiling::Frame webrender::frame_builder::FrameBuilder::build(struct webrender::resource_cache::ResourceCache *, struct webrender::gpu_cache::GpuCache *, struct webrender::render_backend::FrameStamp, struct webrender::clip_scroll_tree::ClipScrollTree *, struct std::collections::hash::map::HashMap<webrender_api::api::PipelineId, alloc::sync::Arc<webrender::scene::ScenePipeline>, core::hash::BuildHasherDefault<fxhash::FxHasher>> *, struct euclid::scale::TypedScale<f32, webrender_api::units::WorldPixel, webrender_api::units::DevicePixel>, char, struct euclid::point::TypedPoint2D<f32, webrender_api::units::WorldPixel>, struct webrender::profiler::TextureCacheProfileCounters *, struct webrender::profiler::GpuCacheProfileCounters *, struct webrender::scene::SceneProperties *, struct webrender::render_backend::FrameResources *, struct webrender::prim_store::PrimitiveScratchBuffer *) [frame_builder.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 375 + 0x2645]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d639600   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13da46c0
    Found by: call frame info
12  xul.dll!static struct webrender::internal_types::RenderedDocument webrender::render_backend::Document::build_frame(struct webrender::resource_cache::ResourceCache *, struct webrender::gpu_cache::GpuCache *, struct webrender::profiler::ResourceProfileCounters *) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 418 + 0x4a]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63a840   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d82f2c
    Found by: call frame info
13  xul.dll!static void webrender::render_backend::RenderBackend::update_document(struct webrender_api::api::DocumentId, struct alloc::vec::Vec<webrender_api::api::ResourceUpdate>, union core::option::Option<webrender::scene_builder::DocumentResourceUpdates>, struct alloc::vec::Vec<webrender_api::api::FrameMsg>, struct alloc::vec::Vec<webrender_api::api::NotificationRequest>, bool, bool, unsigned int *, struct webrender::profiler::BackendProfileCounters *, bool) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 1219 + 0x23]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63a960   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d5dc68
    Found by: call frame info
14  xul.dll!static bool webrender::render_backend::RenderBackend::process_api_msg(union webrender_api::api::ApiMsg, struct webrender::profiler::BackendProfileCounters *, unsigned int *) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 1032 + 0x13d]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63bce0   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d695f7
    Found by: call frame info
15  xul.dll!static void webrender::render_backend::RenderBackend::run(struct webrender::profiler::BackendProfileCounters) [render_backend.rs:c77072391f95afd0c07fbb9b0c92028a6cf44f3d : 807 + 0xa1]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63d820   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d4ce22
    Found by: call frame info
16  xul.dll!static void std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()>(struct closure) [backtrace.rs : 136 + 0xdf7]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63e560   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d49447
    Found by: call frame info
17  xul.dll!static void alloc::boxed::{{impl}}::call_box<(),closure>(struct closure *, <NoType>) [boxed.rs : 646 + 0x5b]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63f470   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb13d4856d
    Found by: call frame info
18  xul.dll!static void std::sys::windows::thread::{{impl}}::new::thread_start() [thread.rs : 56 + 0x64]
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63fb90   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb1420f20d
    Found by: call frame info
19  kernel32.dll + 0x12774
    rbx = 0x00007ffb14da6e6d   rbp = 0x0000008e4d6391f8
    rsp = 0x0000008e4d63fbd0   r12 = 0x00007ffb14da8690
    r13 = 0x00007ffb14da1d30   r14 = 0x0000008e4d639460
    r15 = 0x0000000000000001   rip = 0x00007ffb48122774
    Found by: call frame info


This suggests the AWSY harness is not getting or using the crash symbols correctly.
I also got stacks by forcing AWSY to run with a debug build [1]. According to aosmond, this should be fixed by bug 1492241.

Thanks for helping track this down glandium!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=246bc97e2a5926e4bd0c43bde3ea562e93df262f
Depends on: 1492241
Thanks for looking into this Bobby, sorry i did not continue with retriggers and you had to.
Looks fixed \o/

(In reply to Andreea Pavel [:apavel] from comment #14)
> Thanks for looking into this Bobby, sorry i did not continue with retriggers
> and you had to.

No worries!
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Bobby, this is not fixed: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&group_state=expanded&fromchange=82c890b2e1ec940404e5789044d2bcc1b9ad2cd9&selectedJob=215854334

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215854334&repo=autoland&lineNumber=2289

00:32:09     INFO - TEST-PASS | awsy\test_memory_usage.py TestMemoryUsage.test_open_tabs | took 3488060ms
00:32:09     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/CqGiHo1bTL-eXGWRAnPpzA/artifacts/public/build/target.crashreporter-symbols.zip
00:32:13     INFO - mozcrash Copy/paste: Z:\task_1544138790\build\win32-minidump_stackwalk.exe c:\users\task_1544138790\appdata\local\temp\tmp_xqtyh.profile\minidumps\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp c:\users\task_1544138790\appdata\local\temp\tmpbp4tbd
00:32:14     INFO - mozcrash Saved minidump as Z:\task_1544138790\build\blobber_upload_dir\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp
00:32:14     INFO - mozcrash Saved app info as Z:\task_1544138790\build\blobber_upload_dir\3108ad40-f986-4b97-a90d-34359f7b54c3.extra
00:32:14     INFO - PROCESS-CRASH | marionette.py | application crashed [@ xul.dll + 0x3b196a]
00:32:14     INFO - Crash dump filename: c:\users\task_1544138790\appdata\local\temp\tmp_xqtyh.profile\minidumps\3108ad40-f986-4b97-a90d-34359f7b54c3.dmp
00:32:14     INFO - Operating system: Windows NT
00:32:14     INFO -                   10.0.15063 
00:32:14     INFO - CPU: amd64
00:32:14     INFO -      family 6 model 79 stepping 1
00:32:14     INFO -      16 CPUs
00:32:14     INFO - 
00:32:14     INFO - GPU: UNKNOWN
00:32:14     INFO - 
00:32:14     INFO - Crash reason:  EXCEPTION_BREAKPOINT
00:32:14     INFO - Crash address: 0x7ffbfc30196a
00:32:14     INFO - Assertion: Unknown assertion type 0x00000000
00:32:14     INFO - Process uptime: 3428 seconds
00:32:14     INFO - 
00:32:14     INFO - Thread 31 (crashed)
00:32:14     INFO -  0  xul.dll + 0x3b196a
00:32:14     INFO -     rax = 0x00007ffc216cfa88   rdx = 0x000000ada41b7e32
00:32:14     INFO -     rcx = 0x00000000000003e8   rbx = 0x00007ffc00b1c91d
00:32:14     INFO -     rsi = 0x0000000000000011   rdi = 0x00000000000003e8
00:32:14     INFO -     rbp = 0x000000ada41b83d8   rsp = 0x000000ada41b7db0
00:32:14     INFO -      r8 = 0x000000ada41b7e32    r9 = 0x0000000000000000
00:32:14     INFO -     r10 = 0x000000ada41b77f0   r11 = 0x0000000000000011
00:32:14     INFO -     r12 = 0x00007ffc00b1e140   r13 = 0x00007ffc00b177e0
00:32:14     INFO -     r14 = 0x000000ada41b8640   r15 = 0x0000000000000001
00:32:14     INFO -     rip = 0x00007ffbfc30196a
00:32:14     INFO -     Found by: given as instruction pointer in context
00:32:14     INFO -  1  0x23b1fafd7a0
00:32:14     INFO -     rbp = 0x000000ada41b84d8   rsp = 0x000000ada41b83e8
00:32:14     INFO -     rip = 0x0000023b1fafd7a0
00:32:14     INFO -     Found by: previous frame's frame pointer
00:32:14     INFO -  2  0x1
00:32:14     INFO -     rbp = 0x00007ffc00b1d2b0   rsp = 0x000000ada41b84e8
00:32:14     INFO -     rip = 0x0000000000000001
00:32:14     INFO -     Found by: previous frame's frame pointer
00:32:14     INFO -  3  mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:3347058499c706d9651ad1cfa3aaae3c1d45bbe7 : 2804 + 0x9]
00:32:14     INFO -     rbp = 0x00007ffc00b1d2b0   rsp = 0x000000ada41b8520
00:32:14     INFO -     rip = 0x00007ffc216a1b71
00:32:14     INFO -     Found by: stack scanning
00:32:14     INFO -  4  mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:3347058499c706d9651ad1cfa3aaae3c1d45bbe7 : 3935 + 0x23]
00:32:14     INFO -     rbp = 0x00007ffc00b1d2b0   rsp = 0x000000ada41b8590
00:32:14     INFO -     rip = 0x00007ffc216a2846
00:32:14     INFO -     Found by: stack scanning
Status: RESOLVED → REOPENED
Flags: needinfo?(bobbyholley)
Resolution: FIXED → ---
Doing another try push to run AWSY on a debug build to get stacks: https://treeherder.mozilla.org/#/jobs?repo=try&revision=93270b5cea3ead7db5d71ce4967679ab3d8cbf69
(In reply to Bobby Holley (:bholley) from comment #17)
> Doing another try push to run AWSY on a debug build to get stacks:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=93270b5cea3ead7db5d71ce4967679ab3d8cbf69

Andrew, see above. Looks like we're still crashing even with the fix from bug 1512156.
Flags: needinfo?(bobbyholley) → needinfo?(aosmond)
Crash Signature: [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] [@ xul.dll + 0x3b112a] [@ xul.dll + 0x3b187a] → [@ xul.dll + 0x3b066a] [@ xul.dll + 0x3b072a] [@ xul.dll + 0x3b063a] [@ xul.dll + 0x3b064a] [@ xul.dll + 0x3b0cca] [@ xul.dll + 0x3b0e1a] [@ xul.dll + 0x3b112a] [@ xul.dll + 0x3b187a] [@ xul.dll + 0x3b196a]
Sounds like https://github.com/servo/webrender/pull/3398 might help here.
(In reply to Bobby Holley (:bholley) from comment #20)
> Sounds like https://github.com/servo/webrender/pull/3398 might help here.

Yes -- sorry meant to follow up.
Flags: needinfo?(aosmond)
(In reply to Bobby Holley (:bholley) from comment #20)
> Sounds like https://github.com/servo/webrender/pull/3398 might help here.

So far so good, I don't see any failures after [1]. Andreea, can you double-check and close this bug if so?

[1] https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=sy&revision=5d24ac329431289a8966eabc3325228dfd7f2c4d
Flags: needinfo?(apavel)
This looks fixed based on the TH intermittent-failures listings. Feel free to reopen if it comes back.
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → FIXED
Flags: needinfo?(apavel)
Target Milestone: --- → mozilla66
You need to log in before you can comment on or make changes to this bug.