Closed Bug 1600735 Opened 5 years ago Closed 4 years ago

crash in [@ Servo_IsUnknownPropertyRecordedInUseCounter]

Categories

(Core :: CSS Parsing and Computation, defect, P4)

defect

Tracking

()

RESOLVED FIXED
mozilla73
Tracking Status
firefox-esr68 --- unaffected
firefox71 --- unaffected
firefox72 --- wontfix
firefox73 --- fixed

People

(Reporter: tsmith, Assigned: emilio)

References

(Blocks 1 open bug)

Details

(Keywords: crash)

Attachments

(4 files)

The fuzzers have been hitting this for a while now but it does not seem to be reproducable. Seems to be hit on fuzzing-debug and ccov builds.

First seen on m-c 20191006-3fa65bda1e50

Report from m-c 20191202-778b6b11194c

rax = 0x0000000000000060   rdx = 0x2000000000000000
rcx = 0x00002c513fae4c7d   rbx = 0x0000000000000000
rsi = 0x0000000000000300   rdi = 0x00007f184d2fefa0
rbp = 0x00007ffdf67062f0   rsp = 0x00007ffdf67062f0
r8 = 0x0000000000000000    r9 = 0x00007f184c10966c
r10 = 0x000000004c100008   r11 = 0x00007f184c10965f
r12 = 0x00007f184d2b6000   r13 = 0x00007f184d2fefa0
r14 = 0x00007f1866e6e278   r15 = 0x00007f184d65d2d0
rip = 0x00007f185a06e3dc
OS|Linux|0.0.0 Linux 4.19.34-coreos #1 SMP Mon Apr 22 20:32:34 -00 2019 x86_64
CPU|amd64|family 6 model 79 stepping 1|64
GPU|||
Crash|SIGSEGV|0x7f184d2ff048|0
0|0|libxul.so|Servo_IsUnknownPropertyRecordedInUseCounter|hg:hg.mozilla.org/mozilla-central:servo/ports/geckolib/glue.rs:778b6b11194c072d2603e58118aaf6959e98902f|6777|0x8
0|1|libxul.so|mozilla::dom::Document::SetCssUseCounterBits()|hg:hg.mozilla.org/mozilla-central:dom/base/Document.cpp:778b6b11194c072d2603e58118aaf6959e98902f|14276|0xb
0|2|libxul.so|mozilla::dom::Document::ReportUseCounters()|hg:hg.mozilla.org/mozilla-central:dom/base/Document.cpp:778b6b11194c072d2603e58118aaf6959e98902f|14320|0x5
0|3|libxul.so|mozilla::dom::Document::Destroy()|hg:hg.mozilla.org/mozilla-central:dom/base/Document.cpp:778b6b11194c072d2603e58118aaf6959e98902f|10277|0x8
0|4|libxul.so|nsDocumentViewer::Destroy()|hg:hg.mozilla.org/mozilla-central:layout/base/nsDocumentViewer.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1874|0x18
0|5|libxul.so|nsDocShell::Destroy()|hg:hg.mozilla.org/mozilla-central:docshell/base/nsDocShell.cpp:778b6b11194c072d2603e58118aaf6959e98902f|4684|0x14
0|6|libxul.so|nsWebBrowser::SetDocShell(nsIDocShell*)|hg:hg.mozilla.org/mozilla-central:toolkit/components/browser/nsWebBrowser.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1240|0x11
0|7|libxul.so|nsWebBrowser::InternalDestroy()|hg:hg.mozilla.org/mozilla-central:toolkit/components/browser/nsWebBrowser.cpp:778b6b11194c072d2603e58118aaf6959e98902f|202|0xa
0|8|libxul.so|nsWebBrowser::Destroy()|hg:hg.mozilla.org/mozilla-central:toolkit/components/browser/nsWebBrowser.cpp:778b6b11194c072d2603e58118aaf6959e98902f|917|0x5
0|9|libxul.so|mozilla::dom::BrowserChild::DestroyWindow()|hg:hg.mozilla.org/mozilla-central:dom/ipc/BrowserChild.cpp:778b6b11194c072d2603e58118aaf6959e98902f|998|0x18
0|10|libxul.so|mozilla::dom::BrowserChild::RecvDestroy()|hg:hg.mozilla.org/mozilla-central:dom/ipc/BrowserChild.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2488|0x8
0|11|libxul.so|mozilla::dom::PBrowserChild::OnMessageReceived(IPC::Message const&)|s3:gecko-generated-sources:289bc9d1a8753697164100bde482ea3411d20a8538e4fa2388a5ddea71fbb1f303b8d8725e565f8f956c951d9cbfc875ea04a058a53aaf0220296e826fef43eb/ipc/ipdl/PBrowserChild.cpp:|6326|0x9
0|12|libxul.so|mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&)|s3:gecko-generated-sources:65518ffd7b57fd1df7143167ed0bada056f671ca93262ea53afdd1cd9af4f7991c73326585ddf9dcd7dc58dce81c40fc4e3784fc15a692f8b3e7ae6e0b9e14a4/ipc/ipdl/PContentChild.cpp:|8168|0x15
0|13|libxul.so|mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2208|0x6
0|14|libxul.so|mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2130|0xb
0|15|libxul.so|mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1972|0xb
0|16|libxul.so|mozilla::ipc::MessageChannel::MessageTask::Run()|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessageChannel.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2003|0xc
0|17|libxul.so|mozilla::SchedulerGroup::Runnable::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/SchedulerGroup.cpp:778b6b11194c072d2603e58118aaf6959e98902f|295|0x15
0|18|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1250|0x15
0|19|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:778b6b11194c072d2603e58118aaf6959e98902f|486|0x11
0|20|libxul.so|mozilla::dom::ContentChild::ProvideWindowCommon(mozilla::dom::BrowserChild*, mozIDOMWindowProxy*, bool, unsigned int, bool, bool, bool, nsIURI*, nsTSubstring<char16_t> const&, nsTSubstring<char> const&, bool, bool, nsDocShellLoadState*, bool*, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:dom/ipc/ContentChild.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1254|0x77
0|21|libxul.so|mozilla::dom::BrowserChild::ProvideWindow(mozIDOMWindowProxy*, unsigned int, bool, bool, bool, nsIURI*, nsTSubstring<char16_t> const&, nsTSubstring<char> const&, bool, bool, nsDocShellLoadState*, bool*, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:dom/ipc/BrowserChild.cpp:778b6b11194c072d2603e58118aaf6959e98902f|956|0x1b
0|22|libxul.so|nsWindowWatcher::OpenWindowInternal(mozIDOMWindowProxy*, char const*, char const*, char const*, bool, bool, bool, nsIArray*, bool, bool, bool, nsDocShellLoadState*, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:toolkit/components/windowwatcher/nsWindowWatcher.cpp:778b6b11194c072d2603e58118aaf6959e98902f|807|0x3f
0|23|libxul.so|nsWindowWatcher::OpenWindow2(mozIDOMWindowProxy*, char const*, char const*, char const*, bool, bool, bool, nsISupports*, bool, bool, bool, nsDocShellLoadState*, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:toolkit/components/windowwatcher/nsWindowWatcher.cpp:778b6b11194c072d2603e58118aaf6959e98902f|381|0xa
0|24|libxul.so|nsGlobalWindowOuter::OpenInternal(nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, bool, bool, bool, bool, bool, nsIArray*, nsISupports*, nsDocShellLoadState*, bool, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:dom/base/nsGlobalWindowOuter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|7218|0x5f
0|25|libxul.so|nsGlobalWindowOuter::OpenJS(nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, mozilla::dom::BrowsingContext**)|hg:hg.mozilla.org/mozilla-central:dom/base/nsGlobalWindowOuter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|5767|0x18
0|26|libxul.so|nsGlobalWindowOuter::OpenOuter(nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, mozilla::ErrorResult&)|hg:hg.mozilla.org/mozilla-central:dom/base/nsGlobalWindowOuter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|5731|0x1f
0|27|libxul.so|nsGlobalWindowInner::Open(nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, mozilla::ErrorResult&)|hg:hg.mozilla.org/mozilla-central:dom/base/nsGlobalWindowInner.cpp:778b6b11194c072d2603e58118aaf6959e98902f|3732|0x29
0|28|libxul.so|mozilla::dom::Window_Binding::open|s3:gecko-generated-sources:5716d1d875843b868fa2c899a937865aabf3326a81fd1e81ab138d259b999f3b47fba929b13b4dd71bd23ba1a7833d42cc01def826eae3089ee297c1194aba02/dom/bindings/WindowBinding.cpp:|2623|0x2d
0|29|libxul.so|bool mozilla::dom::binding_detail::GenericMethod<mozilla::dom::binding_detail::MaybeCrossOriginObjectThisPolicy, mozilla::dom::binding_detail::ThrowExceptions>(JSContext*, unsigned int, JS::Value*)|hg:hg.mozilla.org/mozilla-central:dom/bindings/BindingUtils.cpp:778b6b11194c072d2603e58118aaf6959e98902f|3153|0x24
0|30|libxul.so|CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&)|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|456|0x15
0|31|libxul.so|js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason)|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|548|0x12
0|32|libxul.so|InternalCall|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|617|0x10
0|33|libxul.so|js::jit::DoCallFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICCall_Fallback*, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>)|hg:hg.mozilla.org/mozilla-central:js/src/jit/BaselineIC.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2940|0x13
0|34|||||0xd3585d8263
0|35|||||0x7f184d427c78
0|36|||||0xd3585cda64
0|37|libxul.so|EnterJit|hg:hg.mozilla.org/mozilla-central:js/src/jit/Jit.cpp:778b6b11194c072d2603e58118aaf6959e98902f|109|0x30
0|38|libxul.so|js::RunScript(JSContext*, js::RunState&)|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|408|0xb
0|39|libxul.so|js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason)|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|589|0xf
0|40|libxul.so|InternalCall|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|617|0x10
0|41|libxul.so|js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason)|hg:hg.mozilla.org/mozilla-central:js/src/vm/Interpreter.cpp:778b6b11194c072d2603e58118aaf6959e98902f|634|0x8
0|42|libxul.so|JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>)|hg:hg.mozilla.org/mozilla-central:js/src/jsapi.cpp:778b6b11194c072d2603e58118aaf6959e98902f|2752|0x1f
0|43|libxul.so|mozilla::dom::Function::Call(JSContext*, JS::Handle<JS::Value>, nsTArray<JS::Value> const&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&)|s3:gecko-generated-sources:a7c2e5fbc8d754dda6a548b7b47f3620d4c9b10ea79911d63eee97dce3943509e5dac2b094e8f79cc317b469bbff7142455a8edf1bb906af40be582eb3c8ce34/dom/bindings/FunctionBinding.cpp:|41|0x5
0|44|libxul.so|void mozilla::dom::Function::Call<nsCOMPtr<nsIGlobalObject> >(nsCOMPtr<nsIGlobalObject> const&, nsTArray<JS::Value> const&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&, char const*, mozilla::dom::CallbackObject::ExceptionHandling, JS::Realm*)|s3:gecko-generated-sources:46fb03c1a8c0e11d34f5e031c89d1a474e669ef986de20768612085e4233f6871c6674b6fc7e076da097557e955e8fa4dd7337ed126d39cecea8d5c6e05d7972/dist/include/mozilla/dom/FunctionBinding.h:|73|0x23
0|45|libxul.so|mozilla::dom::CallbackTimeoutHandler::Call(char const*)|hg:hg.mozilla.org/mozilla-central:dom/base/TimeoutHandler.cpp:778b6b11194c072d2603e58118aaf6959e98902f|167|0x27
0|46|libxul.so|nsGlobalWindowInner::RunTimeoutHandler(mozilla::dom::Timeout*, nsIScriptContext*)|hg:hg.mozilla.org/mozilla-central:dom/base/nsGlobalWindowInner.cpp:778b6b11194c072d2603e58118aaf6959e98902f|5904|0x14
0|47|libxul.so|mozilla::dom::TimeoutManager::RunTimeout(mozilla::TimeStamp const&, mozilla::TimeStamp const&, bool)|hg:hg.mozilla.org/mozilla-central:dom/base/TimeoutManager.cpp:778b6b11194c072d2603e58118aaf6959e98902f|892|0x1b
0|48|libxul.so|mozilla::dom::TimeoutExecutor::MaybeExecute()|hg:hg.mozilla.org/mozilla-central:dom/base/TimeoutExecutor.cpp:778b6b11194c072d2603e58118aaf6959e98902f|179|0x13
0|49|libxul.so|mozilla::dom::TimeoutExecutor::Notify(nsITimer*)|hg:hg.mozilla.org/mozilla-central:dom/base/TimeoutExecutor.cpp:778b6b11194c072d2603e58118aaf6959e98902f|246|0x5
0|50|libxul.so|nsTimerImpl::Fire(int)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsTimerImpl.cpp:778b6b11194c072d2603e58118aaf6959e98902f|564|0xe
0|51|libxul.so|nsTimerEvent::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/TimerThread.cpp:778b6b11194c072d2603e58118aaf6959e98902f|260|0x14
0|52|libxul.so|mozilla::ThrottledEventQueue::Inner::ExecuteRunnable()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/ThrottledEventQueue.cpp:778b6b11194c072d2603e58118aaf6959e98902f|252|0x11
0|53|libxul.so|mozilla::ThrottledEventQueue::Inner::Executor::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/ThrottledEventQueue.cpp:778b6b11194c072d2603e58118aaf6959e98902f|80|0xd
0|54|libxul.so|mozilla::SchedulerGroup::Runnable::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/SchedulerGroup.cpp:778b6b11194c072d2603e58118aaf6959e98902f|295|0x15
0|55|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:778b6b11194c072d2603e58118aaf6959e98902f|1250|0x15
0|56|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:778b6b11194c072d2603e58118aaf6959e98902f|486|0x11
0|57|libxul.so|mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:778b6b11194c072d2603e58118aaf6959e98902f|110|0xd
0|58|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:778b6b11194c072d2603e58118aaf6959e98902f|315|0x17
0|59|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:778b6b11194c072d2603e58118aaf6959e98902f|290|0x8
0|60|libxul.so|nsBaseAppShell::Run()|hg:hg.mozilla.org/mozilla-central:widget/nsBaseAppShell.cpp:778b6b11194c072d2603e58118aaf6959e98902f|137|0xd
0|61|libxul.so|XRE_RunAppShell()|hg:hg.mozilla.org/mozilla-central:toolkit/xre/nsEmbedFunctions.cpp:778b6b11194c072d2603e58118aaf6959e98902f|934|0x11
0|62|libxul.so|mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:778b6b11194c072d2603e58118aaf6959e98902f|238|0x5
0|63|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:778b6b11194c072d2603e58118aaf6959e98902f|315|0x17
0|64|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:778b6b11194c072d2603e58118aaf6959e98902f|290|0x8
0|65|libxul.so|XRE_InitChildProcess(int, char**, XREChildData const*)|hg:hg.mozilla.org/mozilla-central:toolkit/xre/nsEmbedFunctions.cpp:778b6b11194c072d2603e58118aaf6959e98902f|769|0x8
0|66|firefox-bin|content_process_main(mozilla::Bootstrap*, int, char**)|hg:hg.mozilla.org/mozilla-central:ipc/contentproc/plugin-container.cpp:778b6b11194c072d2603e58118aaf6959e98902f|56|0x14
0|67|firefox-bin|main|hg:hg.mozilla.org/mozilla-central:browser/app/nsBrowserApp.cpp:778b6b11194c072d2603e58118aaf6959e98902f|272|0x12
0|68|libc-2.27.so||||0x21b97
0|69|firefox-bin|MOZ_ReportCrash|hg:hg.mozilla.org/mozilla-central:mfbt/Assertions.h:778b6b11194c072d2603e58118aaf6959e98902f|203|0x5

It seems it could be a crash near null looking at the registers, can you confirm?

What is the full output, if possible? MOZ_ReportCrash should give a more informative message...

But I have no idea how this could happen honestly, looking at the code (short of a miscompilation of course). There's nothing funky going on at the loop.

(ni? for the question above)

Flags: needinfo?(twsmith)
Attached file log_stderr.txt

I wonder if it is a miscompile these builds both use GCC[1] and the ASan builds use Clang.

[1] https://searchfox.org/mozilla-central/source/taskcluster/ci/build/linux.yml#91

Flags: needinfo?(twsmith)

Hmmm.... could you give me a link to a build known to have triggered this?

Flags: needinfo?(twsmith)

Alright, will take a look.

Flags: needinfo?(emilio)

Marking as P4 for now, given this doesn't appear to affect the builds we actually ship; but if it turns up more widely, that could change.

Priority: -- → P4

The code generated for the rust function is the same in GCC and Clang builds (as expected, as it is rustc generating it):

(gdb) disas Servo_IsUnknownPropertyRecordedInUseCounter 
Dump of assembler code for function Servo_IsUnknownPropertyRecordedInUseCounter:
   0x000000000564e3d0 <+0>:	push   %rbp
   0x000000000564e3d1 <+1>:	mov    %rsp,%rbp
   0x000000000564e3d4 <+4>:	mov    %esi,%eax
   0x000000000564e3d6 <+6>:	shr    $0x3,%eax
   0x000000000564e3d9 <+9>:	and    $0xfffffff8,%eax
   0x000000000564e3dc <+12>:	mov    0x48(%rdi,%rax,1),%rax
   0x000000000564e3e1 <+17>:	bt     %rsi,%rax
   0x000000000564e3e5 <+21>:	setb   %al
   0x000000000564e3e8 <+24>:	pop    %rbp
   0x000000000564e3e9 <+25>:	retq   
End of assembler dump.

So it might be an issue with the caller. I tried to disassemble that but I didn't manage to get useful addresses and I'm a bit confused about the crashreporter-symbols* stuff.

David, do you know how can I disassemble Document::SetCssUseCounterBits from a build from try? I've tried to load the symbols with add-symbol-file path/to/symbols/libxul.so/<something>/libxul.so.dbg (after unzipping the crashreporter-symbols-full and gunzipping the .dbg file), but that just shows:

(gdb) disas mozilla::dom::Document::SetCssUseCounterBits
Dump of assembler code for function mozilla::dom::Document::SetCssUseCounterBits():
   0x0000000002970c2e <+0>:	test   %al,%al
   0x0000000002970c30 <+2>:	jne    0x2970c40 <mozilla::dom::Document::SetCssUseCounterBits()+18>
   0x0000000002970c32 <+4>:	mov    %rbx,%rdi
   0x0000000002970c35 <+7>:	add    $0x8,%rsp
   0x0000000002970c39 <+11>:	pop    %rbx
   0x0000000002970c3a <+12>:	pop    %rbp
   0x0000000002970c3b <+13>:	jmpq   0x2970c7a <_ZN7mozilla17LinkedListElementINS_15SegmentedVectorI6RefPtrINS_29WebGLExtensionInstancedArraysEELm4096ENS_17MallocAllocPolicyEE11SegmentImplILm509EEEED2Ev>
   0x0000000002970c40 <+18>:	lea    0x438ddf3(%rip),%rdi        # 0x6cfea3a <style::properties::<impl style::gecko_properties::ComputedValues>::differing_properties+13818>
   0x0000000002970c47 <+25>:	lea    0x4378095(%rip),%rsi        # 0x6ce8ce3 <<style::stylesheets::supports_rule::SupportsCondition as style_traits::values::ToCss>::to_css+643>
   0x0000000002970c4e <+32>:	mov    $0x1b1,%edx
   0x0000000002970c53 <+37>:	callq  0x28c6af5 <_ZL26MOZ_ReportAssertionFailurePKcS0_i>
   0x0000000002970c58 <+42>:	lea    0x438de76(%rip),%rax        # 0x6cfead5 <style::properties::<impl style::gecko_properties::ComputedValues>::differing_properties+13973>
   0x0000000002970c5f <+49>:	mov    0x70fb062(%rip),%rcx        # 0x9a6bcc8
   0x0000000002970c66 <+56>:	mov    %rax,(%rcx)
   0x0000000002970c69 <+59>:	movl   $0x1b1,0x0
   0x0000000002970c74 <+70>:	callq  0xe102a0 <abort@plt>
   0x0000000002970c79 <+75>:	nop
   0x0000000002970c7a <+76>:	push   %rbp
   0x0000000002970c7b <+77>:	mov    %rsp,%rbp
   0x0000000002970c7e <+80>:	push   %rbx
   0x0000000002970c7f <+81>:	push   %rax
   0x0000000002970c80 <+82>:	cmpb   $0x0,0x10(%rdi)
   0x0000000002970c84 <+86>:	jne    0x2970ca0 <mozilla::dom::Document::SetCssUseCounterBits()+114>
   0x0000000002970c86 <+88>:	mov    %rdi,%rbx
   0x0000000002970c89 <+91>:	callq  0x297090e <_ZNK7mozilla17LinkedListElementINS_15SegmentedVectorI6RefPtrINS_29WebGLExtensionInstancedArraysEELm4096ENS_17MallocAllocPolicyEE11SegmentImplILm509EEEE8isInListEv>
   0x0000000002970c8e <+96>:	test   %al,%al
   0x0000000002970c90 <+98>:	je     0x2970ca0 <mozilla::dom::Document::SetCssUseCounterBits()+114>
   0x0000000002970c92 <+100>:	mov    %rbx,%rdi
   0x0000000002970c95 <+103>:	add    $0x8,%rsp
   0x0000000002970c99 <+107>:	pop    %rbx
   0x0000000002970c9a <+108>:	pop    %rbp
   0x0000000002970c9b <+109>:	jmpq   0x2970b6e <_ZN7mozilla17LinkedListElementINS_15SegmentedVectorI6RefPtrINS_29WebGLExtensionInstancedArraysEELm4096ENS_17MallocAllocPolicyEE11SegmentImplILm509EEEE6removeEv>
   0x0000000002970ca0 <+114>:	add    $0x8,%rsp
   0x0000000002970ca4 <+118>:	pop    %rbx
   0x0000000002970ca5 <+119>:	pop    %rbp
   0x0000000002970ca6 <+120>:	retq   
   0x0000000002970ca7 <+121>:	push   %rbp
   0x0000000002970ca8 <+122>:	mov    %rsp,%rbp
   0x0000000002970cab <+125>:	push   %r15
   0x0000000002970cad <+127>:	push   %r14
   0x0000000002970caf <+129>:	push   %r13
   0x0000000002970cb1 <+131>:	push   %r12
   0x0000000002970cb3 <+133>:	push   %rbx
   0x0000000002970cb4 <+134>:	sub    $0x38,%rsp
   0x0000000002970cb8 <+138>:	mov    %rcx,%r14
   0x0000000002970cbb <+141>:	mov    %rdx,%r15
   0x0000000002970cbe <+144>:	mov    %rdi,%r12
   0x0000000002970cc1 <+147>:	mov    %fs:0x28,%rax
   0x0000000002970cca <+156>:	mov    %rax,-0x30(%rbp)
   0x0000000002970cce <+160>:	movabs $0xaaaaaaaaaaaaaaaa,%rax
   0x0000000002970cd8 <+170>:	lea    -0x48(%rbp),%rcx
--Type <RET> for more, q to quit, c to continue without paging--
   0x0000000002970cdc <+174>:	mov    %rax,(%rcx)
   0x0000000002970cdf <+177>:	movb   $0x0,(%rcx)
   0x0000000002970ce2 <+180>:	mov    %rcx,-0x38(%rbp)
   0x0000000002970ce6 <+184>:	mov    0x88(%rdi),%rdi
   0x0000000002970ced <+191>:	mov    %rdi,0x8(%rcx)
   0x0000000002970cf1 <+195>:	test   %rdi,%rdi
   0x0000000002970cf4 <+198>:	je     0x2970d19 <mozilla::dom::Document::SetCssUseCounterBits()+235>
   0x0000000002970cf6 <+200>:	lea    0x50f6ad1(%rip),%rsi        # 0x7a677ce
   0x0000000002970cfd <+207>:	lea    0x523a59a(%rip),%rdx        # 0x7bab29e
   0x0000000002970d04 <+214>:	lea    -0x48(%rbp),%rcx
   0x0000000002970d08 <+218>:	mov    $0x18,%r8d
   0x0000000002970d0e <+224>:	mov    $0x90,%r9d
   0x0000000002970d14 <+230>:	callq  0xe768a6 <_ZN14ProfilingStack14pushLabelFrameEPKcS1_PvN2JS21ProfilingCategoryPairEj>
   0x0000000002970d19 <+235>:	lea    -0x38(%rbp),%rdi
   0x0000000002970d1d <+239>:	callq  0xe17c2e <_ZN7mozilla6detail19GuardObjectNotifierD2Ev>
   0x0000000002970d22 <+244>:	mov    0x8(%r14),%ecx
   0x0000000002970d26 <+248>:	cmp    $0x3,%ecx
   0x0000000002970d29 <+251>:	jbe    0x2970e29 <mozilla::dom::Document::InlineScriptAllowedByCSP()+229>
   0x0000000002970d2f <+257>:	lea    -0x4c(%rbp),%rbx
   0x0000000002970d33 <+261>:	movl   $0xaaaaaaaa,(%rbx)
   0x0000000002970d39 <+267>:	xor    %r13d,%r13d
   0x0000000002970d3c <+270>:	mov    %r14,%rdi
   0x0000000002970d3f <+273>:	xor    %esi,%esi
   0x0000000002970d41 <+275>:	callq  0x21e95d6 <_ZNK2JS6detail12CallArgsBaseINS0_10NoUsedRvalEEixEj

Which I don't think is the correct disassembly for the function.

Flags: needinfo?(emilio) → needinfo?(dmajor)

I only speak WinDbg... but, fun tip, WinDbgX can open ELF files nowadays! (When all you have is a hammer...)

0:000> uf libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv
libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv:
00000000`01ebf082 55              push    rbp
00000000`01ebf083 4889e5          mov     rbp,rsp
00000000`01ebf086 4155            push    r13
00000000`01ebf088 4154            push    r12
00000000`01ebf08a 53              push    rbx
00000000`01ebf08b 4883ec08        sub     rsp,8
00000000`01ebf08f 4c8baf78060000  mov     r13,qword ptr [rdi+678h]
00000000`01ebf096 4d85ed          test    r13,r13
00000000`01ebf099 0f84a7000000    je      libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xc4 (00000000`01ebf146)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x1d:
00000000`01ebf09f 4989fc          mov     r12,rdi
00000000`01ebf0a2 e833c518ff      call    libxul!NS_IsMainThread (00000000`0104b5da)
00000000`01ebf0a7 84c0            test    al,al
00000000`01ebf0a9 7536            jne     libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x5f (00000000`01ebf0e1)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x29:
00000000`01ebf0ab 488d35d66b2c04  lea     rsi,[libxul!ZL20kINIParserFactoryCID+0x7f68 (00000000`06185c88)]
00000000`01ebf0b2 488d3dd8800105  lea     rdi,[libxul!ZZN22nsXMLContentSerializer12IsJavaScriptEP10nsIContentP6nsAtomiRK12nsTSubstringIDsEE11kJavaScript+0x2431 (00000000`06ed7191)]
00000000`01ebf0b9 ba31020000      mov     edx,231h
00000000`01ebf0be e8d71b06ff      call    libxul!MOZ_ReportAssertionFailure (00000000`00f20c9a)
00000000`01ebf0c3 488b05966bec06  mov     rax,qword ptr [libxul!+0x47a8 (00000000`08d85c60)]
00000000`01ebf0ca 488d0d54810105  lea     rcx,[libxul!ZZN22nsXMLContentSerializer12IsJavaScriptEP10nsIContentP6nsAtomiRK12nsTSubstringIDsEE11kJavaScript+0x24c5 (00000000`06ed7225)]
00000000`01ebf0d1 488908          mov     qword ptr [rax],rcx
00000000`01ebf0d4 c704250000000000000000 mov dword ptr [0],0
00000000`01ebf0df 0f0b            ud2

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x5f:
00000000`01ebf0e1 803d285ded0600  cmp     byte ptr [libxul!ZN7mozilla11StaticPrefs39sMirror_layout_css_use_counters_enabledE (00000000`08d94e10)],0
00000000`01ebf0e8 742a            je      libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x92 (00000000`01ebf114)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x68:
00000000`01ebf0ea 31db            xor     ebx,ebx

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x6a:
00000000`01ebf0ec 89de            mov     esi,ebx
00000000`01ebf0ee 4c89ef          mov     rdi,r13
00000000`01ebf0f1 e86af28803      call    libxul!Servo_IsPropertyIdRecordedInUseCounter (00000000`0574e360)
00000000`01ebf0f6 84c0            test    al,al
00000000`01ebf0f8 740e            je      libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x86 (00000000`01ebf108)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x78:
00000000`01ebf0fa 8db350010000    lea     esi,[rbx+150h]
00000000`01ebf100 4c89e7          mov     rdi,r12
00000000`01ebf103 e8f2c8ffff      call    libxul!ZN7mozilla3dom8Document13SetUseCounterENS_10UseCounterE (00000000`01ebb9fa)

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x86:
00000000`01ebf108 48ffc3          inc     rbx
00000000`01ebf10b 4881fb2e020000  cmp     rbx,22Eh
00000000`01ebf112 75d8            jne     libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x6a (00000000`01ebf0ec)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x92:
00000000`01ebf114 8b05f25ced06    mov     eax,dword ptr [libxul!ZN7mozilla11StaticPrefs53sMirror_layout_css_use_counters_unimplemented_enabledE (00000000`08d94e0c)]
00000000`01ebf11a 85c0            test    eax,eax
00000000`01ebf11c 7428            je      libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xc4 (00000000`01ebf146)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x9c:
00000000`01ebf11e 31db            xor     ebx,ebx

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x9e:
00000000`01ebf120 4088de          mov     sil,bl
00000000`01ebf123 4c89ef          mov     rdi,r13
00000000`01ebf126 e8a5f28803      call    libxul!Servo_IsUnknownPropertyRecordedInUseCounter (00000000`0574e3d0)
00000000`01ebf12b 84c0            test    al,al
00000000`01ebf12d 740e            je      libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xbb (00000000`01ebf13d)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xad:
00000000`01ebf12f 8db37e030000    lea     esi,[rbx+37Eh]
00000000`01ebf135 4c89e7          mov     rdi,r12
00000000`01ebf138 e8bdc8ffff      call    libxul!ZN7mozilla3dom8Document13SetUseCounterENS_10UseCounterE (00000000`01ebb9fa)

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xbb:
00000000`01ebf13d 48ffc3          inc     rbx
00000000`01ebf140 4883fb70        cmp     rbx,70h
00000000`01ebf144 75da            jne     libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x9e (00000000`01ebf120)  Branch

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0xc4:
00000000`01ebf146 58              pop     rax
00000000`01ebf147 5b              pop     rbx
00000000`01ebf148 415c            pop     r12
00000000`01ebf14a 415d            pop     r13
00000000`01ebf14c 5d              pop     rbp
00000000`01ebf14d c3              ret
Flags: needinfo?(dmajor)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)

There's nothing funky going on at the loop.

My reading of it is that the Document has a bogus mStyleUseCounters.

(In reply to :dmajor from comment #10)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)
My reading of it is that the Document has a bogus mStyleUseCounters.

Hmm, that's odd, because it's a UniquePtr owned by the document that never changes... https://searchfox.org/mozilla-central/rev/efdf9bb55789ea782ae3a431bda6be74a87b041e/dom/base/Document.cpp#1368

We've seen some oddness on other GCC builds and functions returning this POD-but-generic generic struct... https://searchfox.org/mozilla-central/source/__GENERATED__/layout/style/ServoStyleConsts.h#928 but this should be crashing much more often in that case...

So something must be messing up with that pointer or something...

(In reply to :dmajor from comment #10)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)

There's nothing funky going on at the loop.

My reading of it is that the Document has a bogus mStyleUseCounters.

Hm or maybe not. Do we know that an id of 0x300 is valid?

Do we know that an id of 0x300 is valid?

Divide by eight, that's 0x60, but the use counters object is only 0x58 bytes long.

0x60 is a pretty perfectly acceptable ID, afaict. There are more than a hundred counted unknown properties.

That object is a bitfield effectively, see https://searchfox.org/mozilla-central/rev/efdf9bb55789ea782ae3a431bda6be74a87b041e/servo/components/style/use_counters/mod.rs#56

Right, so the ID is 0x300, the 0x60 is the byte offset for the 0x300th bit (the result of the shift right by three).

Hmmm... It seems rust should've panicked in that case, right?

If I understand correctly, the total counters are 0x58 bytes, of which the first 0x48 are non_custom_properties, leaving 0x10 == 16 bytes == 128 bits for unknown properties. That seems OK by https://searchfox.org/mozilla-central/source/__GENERATED__/layout/style/CountedUnknownProperties.h#114.

So how on earth did the loop count up to 0x300? :-o

So how on earth did the loop count up to 0x300? :-o

Here's how:

libxul!ZN7mozilla3dom8Document20SetCssUseCounterBitsEv+0x9e:
00000000`01ebf120 4088de mov sil,bl

The caller only sets the low byte of the parameter, it was actually 0x00, and the 0x3__ was previous garbage in the register, but the callee interpreted it as a full 32-bit value:

0x000000000564e3d4 <+4>: mov %esi,%eax

It is not clear to me who is in the wrong, caller or callee.

Hmm, so the rust definition of the enum has repr(u8), which should make the C++ definition just fine.

How is the ABI for enum classes in extern functions defined?

Seems like this could explain the heisenbugs we've seen with gcc builds such as bug 1581121 and co...

How is the ABI for enum classes in extern functions defined?

Good question, I'm not sure! (I'm not even sure who to ask, a Rust person or a C++ person...)

Ok, I can try to do some digging tomorrow. You already saved me a whole lot of time, thank you so much!

Flags: needinfo?(emilio)

I would assume the passing of the enum class would be the same as if you passed an integer of the same size and signedness of the underlying integer for the enum class. This is kind of a mess, see the (almost four years old!) thread for trying to figure out who should be performing extension for < 32-bit arguments at:

https://groups.google.com/forum/?hl=en#!topic/x86-64-abi/E8O33onbnGQ

and the last comment on the GCC bug at:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46942#c10

I think llvm will always pass the argument extended to 32 bits, per:

https://reviews.llvm.org/rL34623

Though the code on trunk is radically different:

https://raw.githubusercontent.com/llvm/llvm-project/master/llvm/lib/Target/X86/X86ISelLowering.cpp

      // If this is an 8 or 16-bit value, it is really passed promoted to 32
      // bits.  Insert an assert[sz]ext to capture this, then truncate to the
      // right size.
      if (VA.getLocInfo() == CCValAssign::SExt)
        ArgValue = DAG.getNode(ISD::AssertSext, dl, RegVT, ArgValue,
                               DAG.getValueType(VA.getValVT()));
      else if (VA.getLocInfo() == CCValAssign::ZExt)
        ArgValue = DAG.getNode(ISD::AssertZext, dl, RegVT, ArgValue,
                               DAG.getValueType(VA.getValVT()));
      else if (VA.getLocInfo() == CCValAssign::BCvt)
        ArgValue = DAG.getBitcast(VA.getValVT(), ArgValue);

I don't speak enough LLVM backend to know whether that does the same thing as the patch (I assume that it does).

A stripped-down example:

#include <stdint.h>
#include <stdlib.h>

enum class CountedProperty : uint8_t {
  A, B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,
  Count,
};

extern "C" bool Servo_IsUnknownProperty(void*, CountedProperty);
extern void SetUseCounter(int);

void SetCssUseCounterBits(void* x)
{
  for (size_t i = 0; i < size_t(CountedProperty::Count); ++i) {
    auto id = CountedProperty(i);
    if (Servo_IsUnknownProperty(x, id)) {
      SetUseCounter(i + 5);
    }
  }
}

compiled with GCC and Clang locally appears to zero-extend id prior to calling, though only clang actually extends from an 8-bit value; GCC apparently knows the upper bits are already zero and 32-bit MOVs id into the right register.

So I think this is a bug somewhere in clang where the zero-extension on the caller side is getting dropped, but how I couldn't really say. :(

You mean a bug in gcc, right? We're only seeing this in gcc builds...

Flags: needinfo?(nfroyd)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #24)

You mean a bug in gcc, right? We're only seeing this in gcc builds...

The ccov builds are gcc, AIUI, but aren't the fuzzing builds clang? Tyson, what C/C++ compiler is being used here?

Flags: needinfo?(nfroyd) → needinfo?(twsmith)

See comment 3, they use gcc.

Flags: needinfo?(twsmith)
Attached file test-case

This seems to be fixed with newer versions of gcc.

With my system gcc (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)), if I disassemble Document::SetCssUseCounterBits, I get the following in the relevant section of the function (the call to Servo_IsUnknownProperty, when compiled wirh gcc t.cc -g -Os -c -o t1.o:

   0x000000000000006b <+107>:	mov    r13d,0x1
   0x0000000000000071 <+113>:	mov    esi,ebp
   0x0000000000000073 <+115>:	mov    rdi,r12
   0x0000000000000076 <+118>:	call   0x7b <Document::SetCssUseCounterBits()+123>
   0x000000000000007b <+123>:	test   al,al
   0x000000000000007d <+125>:	je     0x9a <Document::SetCssUseCounterBits()+154>

Note the mov being an esi (lower 32-bit) move, IIUC.

However from automation's gcc, which is gcc (GCC) 7.4.0, I get, with the same arguments:

   0x000000000000006e <+110>:	mov    r13d,0x1
   0x0000000000000074 <+116>:	mov    sil,bpl
   0x0000000000000077 <+119>:	mov    rdi,r12
   0x000000000000007a <+122>:	call   0x7f <Document::SetCssUseCounterBits()+127>
   0x000000000000007f <+127>:	test   al,al
   0x0000000000000081 <+129>:	je     0x9e <Document::SetCssUseCounterBits()+158>

Which has the problematic mov, IIUC.

What's the usual procedure for this?

Should I try to bisect gcc to make sure what fixed is an intentional change, and report a bug otherwise?

If so, any good tool to do that or should I just check out gcc and do manual builds?

Flags: needinfo?(nfroyd)
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(emilio)

You can probably do a rough bisection using https://gcc.godbolt.org and that's probably good enough to create a gcc bug report.

It seems like it's broken with gcc 7.5 and fixed with gcc 8.1: https://godbolt.org/z/4rQFAk

It also seems it only affects -Os, so that's nice, I guess, in the sense that it probably only affects debug builds.

Thanks jeff. I'll try to check gcc 8 manually tomorrow, and bisect to see if the fix was intentional or not.

Flags: needinfo?(emilio)

This was fixed by https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82260. It doesn't look like the fix was intentional.

I filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92821

We'll see if the GCC folks think this is a bug in GCC or LLVM, and if it's LLVM making too many assumptions about the argument being zero-extended I'll go file a bug there I guess :)

Flags: needinfo?(emilio)

Relevant LLVM bug, as the GCC developers seem to think it is an LLVM bug: https://bugs.llvm.org/show_bug.cgi?id=44228

CC'ing :jhorak (it seems :stransky's account has been disabling for bouncing email), as IIRC Fedora ships GCC-built Firefox, and this may be relevant to them.

I'm not at all sure I understand this adequately, but according to Figure 3.1 "Scalar Types" in https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf, enum types are represented as a signed fourbyte; it doesn't say anything about smaller explicitly-sized enums, so arguably even though this is a [repr(u8)] enum, it should still be a 32-bit value for ABI purposes. (This is in the Data Representation section, not Parameter Passing, but seems like it would apply to the computation of the argument values, prior to assigning them to registers or stack locations for passing.) That could make this a GCC bug after all, IIUC.

The LLVM issue also reproduces with uint8_t, so it's still an issue even without that.

Also I think the paragraph you quote does not apply to rust #[repr(u8)] enums or enum classes, otherwise sizeof(EnumClassUint8) should be 4, which would defeat the purpose of specifying the underlying type explicitly.

Apparently this is not an issue for regular uint8_ts because of C/C++'s integer promotion rules.

So comment 35 may seem a bit more appealing now. I mentioned that in the gcc bug, let's see what they think.

GCC8 happens not to generate the code that causes the crash, so do that for now
to unblock fuzzers from hitting this.

We still need to figure out what to do about the more general issue of course...

Assignee: nobody → emilio
Pushed by ealvarez@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ed92ceb9cb68
Make debug fuzzing builds use gcc 8. r=froydnj
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
Flags: needinfo?(mh+mozilla)

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression
Keywords: regression
Flags: needinfo?(nfroyd)
See Also: → 1726515
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: