Closed Bug 1648612 Opened 5 years ago Closed 5 years ago

Data race in mozilla::AudioCallbackDriver::StateCallback_s

Categories

(Core :: Audio/Video: cubeb, defect, P2)

Product:
Core
Component:
Audio/Video: cubeb
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
87 Branch
Tracking Flags:
Tracking Status
firefox-esr78 87+ fixed
firefox85 --- wontfix
firefox86 --- wontfix
firefox87 --- fixed

People

(Reporter: bwc, Assigned: kinetik)

References

Details

(Keywords: csectype-race, sec-moderate, Whiteboard: [post-critsmash-triage][adv-main87-][adv-esr78.9-])

Attachments

(1 file)

Bug 1648612 - Backport AudioIPC crash fix for ESR. r?chunmin
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr78+
Details | Review

Found this while trying to run the webrtc mochitests on TSan. Maybe related to bug 1599922, or maybe bug 1610263?

2:05.50 GECKO(2870758) ==================
2:05.50 GECKO(2870758) WARNING: ThreadSanitizer: data race (pid=2870876)
2:05.50 GECKO(2870758) Write of size 8 at 0x7b4800127d70 by thread T54:
2:05.50 GECKO(2870758) #0 free /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706:3 (firefox+0x9e898)
2:05.50 GECKO(2870758) #1 operator delete /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/cxxalloc.h:51:10 (libxul.so+0x93d00b1)
2:05.50 GECKO(2870758) #2 mozilla::AudioCallbackDriver::~AudioCallbackDriver() /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.cpp:537:45 (libxul.so+0x93d00b1)
2:05.50 GECKO(2870758) #3 mozilla::AudioCallbackDriver::Release() /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.h:560:3 (libxul.so+0x93e57cd)
2:05.50 GECKO(2870758) #4 Release /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/RefPtr.h:50:40 (libxul.so+0x93ce654)
2:05.50 GECKO(2870758) #5 Release /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/RefPtr.h:381:36 (libxul.so+0x93ce654)
2:05.50 GECKO(2870758) #6 assign_assuming_AddRef /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/RefPtr.h:69:7 (libxul.so+0x93ce654)
2:05.50 GECKO(2870758) #7 operator= /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/RefPtr.h:168:5 (libxul.so+0x93ce654)
2:05.50 GECKO(2870758) #8 mozilla::AsyncCubebTask::Run() /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.cpp:316:15 (libxul.so+0x93ce654)
2:05.50 GECKO(2870758) #9 nsThreadPool::Run() /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadPool.cpp:299:14 (libxul.so+0x5c5b01e)
2:05.50 GECKO(2870758) #10 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5c54050)
2:05.50 GECKO(2870758) #11 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:504:10 (libxul.so+0x5c58c05)
2:05.50 GECKO(2870758) #12 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:302:20 (libxul.so+0x6749f5e)
2:05.50 GECKO(2870758) #13 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:316:10 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #14 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #15 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:291:3 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #16 nsThread::ThreadFunc(void*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:447:10 (libxul.so+0x5c4fcc8)
2:05.50 GECKO(2870758) #17 _pt_root /home/bcampen/checkouts/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:201:5 (libnspr4.so+0x50be0)
2:05.50 GECKO(2870758) Previous atomic write of size 4 at 0x7b4800127d70 by thread T34:
2:05.50 GECKO(2870758) #0 __tsan_atomic32_exchange /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:588:3 (firefox+0xeb0b9)
2:05.50 GECKO(2870758) #1 exchange /usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/atomic:272:2 (libxul.so+0x93d041c)
2:05.50 GECKO(2870758) #2 exchange /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/Atomics.h:199:17 (libxul.so+0x93d041c)
2:05.50 GECKO(2870758) #3 exchange /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/Atomics.h:305:31 (libxul.so+0x93d041c)
2:05.50 GECKO(2870758) #4 StateCallback /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.cpp:1068:52 (libxul.so+0x93d041c)
2:05.50 GECKO(2870758) #5 mozilla::AudioCallbackDriver::StateCallback_s(cubeb_stream*, void*, cubeb_state) /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.cpp:838:11 (libxul.so+0x93d041c)
2:05.50 GECKO(2870758) #6 audioipc_client::run_in_callback::hdf52d8e263853b9c /home/bcampen/checkouts/mozilla-central/media/audioipc/client/src/lib.rs:54:12 (libxul.so+0xed0742d)
2:05.50 GECKO(2870758) Thread T54 'CubebOp~tion #1' (tid=2871387, running) created by main thread at:
2:05.50 GECKO(2870758) #0 pthread_create /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967:3 (firefox+0x9fb2b)
2:05.50 GECKO(2870758) #1 _PR_CreateThread /home/bcampen/checkouts/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:458:14 (libnspr4.so+0x47831)
2:05.50 GECKO(2870758) #2 PR_CreateThread /home/bcampen/checkouts/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:533:12 (libnspr4.so+0x3be67)
2:05.50 GECKO(2870758) #3 nsThread::Init(nsTSubstring<char> const&) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:659:8 (libxul.so+0x5c51596)
2:05.50 GECKO(2870758) #4 nsThreadManager::NewNamedThread(nsTSubstring<char> const&, unsigned int, nsIThread**) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadManager.cpp:623:12 (libxul.so+0x5c5831e)
2:05.50 GECKO(2870758) #5 NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, already_AddRefed<nsIRunnable>, unsigned int) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:160:57 (libxul.so+0x5c5e9b7)
2:05.50 GECKO(2870758) #6 NS_NewNamedThread /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:151:10 (libxul.so+0x5c5a57e)
2:05.50 GECKO(2870758) #7 nsThreadPool::PutEvent(already_AddRefed<nsIRunnable>, unsigned int) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadPool.cpp:115:17 (libxul.so+0x5c5a57e)
2:05.50 GECKO(2870758) #8 nsThreadPool::Dispatch(already_AddRefed<nsIRunnable>, unsigned int) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadPool.cpp:350:5 (libxul.so+0x5c5b902)
2:05.50 GECKO(2870758) #9 non-virtual thunk to nsThreadPool::Dispatch(already_AddRefed<nsIRunnable>, unsigned int) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadPool.cpp (libxul.so+0x5c5bdad)
2:05.50 GECKO(2870758) #10 mozilla::SharedThreadPool::Dispatch(already_AddRefed<nsIRunnable>, unsigned int) /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/SharedThreadPool.h:70:42 (libxul.so+0x5c3f3f4)
2:05.50 GECKO(2870758) #11 Dispatch /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/nsIEventTarget.h:38:14 (libxul.so+0x93d0b23)
2:05.50 GECKO(2870758) #12 Dispatch /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.h:803:42 (libxul.so+0x93d0b23)
2:05.50 GECKO(2870758) #13 mozilla::AudioCallbackDriver::Start() /home/bcampen/checkouts/mozilla-central/dom/media/GraphDriver.cpp:765:14 (libxul.so+0x93d0b23)
2:05.50 GECKO(2870758) #14 mozilla::MediaTrackGraphImpl::RunInStableState(bool) /home/bcampen/checkouts/mozilla-central/dom/media/MediaTrackGraph.cpp:1764:15 (libxul.so+0x953aa22)
2:05.50 GECKO(2870758) #15 mozilla::(anonymous namespace)::MediaTrackGraphStableStateRunnable::Run() /home/bcampen/checkouts/mozilla-central/dom/media/MediaTrackGraph.cpp:1647:15 (libxul.so+0x9548b5d)
2:05.50 GECKO(2870758) #16 mozilla::CycleCollectedJSContext::ProcessStableStateQueue() /home/bcampen/checkouts/mozilla-central/xpcom/base/CycleCollectedJSContext.cpp:400:12 (libxul.so+0x5b377f3)
2:05.50 GECKO(2870758) #17 mozilla::CycleCollectedJSContext::AfterProcessTask(unsigned int) /home/bcampen/checkouts/mozilla-central/xpcom/base/CycleCollectedJSContext.cpp:462:3 (libxul.so+0x5b39309)
2:05.50 GECKO(2870758) #18 XPCJSContext::AfterProcessTask(unsigned int) /home/bcampen/checkouts/mozilla-central/js/xpconnect/src/XPCJSContext.cpp:1363:28 (libxul.so+0x6e4e7ff)
2:05.50 GECKO(2870758) #19 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1269:24 (libxul.so+0x5c54423)
2:05.50 GECKO(2870758) #20 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:504:10 (libxul.so+0x5c58c05)
2:05.50 GECKO(2870758) #21 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:87:21 (libxul.so+0x674938a)
2:05.50 GECKO(2870758) #22 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:270:30 (libxul.so+0x6749e6b)
2:05.50 GECKO(2870758) #23 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:316:10 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #24 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #25 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:291:3 (libxul.so+0x667c32c)
2:05.50 GECKO(2870758) #26 nsBaseAppShell::Run() /home/bcampen/checkouts/mozilla-central/widget/nsBaseAppShell.cpp:137:27 (libxul.so+0xa16c955)
2:05.50 GECKO(2870758) #27 XRE_RunAppShell() /home/bcampen/checkouts/mozilla-central/toolkit/xre/nsEmbedFunctions.cpp:913:20 (libxul.so+0xc4ab699)
2:05.51 GECKO(2870758) #28 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:237:9 (libxul.so+0x6749e1d)
2:05.51 GECKO(2870758) #29 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:316:10 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #30 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #31 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:291:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #32 XRE_InitChildProcess(int, char**, XREChildData const*) /home/bcampen/checkouts/mozilla-central/toolkit/xre/nsEmbedFunctions.cpp:744:34 (libxul.so+0xc4aaf49)
2:05.51 GECKO(2870758) #33 mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) /home/bcampen/checkouts/mozilla-central/toolkit/xre/Bootstrap.cpp:67:12 (libxul.so+0xc4b3c17)
2:05.51 GECKO(2870758) #34 content_process_main /home/bcampen/checkouts/mozilla-central/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28 (firefox+0x11171a)
2:05.51 GECKO(2870758) #35 main /home/bcampen/checkouts/mozilla-central/browser/app/nsBrowserApp.cpp:303:18 (firefox+0x11171a)
2:05.51 GECKO(2870758) Thread T34 'AudioIPC0' (tid=2871101, running) created by main thread at:
2:05.51 GECKO(2870758) #0 pthread_create /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967:3 (firefox+0x9fb2b)
2:05.51 GECKO(2870758) #1 std::sys::unix::thread::Thread::new::h6d6372b6bc98aed3 /rustc/f3e1a954d2ead4e2fc197c7da7d71e6c61bad196/src/libstd/sys/unix/thread.rs:68:18 (libxul.so+0xf714acd)
2:05.51 GECKO(2870758) #2 mozilla::CubebUtils::InitPreferredSampleRate() /home/bcampen/checkouts/mozilla-central/dom/media/CubebUtils.cpp:337:20 (libxul.so+0x932f5cf)
2:05.51 GECKO(2870758) #3 mozilla::CubebUtils::PreferredSampleRate() /home/bcampen/checkouts/mozilla-central/dom/media/CubebUtils.cpp:357:8 (libxul.so+0x932f6c6)
2:05.51 GECKO(2870758) #4 mozilla::MediaTrackGraph::GetInstance(mozilla::MediaTrackGraph::GraphDriverType, nsPIDOMWindowInner*, int, void const*) /home/bcampen/checkouts/mozilla-central/dom/media/MediaTrackGraph.cpp:3094:35 (libxul.so+0x954234f)
2:05.51 GECKO(2870758) #5 mozilla::GetUserMediaStreamRunnable::Run() /home/bcampen/checkouts/mozilla-central/dom/media/MediaManager.cpp:1199:28 (libxul.so+0x950cc91)
2:05.51 GECKO(2870758) #6 nsThread::ProcessNextEvent(bool, bool*) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThread.cpp:1234:14 (libxul.so+0x5c54050)
2:05.51 GECKO(2870758) #7 NS_ProcessNextEvent(nsIThread*, bool) /home/bcampen/checkouts/mozilla-central/xpcom/threads/nsThreadUtils.cpp:504:10 (libxul.so+0x5c58c05)
2:05.51 GECKO(2870758) #8 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:87:21 (libxul.so+0x674938a)
2:05.51 GECKO(2870758) #9 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:270:30 (libxul.so+0x6749e6b)
2:05.51 GECKO(2870758) #10 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:316:10 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #11 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #12 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:291:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #13 nsBaseAppShell::Run() /home/bcampen/checkouts/mozilla-central/widget/nsBaseAppShell.cpp:137:27 (libxul.so+0xa16c955)
2:05.51 GECKO(2870758) #14 XRE_RunAppShell() /home/bcampen/checkouts/mozilla-central/toolkit/xre/nsEmbedFunctions.cpp:913:20 (libxul.so+0xc4ab699)
2:05.51 GECKO(2870758) #15 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /home/bcampen/checkouts/mozilla-central/ipc/glue/MessagePump.cpp:237:9 (libxul.so+0x6749e1d)
2:05.51 GECKO(2870758) #16 RunInternal /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:316:10 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #17 RunHandler /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:309:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #18 MessageLoop::Run() /home/bcampen/checkouts/mozilla-central/ipc/chromium/src/base/message_loop.cc:291:3 (libxul.so+0x667c32c)
2:05.51 GECKO(2870758) #19 XRE_InitChildProcess(int, char**, XREChildData const*) /home/bcampen/checkouts/mozilla-central/toolkit/xre/nsEmbedFunctions.cpp:744:34 (libxul.so+0xc4aaf49)
2:05.51 GECKO(2870758) #20 mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) /home/bcampen/checkouts/mozilla-central/toolkit/xre/Bootstrap.cpp:67:12 (libxul.so+0xc4b3c17)
2:05.51 GECKO(2870758) #21 content_process_main /home/bcampen/checkouts/mozilla-central/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28 (firefox+0x11171a)
2:05.51 GECKO(2870758) #22 main /home/bcampen/checkouts/mozilla-central/browser/app/nsBrowserApp.cpp:303:18 (firefox+0x11171a)
2:05.51 GECKO(2870758) SUMMARY: ThreadSanitizer: data race /home/bcampen/checkouts/mozilla-central/objdir-ff-tsan/dist/include/mozilla/cxxalloc.h:51:10 in operator delete
2:05.51 GECKO(2870758) ==================

Marking this a sec-bug because mozilla::AudioCallbackDriver::StateCallback_s seems to be racing with the d'tor somehow.

Group: core-security → media-core-security

What platform is that?

This is on linux.

mozilla::AsyncCubebTask::Run() is releasing mDriver on AsyncCubebOperation::SHUTDOWN while mAudioStreamState is accessed in `StateCallback().

Looks like the cubeb_stream_stop() call may be returning before the state callback is run. I assume that would not be expected.

Component: Audio/Video: MediaStreamGraph → Audio/Video: cubeb

cubeb_stream_destroy() would have also returned before AudioCallbackDriver destruction completes.

Assignee: nobody → kinetik
Status: NEW → ASSIGNED
See Also: → 1645822, 1636363
Severity: -- → S4
Priority: -- → P2

Fixed by bug 1690718.

Depends on: 1690718
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

Does this need backporting? Bug 1690718 grafts cleanly to Beta but would need rebasing for ESR78.

Group: media-core-security → core-security-release
status-firefox85: --- → wontfix
status-firefox86: --- → affected
status-firefox87: --- → fixed
status-firefox-esr78: --- → affected
Flags: needinfo?(kinetik)
Target Milestone: --- → 87 Branch
Flags: qe-verify-
Whiteboard: [post-critsmash-triage]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #6)

Does this need backporting? Bug 1690718 grafts cleanly to Beta but would need rebasing for ESR78.

This is worth fixing in ESR. Backported: https://phabricator.services.mozilla.com/D105162

I'll request beta uplift on the larger update in bug 1690718 also. Thanks!

Flags: needinfo?(kinetik)

The patch landed in nightly and beta is affected.
:kinetik, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(kinetik)

beta uplift was denied as it's RC week.

status-firefox86: affected → wontfix
Flags: needinfo?(kinetik)

Comment on attachment 9203185 [details]
Bug 1648612 - Backport AudioIPC crash fix for ESR. r?chunmin

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Addresses potential crashes during audio stream use if a race occurs during shutdown.
  • User impact if declined:
  • Fix Landed on Version: 87
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Simple, self-contained fix for the issue.
  • String or UUID changes made by this patch: none
Attachment #9203185 - Flags: approval-mozilla-esr78?

Advisory will be attached to 1690718

Whiteboard: [post-critsmash-triage] → [post-critsmash-triage][adv-main87-][adv-esr78.9-]

Comment on attachment 9203185 [details]
Bug 1648612 - Backport AudioIPC crash fix for ESR. r?chunmin

Approved for 78.9esr, thanks for the backport patch.

Attachment #9203185 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: