Closed Bug 929534 Opened 11 years ago Closed 11 years ago

MOZ_CRASH in signaling_unittest

Categories

(Core :: WebRTC: Signaling, defect)

x86
macOS
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla27
Tracking Status
firefox26 --- unaffected
firefox27 --- fixed
firefox-esr17 --- unaffected
firefox-esr24 --- unaffected
b2g18 --- unaffected
b2g-v1.2 --- unaffected

People

(Reporter: jib, Assigned: jib)

References

Details

(4 keywords, Whiteboard: [WebRTC])

Crash Data

Attachments

(3 files, 2 obsolete files)

Attached file output.txt
The signaling unittests are consistently failing on my MBP OSX 10.8.4 (and have been since at least late last week).

STR:
 1. ./obj-x86_64-apple-darwin12.2.1-debug/media/webrtc/signaling/test/signaling_unittests

...
caller: SDPSanityCheck flags for offer = 0x3 SHOULD_SEND_AUDIO SHOULD_RECV_AUDIO
Modified SDP 
    v=0
    o=Mozilla-SIPUA-27.0a1 22350 0 IN IP4 0.0.0.0
    s=SIP Call
    t=0 0
    a=ice-ufrag:54842d4a
    a=ice-pwd:130706f30b22478cf66a3d59800dd479
    a=fingerprint:sha-256 6C:C0:80:0C:0B:B8:9A:95:E8:C8:FE:79:E9:E3:29:FC:BC:81:2E:B5:9E:27:0D:E3:B1:0F:0C:60:EE:C9:9D:49
    m=audio 63453 RTP/SAVPF 109 0 8 101
    c=IN IP4 68.80.94.125
    a=rtpmap:109 opus/48000/2
    a=ptime:20
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:101 telephone-event/8000
    a=fmtp:101 0-15
    a=sendrecv
    a=setup:G4rb4g3V4lu3
    a=candidate:0 1 UDP 2130379007 192.168.1.103 63453 typ host
    a=candidate:1 1 UDP 1694236671 68.80.94.125 63453 typ srflx raddr 192.168.1.103 rport 63453
    a=candidate:0 2 UDP 2130379006 192.168.1.103 58096 typ host
    a=candidate:1 2 UDP 1694236670 68.80.94.125 58096 typ srflx raddr 192.168.1.103 rport 58096
    a=rtcp-mux

caller: Signaling State: SignalingHaveLocalOffer
caller: onSetLocalDescriptionSuccess
callee: OnAddStream called hints=0 thread=0x110b20d80
callee: Signaling State: SignalingHaveRemoteOffer
callee: onSetRemoteDescriptionSuccess
Hit MOZ_CRASH() at /Users/Jan/moz/mozilla-central/memory/mozjemalloc/jemalloc.c:1572

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
[Switching to process 62886 thread 0x3e03]
jemalloc_crash () at /Users/Jan/moz/mozilla-central/memory/mozjemalloc/jemalloc.c:1572
1572		MOZ_CRASH();
(gdb) bt
#0  jemalloc_crash () at memory/mozjemalloc/jemalloc.c:1572
#1  0x000000010fe074d1 in arena_run_reg_dalloc (run=0x12d4ff000, bin=0x110a001a0, ptr=0x12d4fffd0, size=16) at memory/mozjemalloc/jemalloc.c:3314
#2  0x000000010fe0367e in arena_dalloc_small (arena=0x110a00040, chunk=0x12d400000, ptr=0x12d4fffd0, mapelm=0x12d401818) at memory/mozjemalloc/jemalloc.c:4518
#3  0x000000010fdf8a06 in arena_dalloc (ptr=0x12d4fffd0, offset=1048528) at memory/mozjemalloc/jemalloc.c:4646
#4  0x000000010fdf85d7 in je_free (ptr=0x12d4fffd0) at memory/mozjemalloc/jemalloc.c:6556
#5  0x000000010fdfcc2f in ozone_free_definite_size (zone=0x10fe4c000, ptr=0x12d4fffd0, size=16) at memory/mozjemalloc/jemalloc.c:7051
#6  0x00007fff82a4c8f8 in free ()
#7  0x000000010f896545 in moz_free (ptr=0x12d4fffd0) at memory/mozalloc/mozalloc.cpp:48
#8  0x000000010014297b in fsmdef_free_constraints (constraints=0x12d4fffd0) at media/webrtc/signaling/src/sipcc/core/gsm/fsmdef.c:1113
#9  0x0000000100150bb8 in fsmdef_ev_createanswer (event=0x110cf6e18) at signaling/src/sipcc/core/gsm/fsmdef.c:3373
#10 0x00000001001817d9 in sm_process_event (tbl=0x1012eec18, event=0x110cf6e18) at media/webrtc/signaling/src/sipcc/core/gsm/sm.c:48
#11 0x00000001001382be in fim_process_event (data=0x1370c0000, cac_passed=0 '\0') at media/webrtc/signaling/src/sipcc/core/gsm/fim.c:651
#12 0x0000000100160510 in gsm_process_msg (cmd=158, msg=0x1370c0000) at media/webrtc/signaling/src/sipcc/core/gsm/gsm.c:133
#13 0x0000000100160a68 in GSMTask (arg=0x1117756c0) at media/webrtc/signaling/src/sipcc/core/gsm/gsm.c:321
#14 0x00007fff82a337a2 in _pthread_start ()
#15 0x00007fff82a201e1 in thread_start ()

Full output attached.
I see fsmdef_free_constraints is involved. Note that I changed how we store constraints in Bug 917328 patch 2.
Just got this to happen for me under the debugger finally.  I noticed the my constraints pointer had this memory under it:

constraints pointer should point to a list of booleans:

(gdb) x/50x 0x7ffef03fcd20
0x7ffef03fcd20: 0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a
0x7ffef03fcd28: 0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a
0x7ffef03fcd30: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
0x7ffef03fcd38: 0x60    0x2d    0x87    0x01    0x00    0x00    0x00    0x00
0x7ffef03fcd40: 0xff    0xff    0xff    0xff    0xff    0xff    0xff    0xff
0x7ffef03fcd48: 0x60    0x2d    0x87    0x01    0x00    0x00    0x00    0x00
EKR's buildbot shows a different error on the latest run:

PROCESS-CRASH | signaling_unittests | application crashed [@ sdp_verify_sdp_ptr]
Crash dump filename: /tmp/tmpdmhmuX/67790879-4b93-13b4-254f40cd-75de96f6.dmp
Operating system: Linux
                  0.0.0 Linux 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64
CPU: amd64
     family 6 model 45 stepping 7
     12 CPUs

Crash reason:  SIGSEGV
Crash address: 0x100000015

Thread 10 (crashed)
 0  signaling_unittests!sdp_verify_sdp_ptr [sdp_main.c:2fa3dc3a48c4 : 775 + 0x0]
    rbx = 0x0000000100000001   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x00000000004ddf8c
    rsp = 0x00002aecebdf54e0   rbp = 0x0000000000000005
    Found by: given as instruction pointer in context
 1  signaling_unittests!sdp_free_description [sdp_main.c:2fa3dc3a48c4 : 1229 + 0x4]
    rbx = 0x0000000100000001   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x00000000004de27d
    rsp = 0x00002aecebdf54f0   rbp = 0x0000000000000005
    Found by: call frame info
 2  signaling_unittests!sipsdp_free [ccsip_sdp.c:2fa3dc3a48c4 : 174 + 0x4]
    rbx = 0x00002aeceb8900a0   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x000000000051388e
    rsp = 0x00002aecebdf5520   rbp = 0x0000000000000005
    Found by: call frame info
 3  signaling_unittests!gsmsdp_free [gsm_sdp.c:2fa3dc3a48c4 : 7154 + 0xb]
    rbx = 0x00002aeceb890000   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x00000000004bab4f
    rsp = 0x00002aecebdf5540   rbp = 0x0000000000000005
    Found by: call frame info
 4  signaling_unittests!fsmdef_free_dcb [fsmdef.c:2fa3dc3a48c4 : 1375 + 0x7]
    rbx = 0x00002aeceb890000   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x00000000004a6556
    rsp = 0x00002aecebdf5550   rbp = 0x0000000000000005
    Found by: call frame info
 5  signaling_unittests!fsmdef_release [fsmdef.c:2fa3dc3a48c4 : 2067 + 0x7]
    rbx = 0x00002aeceb890000   r12 = 0x00002aeceb89e090
    r13 = 0x0000000000938f00   r14 = 0x00002aecebdf55f8
    r15 = 0x00000000004abee8   rip = 0x00000000004a7628
    rsp = 0x00002aecebdf5560   rbp = 0x0000000000000005
    Found by: call frame info
 6  signaling_unittests!fsmdef_ev_onhook [fsmdef.c:2fa3dc3a48c4 : 6724 + 0xc]
    rbx = 0x00002aeceb890000   r12 = 0x00002aecebdf5b70
    r13 = 0x0000000000938f54   r14 = 0x0000000000000004
    r15 = 0x00000000004abee8   rip = 0x00000000004abfdc
    rsp = 0x00002aecebdf58b0   rbp = 0x00002aeceb89e090
    Found by: call frame info
 7  signaling_unittests!sm_process_event [sm.c:2fa3dc3a48c4 : 48 + 0x5]
    rbx = 0x000000000000000b   r12 = 0x00002aecebdf5b70
    r13 = 0x0000000000938f54   r14 = 0x0000000000000004
    r15 = 0x00000000004abee8   rip = 0x00000000004c3f5f
    rsp = 0x00002aecebdf5a10   rbp = 0x0000000000000011
    Found by: call frame info
 8  signaling_unittests!fim_process_event [fim.c:2fa3dc3a48c4 : 651 + 0x14]
    rbx = 0x00002aeceb89e090   r12 = 0x0000000000000011
    r13 = 0x000000000000000b   r14 = 0x00002aeceb8a20a0
    r15 = 0x0000000000000001   rip = 0x000000000049d56d
    rsp = 0x00002aecebdf5ab0   rbp = 0x0000000000938f54
    Found by: call frame info
 9  signaling_unittests!gsm_process_msg [gsm.c:2fa3dc3a48c4 : 133 + 0x9]
    rbx = 0x000000000000009e   r12 = 0x000000000000000b
    r13 = 0x00002aecebdf69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00000000004b2e81
    rsp = 0x00002aecebdf5bd0   rbp = 0x00002aeceb8b43a0
    Found by: call frame info
10  signaling_unittests!GSMTask [gsm.c:2fa3dc3a48c4 : 321 + 0x7]
    rbx = 0x00002aeceb8b43a0   r12 = 0x00002aecebdb9780
    r13 = 0x00002aecebdf69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00000000004b324d
    rsp = 0x00002aecebdf5c00   rbp = 0x0000000000000000
    Found by: call frame info
11  libpthread-2.15.so + 0x7e99
    rbx = 0x0000000000000000   r12 = 0x00002aecebdb9780
    r13 = 0x00002aecebdf69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00002aecde948e9a
    rsp = 0x00002aecebdf5c30   rbp = 0x0000000000000000
    Found by: call frame info
Also from buildbot, also jemalloc, but different stack:

PROCESS-CRASH | signaling_unittests | application crashed [@ jemalloc_crash]
Crash dump filename: /tmp/tmpQzFSbp/3fe1dc3c-a291-fac6-5399d3d8-3a0fd6ed.dmp
Operating system: Linux
                  0.0.0 Linux 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64
CPU: amd64
     family 6 model 45 stepping 7
     12 CPUs

Crash reason:  SIGSEGV
Crash address: 0x0

Thread 10 (crashed)
 0  signaling_unittests!jemalloc_crash [jemalloc.c:909898f24e71 : 1572 + 0x0]
    rbx = 0x00002acee6530000   r12 = 0x00002aced9d951a8
    r13 = 0x00002aced9d95040   r14 = 0x00002aced9d95048
    r15 = 0x0000000000000fb0   rip = 0x000000000092bd91
    rsp = 0x00002acee73f54e0   rbp = 0x0000000000000010
    Found by: given as instruction pointer in context
 1  signaling_unittests!arena_dalloc [jemalloc.c:909898f24e71 : 4578 + 0x13]
    rbx = 0x00002acee6530000   r12 = 0x00002aced9d951a8
    r13 = 0x00002aced9d95040   r14 = 0x00002aced9d95048
    r15 = 0x0000000000000fb0   rip = 0x000000000092e45b
    rsp = 0x00002acee73f54f0   rbp = 0x0000000000000010
    Found by: call frame info
 2  signaling_unittests!gsmsdp_free [gsm_sdp.c:909898f24e71 : 7154 + 0xb]
    rbx = 0x00002acee6e90000   r12 = 0x00002acee6e9e090
    r13 = 0x0000000000938f00   r14 = 0x00002acee73f55f8
    r15 = 0x00000000004abe98   rip = 0x00000000004baaff
    rsp = 0x00002acee73f5540   rbp = 0x0000000000000005
    Found by: call frame info
 3  signaling_unittests!fsmdef_free_dcb [fsmdef.c:909898f24e71 : 1375 + 0x7]
    rbx = 0x00002acee6e90000   r12 = 0x00002acee6e9e090
    r13 = 0x0000000000938f00   r14 = 0x00002acee73f55f8
    r15 = 0x00000000004abe98   rip = 0x00000000004a6506
    rsp = 0x00002acee73f5550   rbp = 0x0000000000000005
    Found by: call frame info
 4  signaling_unittests!fsmdef_release [fsmdef.c:909898f24e71 : 2067 + 0x7]
    rbx = 0x00002acee6e90000   r12 = 0x00002acee6e9e090
    r13 = 0x0000000000938f00   r14 = 0x00002acee73f55f8
    r15 = 0x00000000004abe98   rip = 0x00000000004a75d8
    rsp = 0x00002acee73f5560   rbp = 0x0000000000000005
    Found by: call frame info
 5  signaling_unittests!fsmdef_ev_onhook [fsmdef.c:909898f24e71 : 6724 + 0xc]
    rbx = 0x00002acee6e90000   r12 = 0x00002acee73f5b70
    r13 = 0x0000000000938f94   r14 = 0x0000000000000004
    r15 = 0x00000000004abe98   rip = 0x00000000004abf8c
    rsp = 0x00002acee73f58b0   rbp = 0x00002acee6e9e090
    Found by: call frame info
 6  signaling_unittests!sm_process_event [sm.c:909898f24e71 : 48 + 0x5]
    rbx = 0x000000000000000b   r12 = 0x00002acee73f5b70
    r13 = 0x0000000000938f94   r14 = 0x0000000000000004
    r15 = 0x00000000004abe98   rip = 0x00000000004c3f0f
    rsp = 0x00002acee73f5a10   rbp = 0x0000000000000011
    Found by: call frame info
 7  signaling_unittests!fim_process_event [fim.c:909898f24e71 : 651 + 0x14]
    rbx = 0x00002acee6e9e090   r12 = 0x0000000000000011
    r13 = 0x000000000000000b   r14 = 0x00002acee6ea20a0
    r15 = 0x0000000000000001   rip = 0x000000000049d51d
    rsp = 0x00002acee73f5ab0   rbp = 0x0000000000938f94
    Found by: call frame info
 8  signaling_unittests!gsm_process_msg [gsm.c:909898f24e71 : 133 + 0x9]
    rbx = 0x000000000000009e   r12 = 0x000000000000000b
    r13 = 0x00002acee73f69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00000000004b2e31
    rsp = 0x00002acee73f5bd0   rbp = 0x00002acee6ebc480
    Found by: call frame info
 9  signaling_unittests!GSMTask [gsm.c:909898f24e71 : 321 + 0x7]
    rbx = 0x00002acee6ebc480   r12 = 0x00002acee73b9780
    r13 = 0x00002acee73f69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00000000004b31fd
    rsp = 0x00002acee73f5c00   rbp = 0x0000000000000000
    Found by: call frame info
10  libpthread-2.15.so + 0x7e99
    rbx = 0x0000000000000000   r12 = 0x00002acee73b9780
    r13 = 0x00002acee73f69c0   r14 = 0x0000000000000000
    r15 = 0x0000000000000003   rip = 0x00002aced9f9de9a
    rsp = 0x00002acee73f5c30   rbp = 0x0000000000000000
    Found by: call frame info
(In reply to Ethan Hugg [:ehugg] from comment #2)
> constraints pointer should point to a list of booleans:
> 
> (gdb) x/50x 0x7ffef03fcd20
> 0x7ffef03fcd20: 0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a
> 0x7ffef03fcd28: 0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a    0x5a

Looks like it got freed already?
Although the assert that fails is at the end of arena_run_reg_dalloc() so perhaps the same free.
Does look to be freed already.  Though be careful; anything that relies on mainthread semantics can get broken on unit tests
I think I've found the problem. The Part 2 patch in Bug 917328 changed how MediaConstraints are passed in CreateOffer/Answer. It used to be that .buildArray was called inside each CreateXxx, producing a new allocation of cc_constraints to be freed downstream (on the STS thread):

> // Used by unit tests and the IDL CreateOffer.
> NS_IMETHODIMP
>-PeerConnectionImpl::CreateOffer(MediaConstraints& constraints)
+PeerConnectionImpl::CreateOffer(const MediaConstraintsExternal& aConstraints)
> {
>   PC_AUTO_ENTER_API_CALL(true);
> 
>   Timecard *tc = mTimeCard;
>   mTimeCard = nullptr;
>   STAMP_TIMECARD(tc, "Create Offer");
> 
>-  cc_media_constraints_t* cc_constraints = nullptr;
>-  constraints.buildArray(&cc_constraints);
>-
>-  mCall->createOffer(cc_constraints, tc);
>+  NS_ENSURE_TRUE(aConstraints.mConstraints, NS_ERROR_UNEXPECTED);
>+  mInternal->mCall->createOffer(aConstraints.mConstraints, tc);
>   return NS_OK;
> }

But in the new code, one original cc_media_constraints_t is housed inside MediaStreamConstraints, which is what is passed in and what STS ends up freeing. This turns out to be fine in regular use where this came from JS, but in the signaling_unittest case, there are cases of re-use, like here:

TEST_F(SignalingTest, FullCallAudioOnly)
{
  sipcc::MediaConstraints constraints;
  OfferAnswer(constraints, constraints, OFFER_AUDIO | ANSWER_AUDIO,
              true, SHOULD_SENDRECV_AUDIO, SHOULD_SENDRECV_AUDIO);

Here, the same 'constraints' instance is used twice, once for Offer and again for Answer. It gets freed twice, once for each use, interestingly on the STS thread, which creates a race. The free code (at least in the debug build) munges the constraints which is why they sometimes come out as 0x5a and sometimes not depending on how fast STS is at freeing them. Why the double-free only asserts some of the time and not every time is a bit of a mystery to me, but I still think this is the problem.

I'll change it have a buildArray() step again.
Assignee: nobody → jib
> It gets freed twice, once for each use,

"It" meaning mConstraints here http://mxr.mozilla.org/mozilla-central/source/media/webrtc/signaling/test/signaling_unittests.cpp#83
By the way, I've seen this happen in the browser also, so it's not limited to unit tests. Jesup's minimal test case for the "plumbing a stream from one PC into another one" problem seems to hit this specific problem quite reliably. I've attached it for you to play around with.
Blocks: 784491
Attachment #820626 - Attachment is patch: false
Attachment #820626 - Attachment mime type: text/plain → text/html
(In reply to Adam Roach [:abr] from comment #10)
> By the way, I've seen this happen in the browser also, so it's not limited
> to unit tests.

Which symptoms do you see? Mine (comment 0) or ehugg's (comment 3 or comment 4)? I haven't been able to reproduce ehugg's.

> Jesup's minimal test case for the "plumbing a stream from one
> PC into another one" problem seems to hit this specific problem quite
> reliably. I've attached it for you to play around with.

I'm not able to get it to crash. Can you include the output and backtrace?
This fixes the originally reported problem for me as detailed in comment 8.

cc_media_constraints_t is yet again discretely cpr_allocated for each call that frees it.

Please re-test and file new bugs for any remaining issues, unless all or none are fixed.
Attachment #820811 - Flags: review?(rjesup)
Green try. Didn't break anything else at least - https://tbpl.mozilla.org/?tree=Try&rev=a7865e4c5a52
(In reply to Jan-Ivar Bruaroey [:jib] from comment #11)

> I'm not able to get it to crash. Can you include the output and backtrace?

Looking at the stack, the place I'm seeing this crash reliably is in addstream. You won't have this in your tree, since the patch to plumb constraints through for addstream hasn't landed yet. But it's the same root cause: the constraints are being passed in for the audio stream and for the video stream; and they're being freed for each. I'll pull your patch into my setup here and double-check that it prevents the crashing.
(In reply to Adam Roach [:abr] from comment #14)
> I'll pull your patch into my setup here and double-check that it prevents the crashing.

Confirmed: applying this patch and then copying the constraint building pattern into my local addStream fixes the crashes I was seeing.
Ah, good.
Comment on attachment 820811 [details] [diff] [review]
Brought back MediaConstraintsExternal.build() to handle re-use in signaling_unittest

Review of attachment 820811 [details] [diff] [review]:
-----------------------------------------------------------------

::: media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp
@@ +115,5 @@
>  
> +void ConvertConstraints(const MediaConstraintsInternal& aSrc,
> +                        MediaConstraintsExternal *aDst) {
> +  cc_media_constraints_t* c = &aDst->mConstraints;
> +  memset(c, 0, sizeof(*c));

Per conversation, this should be handled in constructors for MediaConstraintsExternal (one () that clears, one (cc_media_constraints *) that sets the initial value).

Also look at making this part of a copy constructor if there's no need for it elsewhere.

::: media/webrtc/signaling/test/signaling_unittests.cpp
@@ +79,5 @@
>  class MediaConstraints : public MediaConstraintsExternal {
>  public:
>    MediaConstraints()
> +  : MediaConstraintsExternal() {
> +    memset(&mConstraints, 0, sizeof(mConstraints));

See other comments
Attachment #820811 - Flags: review?(rjesup) → review-
Update addressing comments.
- Moved ConvertConstraints() to copy constructor.
Attachment #820811 - Attachment is obsolete: true
Attachment #821770 - Flags: review?(rjesup)
Attachment #821770 - Flags: review?(rjesup) → review+
Updated commit msg. No change. Carrying forward r+ from jesup
Attachment #821770 - Attachment is obsolete: true
Attachment #822362 - Flags: review+
https://hg.mozilla.org/mozilla-central/rev/dd7618acdabd
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Blocks: 917328
Group: core-security
Keywords: regression
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: