Closed Bug 809259 Opened 12 years ago Closed 12 years ago

Camera - crash when rapidly switching back and forth between video and picture modes

Categories

(Firefox OS Graveyard :: General, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(blocking-basecamp:+, firefox18 fixed, firefox19 fixed, firefox20 fixed)

VERIFIED FIXED
B2G C2 (20nov-10dec)
blocking-basecamp +
Tracking Status
firefox18 --- fixed
firefox19 --- fixed
firefox20 --- fixed

People

(Reporter: mikeh, Assigned: mikeh)

References

Details

(Keywords: otoro, unagi, Whiteboard: [caf:blocking][fix in bug 816817])

Attachments

(4 files, 3 obsolete files)

Observed on unagi. STR: 1. Open camera app 2. tap the video-mode button to switch to video mode 3. when button changes to picture-mode icon, tap button again 4. when button change to video-mode icon, back to step 2 logcat: E/memalloc( 741): /dev/pmem_adsp: failed to map pmem fd: Out of memory E/memalloc( 741): virtual int gralloc::PmemAshmemController::allocate(gralloc::alloc_data&, int, int): Failed to allocate ADSP/SMI memory E/msm7627a.gralloc( 741): gralloc failed err=Out of memory W/GraphicBufferAllocator( 741): alloc(480, 320, 17, 10100000, ...) failed -12 (Out of memory) I/PRLog ( 791): 26921848[44c15550]: [Child 791] ###!!! ABORT: unexpected type tag: '(mType) == (aType)', file ../../ipc/ipdl/_ipdlheaders/mozilla/layers/LayersSurfaces.h, line 618 I/Gecko ( 791): [Child 791] ###!!! ABORT: unexpected type tag: '(mType) == (aType)', file ../../ipc/ipdl/_ipdlheaders/mozilla/layers/LayersSurfaces.h, line 618 F/libc ( 791): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1) V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92171, type=3, code=57, value=0 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92181, type=3, code=48, value=42 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92184, type=3, code=50, value=4 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92186, type=3, code=53, value=271 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92189, type=3, code=54, value=463 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92191, type=3, code=58, value=42 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92194, type=0, code=2, value=0 V/EventHub( 741): /dev/input/event0 got: t0=831, t1=92198, type=0, code=0, value=0 I/DEBUG ( 112): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** I/DEBUG ( 112): Build fingerprint: 'toro/full_unagi/unagi:4.0.4.0.4.0.4/OPENMASTER/eng.mikeh.20121106.135318:eng/test-keys' I/DEBUG ( 112): pid: 791, tid: 808 >>> /system/b2g/plugin-container <<< I/DEBUG ( 112): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
Status: NEW → ASSIGNED
blocking-basecamp: --- → ?
Keywords: unagi
There are gaia and gecko parts to this bug, when I fix the gaia parts it 'should' be impossible to reproduce https://bugzilla.mozilla.org/show_bug.cgi?id=799586 but shall I leave this open for fixing the gecko side, gaia shouldnt be able to crash the gecko side even by being dumb
(In reply to Dale Harvey (:daleharvey) from comment #1) > > gaia shouldnt be able to crash the gecko side even by being dumb Agreed.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
It looks like one every cycle from still to video mode (or back), we allocate 9 buffers, but on the next switch, only free 8 of them. This happens right from the beginning, and once a buffer has been lost, it's gone. A symptom of this is the ever-increasing fd numbers throughout the log--initially in the 139..169 range, they are in the 283..370 range by the time the camera app aborts.
This is happening because every preview-start creates a new DOMCameraPreview object; when the old DOMCameraPreview is discarded, it holds onto the last graphic buffer that was passed to it[1], until that DOMCameraPreview is collected. Since collection is non-deterministic, we can run out of graphic buffers which causes the next preview-start to fail. This causes NS_ABORT_IF_FALSE() to trigger the SIGSEGV abort observed above. 1. See GonkNativeWindow::freeBufferLocked(), which does not call DeallocSurfaceDescriptorGralloc() on buffers in the BufferSlot::RENDERING state.
Nice diagnosis. We should fix that abort_if_false, and we should try to also release the buffer in the discarded DOMCameraPreview (if we can). Last but not least we should send a GC event if pmem is low. Cjones, do you agree?
The MediaStream class keeps an nsRefPtr<Image> (via VideoFrame) to the last played frame in 'mLastPlayedVideoFrame' to avoid redundant setting of the current video frame. See http://mxr.mozilla.org/mozilla-central/source/content/media/MediaStreamGraph.cpp#1279
This is a partial fix for the rapid mode switching problem; with this patch, we no longer run out of adsp_pmem. It does, however, seem to expose other stability problems.
Attachment #679387 - Flags: review?(roc)
Comment on attachment 679387 [details] [diff] [review] Part 1: MediaStreamGraphImpl no longer indefinitely keeps the last-played video frame Review of attachment 679387 [details] [diff] [review]: ----------------------------------------------------------------- You also need to ensure that, in PlayVideo, if mNotifiedFinished is set, then we don't read or write mLastPlayedVideoFrame. ::: content/media/MediaStreamGraph.cpp @@ +830,5 @@ > > if (stream->mFinished && !stream->mNotifiedFinished && > stream->mBufferStartTime + stream->GetBufferEnd() <= nextCurrentTime) { > stream->mNotifiedFinished = true; > + stream->mLastPlayedVideoFrame.SetNull(); = nullptr
New issue looks like this: E/MemoryHeapBase( 1436): mmap(fd=-1, size=155648) failed (Bad file number) F/libc ( 1436): Fatal signal 11 (SIGSEGV) at 0x5a5a5a5a (code=1) I/DEBUG ( 112): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** I/DEBUG ( 112): Build fingerprint: 'toro/full_unagi/unagi:4.0.4.0.4.0.4/OPENMASTER/eng.mikeh.20121106.135318:eng/test-keys' I/DEBUG ( 112): pid: 1436, tid: 1488 >>> /system/b2g/plugin-container <<< I/DEBUG ( 112): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 5a5a5a5a I/DEBUG ( 112): r0 5a5a5a5a r1 00000000 r2 00000000 r3 5a5a5a5a I/DEBUG ( 112): r4 5a5a5a5a r5 00000000 r6 00000000 r7 45781e48 I/DEBUG ( 112): r8 00000000 r9 430a2a40 10 430a2f58 fp 4379a4c8 I/DEBUG ( 112): ip 430a2b98 sp 45781e00 lr 42e5470b pc 42e545fe cpsr 60000030 I/DEBUG ( 112): d0 5a5a5a5a5a5a5a5a d1 5a5a5a5a5a5a5a5a I/DEBUG ( 112): d2 5a5a5a5a5a5a5a5a d3 5a5a5a5a5a5a5a5a I/DEBUG ( 112): d4 6d61434d4f443a3a d5 6976657250617265 I/DEBUG ( 112): d6 656365523a3a7765 d7 656d617246657669 I/DEBUG ( 112): d8 ffffff8200000000 d9 0000000000000000 I/DEBUG ( 112): d10 0000000000000000 d11 0000000000000000 I/DEBUG ( 112): d12 0000000000000000 d13 0000000000000000 I/DEBUG ( 112): d14 0000000000000000 d15 0000000000000000 I/DEBUG ( 112): d16 202c2a64696f7628 d17 3a616c6c697a6f6d I/DEBUG ( 112): d18 6f466567616d493a d19 6f76202c74616d72 I/DEBUG ( 112): d20 6d28292a28206469 d21 3a3a616c6c697a6f I/DEBUG ( 112): d22 3a3a73726579616c d23 202c2a6567616d49 I/DEBUG ( 112): d24 0000000000000000 d25 0000000000000000 I/DEBUG ( 112): d26 0000000000000000 d27 0000000000000000 I/DEBUG ( 112): d28 0000000000000000 d29 0000000000000000 I/DEBUG ( 112): d30 0000000000000000 d31 0000000000000000 I/DEBUG ( 112): scr 20000011 I/DEBUG ( 112): I/DEBUG ( 112): #00 pc 000005fe /system/lib/libgenlock.so I/DEBUG ( 112): #01 pc 00000706 /system/lib/libgenlock.so I/DEBUG ( 112): #02 pc 000007be /system/lib/libgenlock.so (genlock_unlock_buffer) I/DEBUG ( 112): #03 pc 00015ae4 /system/lib/hw/camera.msm7627a.so (_ZN7android22QualcommCameraHardware16runPreviewThreadEPv) I/DEBUG ( 112): #04 pc 00016b6c /system/lib/hw/camera.msm7627a.so (_ZN7android14preview_threadEPv) I/DEBUG ( 112): #05 pc 00012e14 /system/lib/libc.so (__thread_entry) I/DEBUG ( 112): #06 pc 00012968 /system/lib/libc.so (pthread_create) I/DEBUG ( 112): I/DEBUG ( 112): code around pc: I/DEBUG ( 112): 42e545dc e28fc600 e28cca00 e5bcfb2c e28f0004 ........,....... I/DEBUG ( 112): 42e545ec e5900000 eafffff9 42e55120 4603b510 ........ Q.B...F I/DEBUG ( 112): 42e545fc 6802b170 d10b2a0c 280c6880 6859d108 p..h.*...h.(..Yh I/DEBUG ( 112): 42e5460c d1052902 4a076958 d1014290 e0082000 .)..Xi.J.B... .. I/DEBUG ( 112): 42e5461c 20064905 44794a05 f7ff447a f06fefb6 .I. .JyDzD....o. I/DEBUG ( 112): I/DEBUG ( 112): code around lr: I/DEBUG ( 112): 42e546e8 000004a4 00000422 00000422 00000488 ...."..."....... I/DEBUG ( 112): 42e546f8 000003f6 b088b570 460d4604 f7ff4616 ....p....F.F.F.. I/DEBUG ( 112): 42e54708 b130ff77 4a224921 44794b22 447b447a w.0.!I"J"KyDzD{D I/DEBUG ( 112): 42e54718 69a3e00c 6300f413 6c20d132 da0a2800 ...i...c2. l.(.. I/DEBUG ( 112): 42e54728 4a1e491d 44794b1e 447b447a f7ff2006 .I.J.KyDzD{D. .. I/DEBUG ( 112): I/DEBUG ( 112): memory map around addr 5a5a5a5a: I/DEBUG ( 112): 47600000-47899000 /dev/pmem I/DEBUG ( 112): (no map for address) I/DEBUG ( 112): b0001000-b0009000 /system/bin/linker I/DEBUG ( 112): I/DEBUG ( 112): stack: I/DEBUG ( 112): 45781dc0 4223132c /system/lib/libbinder.so I/DEBUG ( 112): 45781dc4 422229cd /system/lib/libbinder.so I/DEBUG ( 112): 45781dc8 43164a00 I/DEBUG ( 112): 45781dcc 44a98040 I/DEBUG ( 112): 45781dd0 44a7c250 I/DEBUG ( 112): 45781dd4 44a7c250 I/DEBUG ( 112): 45781dd8 45781df0 I/DEBUG ( 112): 45781ddc 00000000 I/DEBUG ( 112): 45781de0 430a2a40 /system/lib/hw/camera.msm7627a.so I/DEBUG ( 112): 45781de4 430a2f58 /system/lib/hw/camera.msm7627a.so I/DEBUG ( 112): 45781de8 4379a4c8 I/DEBUG ( 112): 45781dec 4000fd4b /system/b2g/libmozglue.so I/DEBUG ( 112): 45781df0 44a7c260 I/DEBUG ( 112): 45781df4 4224bf1b /system/lib/libutils.so I/DEBUG ( 112): 45781df8 df0027ad I/DEBUG ( 112): 45781dfc 00000000 I/DEBUG ( 112): #00 45781e00 5a5a5a5a I/DEBUG ( 112): 45781e04 42e5470b /system/lib/libgenlock.so I/DEBUG ( 112): #01 45781e08 45781e10 I/DEBUG ( 112): 45781e0c 40009cad /system/b2g/libmozglue.so I/DEBUG ( 112): 45781e10 45781e38 I/DEBUG ( 112): 45781e14 4224be19 /system/lib/libutils.so I/DEBUG ( 112): 45781e18 44a98040 I/DEBUG ( 112): 45781e1c 44a7c250 I/DEBUG ( 112): 45781e20 4497c000 I/DEBUG ( 112): 45781e24 00000000 I/DEBUG ( 112): 45781e28 4497c000 I/DEBUG ( 112): 45781e2c 00000000 I/DEBUG ( 112): 45781e30 4497ca60 I/DEBUG ( 112): 45781e34 42e547c3 /system/lib/libgenlock.so
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #9) > Comment on attachment 679387 [details] [diff] [review] > Part 1: MediaStreamGraphImpl no longer indefinitely keeps the last-played > video frame > > Review of attachment 679387 [details] [diff] [review]: > ----------------------------------------------------------------- > > You also need to ensure that, in PlayVideo, if mNotifiedFinished is set, > then we don't read or write mLastPlayedVideoFrame. Okay. > ::: content/media/MediaStreamGraph.cpp > @@ +830,5 @@ > > > > if (stream->mFinished && !stream->mNotifiedFinished && > > stream->mBufferStartTime + stream->GetBufferEnd() <= nextCurrentTime) { > > stream->mNotifiedFinished = true; > > + stream->mLastPlayedVideoFrame.SetNull(); > > = nullptr I tried that first but it didn't work. g++ complained that operator= was not defined.
Investigation continues: D/memalloc( 109): /dev/pmem: Allocated buffer base:0x4a2a8000 size:32768 offset:2592768 fd:148 D/memalloc( 390): /dev/pmem: Mapped buffer base:0x45fca000 size:2625536 offset:2592768 fd:47 ... I/Gecko ( 390): mikeh: __get_memory:554 I/Gecko ( 390): mikeh: CameraHeapMemory:508 : fd=34 E/MemoryHeapBase( 390): mikeh: dup(fd=34)=-1 E/MemoryHeapBase( 390): mikeh: errno=9 (Bad file number) E/MemoryHeapBase( 390): mmap(fd=-1, size=155648) failed (Bad file number) V/ ( 390): __data_cb E/libgenlock( 390): invalid gralloc handle (at 0x0) E/libgenlock( 390): perform_lock_unlock_operation: handle is invalid E/QualcommCameraHardware( 390): runPreviewThread: genlock_unlock_buffer failed I/PRLog ( 390): 16577824[4490f270]: queueBuffer: E I/Gecko ( 390): getSlotFromBufferLocked: unknown buffer: 0x0 I/Gecko ( 390): queueBuffer: slot index out of range [0, 9]: -22 E/QualcommCameraHardware( 390): runPreviewThread: Failed while queueing buffer 34 for display. Error = -22 I/PRLog ( 390): 16577824[4490f270]: dequeueBuffer: E D/memalloc( 109): /dev/pmem_adsp: Allocated buffer base:0x448b0000 size:155648 fd:139 D/memalloc( 390): /dev/pmem_adsp: Mapped buffer base:0x42f9e000 size:155648, fd:34 ... I/Gecko ( 390): mikeh: __get_memory:554 I/Gecko ( 390): mikeh: CameraHeapMemory:508 : fd=47 E/MemoryHeapBase( 390): mikeh: dup(fd=47)=-1 E/MemoryHeapBase( 390): mikeh: errno=9 (Bad file number) E/MemoryHeapBase( 390): mmap(fd=-1, size=155648) failed (Bad file number) V/ ( 390): __data_cb F/libc ( 390): Fatal signal 11 (SIGSEGV) at 0x00000001 (code=1)
(In reply to Mike Habicher [:mikeh] from comment #11) > > ::: content/media/MediaStreamGraph.cpp > > @@ +830,5 @@ > > > > > > if (stream->mFinished && !stream->mNotifiedFinished && > > > stream->mBufferStartTime + stream->GetBufferEnd() <= nextCurrentTime) { > > > stream->mNotifiedFinished = true; > > > + stream->mLastPlayedVideoFrame.SetNull(); > > > > = nullptr > > I tried that first but it didn't work. g++ complained that operator= was > not defined. Right. Sorry.
Inder, I need some specialized help with this issue. Everyone sees it on unagi, and this morning I'm going to check whether or not it shows up on otoro. The problem arises when the uses taps quickly and repeatedly on the camera app's mode-switch button, cycling back and forth between picture and video mode. Eventually, the camera app crashes, and with this WIP patch (which includes a couple of other fixes), I've narrowed it down to the creation of a CameraHeapMemory object on an invalid fd. e.g. I( 1396:0x5b9) mikeh: __get_memory:569 : fd=97 I( 1396:0x5b9) mikeh: CameraHeapMemory:519 : fd=97 E( 1396:0x5b9) mmap(fd=-1, size=155648) failed (Bad file number) And while the return value from mmap() is checked against MAP_FAILED (i.e. -1), it is never used in MemoryHeapBase::mapfd() to set mBase = nullptr, so the sanity check of mGetMemory against nullptr in runPreviewThread() incorrectly passes. If I hack CameraHeapMemory::commonInitialization() to detect mHeap->base() == -1 and fudge handle.data to nullptr instead, the crash goes away: I( 1396:0x5b9) Fudging handle.data == -1 --> 0 E( 1396:0x5b9) void android::QualcommCameraHardware::runPreviewThread(void*): mGetMemory failed. E( 1396:0x5b9) invalid gralloc handle (at 0x446de7c0) E( 1396:0x5b9) perform_lock_unlock_operation: handle is invalid E( 1396:0x5b9) runPreviewThread: genlock_unlock_buffer failed I( 1396:0x5b9) 9389248[4480f9e0]: queueBuffer: E I( 1396:0x5b9) getSlotFromBufferLocked: unknown buffer: 0x446de7c0 I( 1396:0x5b9) queueBuffer: slot index out of range [0, 9]: -22 E( 1396:0x5b9) runPreviewThread: Failed while queueing buffer 97 for display. Error = -22 I( 1396:0x5b9) 9389248[4480f9e0]: dequeueBuffer: E I( 1396:0x5b9) 9389248[4480f9e0]: dequeueBuffer: returning slot=2 buf=4463d0b0 I( 1396:0x5b9) 9389248[4480f9e0]: dequeueBuffer: X I( 1396:0x5b9) 9389248[4480f9e0]: GonkNativeWindow::lockBuffer E( 1396:0x5b9) Could not find the Frame V( 1396:0x5b9) error Cancelling preview buffers I( 1396:0x5b9) 9389248[4480f9e0]: cancelBuffer: slot=2 ...but the camera eventually locks up.
Flags: needinfo?(ikumar)
Confirmed--problem occurs on Otoro as well. E/QualcommCamera( 514): Qint android::start_preview(camera_device*): E V/QualcommCameraHardware( 514): startPreview E V/QualcommCameraHardware( 514): startPreview : Starting normal preview I/QualcommCameraHardware( 514): getBuffersAndStartPreview : E V/QualcommCameraHardware( 514): getBuffersAndStartPreview: waiting for old frame thread to complete. E/mm-camera( 514): discard frame preview,discard_frame_cnt=0 E/MemoryHeapBase( 514): mikeh: dup(fd=72)=-1 E/MemoryHeapBase( 514): mikeh: errno=9 (Bad file number) E/MemoryHeapBase( 514): mmap(fd=-1, size=155648) failed (Bad file number) V/ ( 514): __data_cb F/libc ( 514): Fatal signal 11 (SIGSEGV) at 0x5a5a5a5a (code=1)
Keywords: otoro
FYI, Provided my comments over email to mikeh.
Flags: needinfo?(ikumar)
blocking-basecamp: ? → +
Comment on attachment 679387 [details] [diff] [review] Part 1: MediaStreamGraphImpl no longer indefinitely keeps the last-played video frame Moved this patch to a separate bug 813190.
Attachment #679387 - Attachment is obsolete: true
Attachment #679387 - Flags: review?(roc)
Depends on: 813190
Whiteboard: [caf:blocking]
Whats the status here?
Resolved :) gecko patch has landed and there was some gaia side code that landed a while ago to fix as well, just needs landed on aurora / beta as far as I can tell
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
(In reply to Dale Harvey (:daleharvey) from comment #21) > Resolved :) gecko patch has landed and there was some gaia side code that > landed a while ago to fix as well, just needs landed on aurora / beta as far > as I can tell O RLY? Link? :)
(In reply to Dale Harvey (:daleharvey) from comment #23) > for what, the gaia code? linked above > > https://bugzilla.mozilla.org/show_bug.cgi?id=799586 I thought this didn't completely fix the case where someone wailed on the mode-switch button as fast as they could?
Dale, MikeH, I see crash in two scenarios: 1) Even with the patch in bug 813190 I see a crash when I keep switching between camera and camcorder mode. 2) I also noticed that I see a crash easily if I record a video in camcorder mode and it tries to display the recorded video thumbnail on top of preview screen. Looking at the logcat this case looks very similar to bug 814063 though and might not be a camera issue. So, for issue #1 please reopen this bug.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
try-server push: https://tbpl.mozilla.org/?tree=Try&rev=4f6680041fee This fixes a number of small bugs that really only showed up when you very rapidly -POUND- on the mode-switch button in the camera: 1. Once upon a time, GetPreviewStreamImpl() created DOMCameraPreview itself and had to run on the main thread; that is no longer the case, but this function was still running on the main thread. Since it also calls StopPreviewInternal(), the multiple calls into the camera driver were making it go crazy. 2. StartPreviewImpl() assumed the preview was always stopped; now it force-stops itself 3. Finally, some drivers will _ignore_ stopPreview() requests if any drivers messages are enabled, so we turn those off. A non-stability fix in GetPreviewStreamImpl() stops the preview before changing its settings, a) to prevent any visual hiccups, and b) some drivers will ignore changes to some settings if the preview is enabled. With this patch, you can ///WAIL/// on the mode-switch button, and once the congestion clears, the camera will keep on running.
Attachment #679709 - Attachment is obsolete: true
Attachment #684541 - Flags: review?(jones.chris.g)
Attachment #684541 - Flags: review?(jones.chris.g) → review?(kchen)
Just to update. Phone - Unagi on 11/22/2012 build It will not only become blank screen. It will now crash. Just let you know.
Attachment #684541 - Flags: review?(kchen) → review+
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Whiteboard: [caf:blocking] → needs-checkin-aurora, [caf:blocking]
Sorry guys this is not fixed yet. Our test guys reported that they can still reproduce it and I confirmed that at the tip if we switch from camera to camcorder and v.v many times then we eventually get a black screen. In the logcat we get the error "can't start stream!": E/memalloc( 127): /dev/pmem_adsp: failed to map pmem fd: Out of memory E/memalloc( 127): virtual int gralloc::PmemAshmemController::allocate(gralloc::alloc_data&, int, int): Failed to allocate ADSP/SMI memory E/msm7627a.gralloc( 127): gralloc failed err=Out of memory W/GraphicBufferAllocator( 127): alloc(480, 320, 17, 10100000, ...) failed -12 (Out of memory) E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): dequeue buf hdl =0x417508f1, size =447818566, stride=1098195073 E/QCameraHWI_Preview( 484): getBufferFromSurface: camera call genlock_lock, hdl=0x417508f1 E/libgenlock( 484): perform_lock_unlock_operation: handle is invalid E/QCameraHWI_Preview( 484): getBufferFromSurface: genlock_lock_buffer(WRITE) failed E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): dequeue buf: 0x4175f70c E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): idx = 5, fd = -2001661848, size = 447818566, offset = 217600616 E/memalloc( 127): /dev/pmem_adsp: failed to map pmem fd: Out of memory E/memalloc( 127): virtual int gralloc::PmemAshmemController::allocate(gralloc::alloc_data&, int, int): Failed to allocate ADSP/SMI memory E/msm7627a.gralloc( 127): gralloc failed err=Out of memory W/GraphicBufferAllocator( 127): alloc(480, 320, 17, 10100000, ...) failed -12 (Out of memory) E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): dequeue buf hdl =0x417508f1, size =447818566, stride=1098195073 E/QCameraHWI_Preview( 484): getBufferFromSurface: camera call genlock_lock, hdl=0x417508f1 E/libgenlock( 484): perform_lock_unlock_operation: handle is invalid E/QCameraHWI_Preview( 484): getBufferFromSurface: genlock_lock_buffer(WRITE) failed E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): dequeue buf: 0x4175f70c E/QCameraHWI_Preview( 484): android::status_t android::QCameraStream_preview::getBufferFromSurface(): idx = 6, fd = -2001661848, size = 447818566, offset = 217600616 E/libgenlock( 484): perform_lock_unlock_operation: handle is invalid E/QCameraHWI_Preview( 484): getBufferFromSurface: genlock_unlock_buffer failed E/QCameraHWI_Preview( 484): virtual android::status_t android::QCameraStream_preview::initDisplayBuffers(): cannot get memory from surface texture client, ret = -22 E/QCameraHWI( 484): android::status_t android::QCameraHardwareInterface::startPreview2(): X error - can't start stream! E/QualcommCamera( 484): Qint android::start_preview(camera_device*): X
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Here is complete log. Seems like something is still leaking memory?
I'm able to reproduce a crash. Just for future reference, it's best to file a new bug unless you know that the landed patches were wrong and need to be backed out. That is, if they fixed a bug that had the same symptoms, then there are probably multiple defects at work.
I'm not sure if we're leaking or not, but here's the crash (gdb) bt #0 mozilla::layers::GrallocBufferActor::GetFrom (aDescriptor=...) at /home/cjones/mozilla/new-b2g/gecko/gfx/layers/ipc/ShadowLayerUtilsGralloc.cpp:282 #1 0x4080e7f2 in android::GonkNativeWindow::dequeueBuffer (this=0x453a5800, buffer=0x4442bd8c) at /home/cjones/mozilla/new-b2g/gecko/dom/camera/GonkNativeWindow.cpp:267 #2 0x4080e0f4 in android::GonkNativeWindow::hook_dequeueBuffer (window=0x4442bd5c, buffer=0x4442bd3c) at /home/cjones/mozilla/new-b2g/gecko/dom/camera/GonkNativeWindow.cpp:97 #3 0x4080d2a4 in android::CameraHardwareInterface::__dequeue_buffer (w=<value optimized out>, buffer=0x43e0aaf0, stride=0x0) at /home/cjones/mozilla/new-b2g/frameworks/base/services/camera/libcameraservice/CameraHardwareInterface.h:582 #4 0x425b8d20 in android::QualcommCameraHardware::getBuffersAndStartPreview() () from /home/cjones/mozilla/new-b2g/out/target/product/otoro/system/lib/hw/camera.msm7627a.so #5 0x425b8d20 in android::QualcommCameraHardware::getBuffersAndStartPreview() () from /home/cjones/mozilla/new-b2g/out/target/product/otoro/system/lib/hw/camera.msm7627a.so (gdb) f 1 #1 0x4080e7f2 in android::GonkNativeWindow::dequeueBuffer (this=0x453a5800, buffer=0x4442bd8c) at /home/cjones/mozilla/new-b2g/gecko/dom/camera/GonkNativeWindow.cpp:267 (gdb) p buffer $1 = { mValue = { VShmem = "T\275BD\001\000\000\000#\346ZB\000\320@F", VSurfaceDescriptorD3D10 = "T\275BD", VSurfaceDescriptorGralloc = "T\275BD\001\000\000\000#\346ZB", VSurfaceDescriptorX11 = "T", VSharedTextureDescriptor = "T\275BD\001\000\000\000#\346ZB\000\320@F5ހ@" }, mType = mozilla::layers::SurfaceDescriptor::T__None } We should never have a buffer of type T__None here, but if we're out of gralloc there's nothing we can do for the camera. We can fix this this crasher and see if camera eventually recovers; if so it's probably GC falling behind, and if not we're probably leaking.
Chris - Did that crash report get sent to socorro by any chance? A big question came up in the platform meeting that crash reports may not be getting sent, so I'm wondering if your crash report ended up on socorro.
With this patch, we no longer crash but the camera never unhorks itself. So I suspect we're leaking gralloc but haven't proven it yet.
Attachment #685963 - Flags: review?(kchen)
Attached file pmemmon, v2
Jetzt mit pmem_adsp Supermaechten. This confirms that we're indeed leaking pmem.
(In reply to Jason Smith [:jsmith] from comment #35) > Chris - Did that crash report get sent to socorro by any chance? A big > question came up in the platform meeting that crash reports may not be > getting sent, so I'm wondering if your crash report ended up on socorro. This is a local build, not an official one, so I hope not! ;) This is easy to reproduce, so please feel free to test with an official build.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #37) > Created attachment 685973 [details] > pmemmon, v2 > > Jetzt mit pmem_adsp Supermaechten. > > This confirms that we're indeed leaking pmem. OK, so we're actually *not* leaking pmem. If I follow these steps (with the above patch applied) (1) Switch camera/video until pmem is exhausted (preview will be blank when that happens) (2) Press HOME to background the camera app (3) Tap camera icon to foreground it Then pmemmon says the allocated pmem_adsp goes back down to 0, and indeed the preview stream starts back up.
Is that behavior sufficient for FC, and can we investigate holding on to the buffers too long in a CS followup?
Flags: needinfo?(mvines)
Attachment #685963 - Flags: review?(kchen) → review+
(In reply to Chris Jones [:cjones] [:warhammer] from comment #33) > I'm able to reproduce a crash. Just for future reference, it's best to file > a new bug unless you know that the landed patches were wrong and need to be > backed out. That is, if they fixed a bug that had the same symptoms, then > there are probably multiple defects at work. Thanks Chris. Will keep it in mind. I reopened thinking the fix was not enough and probably needed a rework. If the same symptoms are there then it's hard to know if the fix is not enough or there is another issue lurking.
Absolutely :). No worries at all.
Not 100% functional but makes the bug go away. This workaround will force video-stuff to be GC'd more frequently. That this works strongly suggests that we're holding on to some DOM object related to the preview streams too long.
Tree looking bad, may not be able to land and stick tonight. https://bugzilla.mozilla.org/attachment.cgi?id=685963
Keywords: checkin-needed
(In reply to Chris Jones [:cjones] [:warhammer] from comment #40) > Is that behavior sufficient for FC, and can we investigate holding on to the > buffers too long in a CS followup? I'm afraid not. The core issue still remains so our test guys will continue to fail that test case. They don't buy workarounds as the point of the exercise is to demonstrate the codebase has achieved the required level of quality that will set us up for a successful CS.
Flags: needinfo?(mvines)
(In reply to Michael Vines [:m1] from comment #45) > (In reply to Chris Jones [:cjones] [:warhammer] from comment #40) > > Is that behavior sufficient for FC, and can we investigate holding on to the > > buffers too long in a CS followup? > > I'm afraid not. The core issue still remains so our test guys will continue > to fail that test case. They don't buy workarounds as the point of the > exercise is to demonstrate the codebase has achieved the required level of > quality that will set us up for a successful CS. 10-4. The patch that just landed isn't a workaround, but rather fixes broken code that's leading to crashes. The underlying issue isn't addressed though. The later WIP "workaround" fix here is having gaia help gecko pick better times to GC, as GC heuristics are rather difficult for DOM objects like this. But yes, it's also not a complete fix.
To recap what's happening here - gaia switches back and forth between still-image preview and video preview mode - if that's done at a "normal" pace, we free up DSP pmem at a rate sufficient to keep everything working - if the switches are as fast as possible, we eventually run out of DSP pmem and preview stops working - we're not leaking anything, because the memory can be freed up (comment 39) and preview made to work again - if the camera app destroys and recreates its <video> after each switch, then we don't hit this error case (attachment 685989 [details] [diff] [review]) This suggests to me that something that participates in <video> CC may be holding on to mozSrcObject streams longer than expected. Destroying the <video> in between stream changes fixes that problem. It also suggests gaia isn't doing anything terribly wrong. Is there a way we can pinpoint either - what may not be participating in cycle collection and causing CC's to delay too long - what <video>-related code might be keeping mozSrcObject streams alive too long ? I'm going to proceed with fixing up the gaia "GC hint" patch as insurance.
Flags: needinfo?(cpearce)
(In reply to Chris Jones [:cjones] [:warhammer] from comment #36) > > With this patch, we no longer crash but the camera never unhorks itself. So > I suspect we're leaking gralloc but haven't proven it yet. Chris, Inder: how are you causing this crash? I have a yesteryday's-tip build running on unagi and can't seem to reproduce. Can you give me an estimate of how fast you're tapping on the mode-switch button? (I've tried ~1 to 5Hz--about as fast as I can tap--and the screen goes blank while I'm doing this, but when I stop, the preview eventually starts.)
> Chris, Inder: how are you causing this crash? I have a yesteryday's-tip > build running on unagi and can't seem to reproduce. Can you give me an > estimate of how fast you're tapping on the mode-switch button? (I've tried > ~1 to 5Hz--about as fast as I can tap--and the screen goes blank while I'm > doing this, but when I stop, the preview eventually starts.) Mike, I am getting an irrecoverable black screen after tapping the "switch mode button" at normal speed for more than ~35 times. This is on Otoro. It gets worse if I take 4-5 pictures first and then try switching. In this case I see a crash instead of black screen in ~15-20 tries.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Whups, forgot to [leave open].
Status: RESOLVED → REOPENED
Flags: needinfo?(cpearce)
Resolution: FIXED → ---
Thanks bugzilla. See comment 48.
Flags: needinfo?(cpearce)
(In reply to Inder from comment #50) > > Mike, > I am getting an irrecoverable black screen after tapping the "switch mode > button" at normal speed for more than ~35 times. This is on Otoro. It gets > worse if I take 4-5 pictures first and then try switching. In this case I > see a crash instead of black screen in ~15-20 tries. Inder, with unagi I took 5 photos and then cycled modes 50+ times. No camera issues observed. I'm using a build made with B2G_DEBUG=1; will try with B2G_DEBUG=0.
Okay, I can reproduce this problem (observed as a crash--I don't have cjones' patch yet), but _only_ on builds with B2G_DEBUG=0. In the Gecko camera layer, this should only affect logging macros; I'm checking those now to make sure none of them accidentally picked up any side-effects.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #48) > To recap what's happening here > - gaia switches back and forth between still-image preview and video > preview mode > - if that's done at a "normal" pace, we free up DSP pmem at a rate > sufficient to keep everything working > - if the switches are as fast as possible, we eventually run out of DSP > pmem and preview stops working > - we're not leaking anything, because the memory can be freed up (comment > 39) and preview made to work again > - if the camera app destroys and recreates its <video> after each switch, > then we don't hit this error case (attachment 685989 [details] [diff] [review] > [review]) > > This suggests to me that something that participates in <video> CC may be > holding on to mozSrcObject streams longer than expected. Destroying the > <video> in between stream changes fixes that problem. > > It also suggests gaia isn't doing anything terribly wrong. > > Is there a way we can pinpoint either > - what may not be participating in cycle collection and causing CC's to > delay too long > - what <video>-related code might be keeping mozSrcObject streams alive too > long > ? Are cycle collections actually happening? If you NSPR_LOG_MODULES=MediaStreamGraph:5 you should be able to see if uncollected MediaStreams are piling up. I would expect that a) MediaStreams are not collected until a CC happens and b) CC should collect all MediaStreams except the one currently attached to the video element.
Flags: needinfo?(cpearce)
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #56) > > Are cycle collections actually happening? > > If you NSPR_LOG_MODULES=MediaStreamGraph:5 you should be able to see if > uncollected MediaStreams are piling up. I would expect that a) MediaStreams > are not collected until a CC happens and b) CC should collect all > MediaStreams except the one currently attached to the video element. If I understand what's going on, we're running out of DSP memory, which is a chunk of physical memory that shouldn't be affected by whether or not the MediaStreams are being CCed. That is, provided the patch in bug 813190 is working properly.
(In reply to Mike Habicher [:mikeh] from comment #55) > > In the Gecko camera layer, this should only affect logging macros; I'm > checking those now to make sure none of them accidentally picked up any > side-effects. Checked: I don't see any logging statements in the camera code with side-effects.
Thanks for the tips, roc. mikeh can you give that a shot? I think that GonkNativeWindow is holding onto these buffers. I have no idea how it fits into the media-stream graph.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #59) > > Thanks for the tips, roc. mikeh can you give that a shot? Yes. I'll take a look later tonight. > I think that GonkNativeWindow is holding onto these buffers. I have no idea > how it fits into the media-stream graph. Something just occurred to me: the patch in bug 813190 fixes the case where a MediaStream holds onto the very last frame viewed indefinitely until the MediaStream is CCed. But what if the MediaStream has more than one frame pending rendering when the <video> element's .mozSrcObject is set to a new MediaStream? I'm seeing something like that in the call to GonkNativeWindow:abandon(). Since we can't free buffers in the RENDERING state, we leave them alone, and bump up buffer-generation counter; new buffers are then allocated with this new generation, and the old ones--when they're eventually released--are simply dealloc'd (instead of being put back into the buffer pool). That works, so long as the RENDERING buffers _are_ eventually released, but if they're stuck in a MediaStream that has no sink and get held onto until CC, then the bug 813190 patch won't help. :(
Flags: needinfo?(roc)
Right, the patch in bug 813190 stops mLastPlayedVideoFrame from hanging onto a buffer, but we could also have buffers queued up in the MediaStream's mStreamBuffer. You should be able to add some logging to mStreamBuffer to see whether that's the case. What *should* happen is that we get into the IsEnded state and StreamBuffer::ForgetUpTo is called to forget up to (or beyond) the end of the stream, releasing all buffers in the StreamBuffer, but possibly that isn't happening.
Flags: needinfo?(roc)
Priority: -- → P1
roc, I've confirmed that the non-CCed MediaStreams are hanging onto "missing" buffers. I tagged the MediaStream subclass' destructor, and when it fires (a whole bunch of them, actually), I see a whole bunch of buffers returned as well. In each case, the MediaStream seems to be hanging onto 1 or 2 buffers (about an even split), and up to 34 (or more) MediaStreams CCed in a single go--or ~50 buffers held until CC (on average). I tagged MediaSegmentBase::RemoveLeading() with a print right before the call to |nsTArray<Chunk> mChunks.RemoveElementsAt()| and it's not cleaning up these buffers--they're getting released when 'mChunks' is destroyed.
Some more info: I'm looking at a case where a MediaStream, which is about to be detached from its <video> element, has three buffers in its queue: 0, 6, and 7 (the numbers are arbitrary). In the logcat, I see this: I( 2671:0xa8d) mikeh: AdvanceTimeVaryingValuesToCurrentTime(nextCurrentTime=2419891, blockedTime=0) stream=0x43c66cc0 I( 2671:0xa8d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):43 : mikeh: aTime=1628013, forget=1625268 I( 2671:0xa8d) mikeh: RemoveElementsAt(aStartIndex=1, chunksToRemove=2) D( 2573:0xa24) /dev/pmem: Allocated buffer base:0x49ecc000 size:32768 offset:1851392 fd:175 D( 2671:0xa6f) /dev/pmem: Mapped buffer base:0x44c5a000 size:1884160 offset:1851392 fd:37 I( 2671:0xa77) GonkNativeWindow::returnBuffer: slot=6 (generation=0) I( 2671:0xa77) GonkNativeWindow::returnBuffer: slot=0 (generation=0) Then this: I( 2671:0xa8d) mikeh: AdvanceTimeVaryingValuesToCurrentTime(nextCurrentTime=2430689, blockedTime=0) stream=0x43c66cc0 I( 2671:0xa8d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):43 : mikeh: aTime=1638811, forget=1625268 I( 2671:0xa8d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):45 : mikeh: exiting early Note that although aTime has increased, forget (rounded to a 50ms increment) has not, so ForgetUpTo() will short-circuit return. Then: I( 2671:0xa8d) mikeh: AdvanceTimeVaryingValuesToCurrentTime(nextCurrentTime=2441435, blockedTime=6788) stream=0x43c66cc0 I( 2671:0xa8d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):43 : mikeh: aTime=1642769, forget=1625268 I( 2671:0xa8d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):45 : mikeh: exiting early At this point, aTime on this stream stops incrementing, and forget never changes from its last value, so there is no more opportunity to "forget" the last remaining buffer in the queue.
Re: the above--would this happen because our frame times are 30Hz (~33ms/frame), which is less than the 50ms pruning interval?
Target Milestone: --- → B2G C2 (20nov-10dec)
(In reply to Mike Habicher [:mikeh] from comment #64) > > Re: the above--would this happen because our frame times are 30Hz > (~33ms/frame), which is less than the 50ms pruning interval? Okay, I tried dropping the roundTo interval to 25ms, but that didn't do it. So after some more digging, I at least understand what's going on (though I'm not exactly sure how to fix it yet). I got here by modifying StreamBuffer::ForgetUpTo() so that every 10 iterations (mSkipCount), it ignores the short-circuit return and proceeds to the for-loop anyway. Once there, it never hits the if-condition, but hits the |track->forgetUpTo(forgetTo)| call with the following values: printf_stderr("%s:%d : mikeh: forgetTo=%lld (track->GetEnd()=%lld, track->TimeToTicksRoundDown(forget)=%lld)\n", __func__, __LINE__, forgetTo, track->GetEnd(), track->TimeToTicksRoundDown(forget) ); track->ForgetUpTo(forgetTo); I( 4192:0x107d) mikeh: AdvanceTimeVaryingValuesToCurrentTime(nextCurrentTime=1238167, blockedTime=10863) stream=0x43cc4cc0 I( 4192:0x107d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):43 : mikeh: aTime=664098, forget=629136 (mSkipCount=10) I( 4192:0x107d) void mozilla::StreamBuffer::ForgetUpTo(mozilla::StreamTime):64 : mikeh: forgetTo=17 (track->GetEnd()=19, track->TimeToTicksRoundDown(forget)=17) I( 4192:0x107d) void mozilla::MediaSegmentBase<C, Chunk>::ForgetUpTo(mozilla::TrackTicks) [with C = mozilla::VideoSegment, Chunk = mozilla::VideoChunk]:164 : mikeh: aDuration=17, mDuration=19, extraToForget=0 I( 4192:0x107d) void mozilla::MediaSegmentBase<C, Chunk>::ForgetUpTo(mozilla::TrackTicks) [with C = mozilla::VideoSegment, Chunk = mozilla::VideoChunk]:174 : mikeh: skipping call to RemoveLeading() In this case, it looks like if forgetTo were 18 (either by not subtracting 1 from track->GetEnd() or perhaps by not rounding down |forget|), then extraToForget in MediaSegmentBase::ForgetUpTo() would be 1, and we could hit the RemoveLeading() function call which may drop the final buffers.
Flags: needinfo?(roc)
Depends on: 816817
The patch in bug 816817 fixes the long-lived buffer condition; with that change, pmem_adsp memory is cleaned up in a timely fashion. Unfortunately, a frequent mode-switch crash still exists, which now appears as: D( 410:0x1a2) /dev/pmem_adsp: Mapped buffer base:0x427a4000 size:155648, fd:34 I( 410:0x1a2) IPDL error: I( 410:0x1a2) error deserializing (better message TODO) I( 410:0x1a2) [Child 410] ###!!! ABORT: [PImageContainerChild] abort()ing as a result: file /home/mikeh/dev/mozilla/btg015/objdir-gecko/ipc/ipdl/PImageContainerChild.cpp, line 527 F( 410:0x1a2) Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1) The "error deserializing" string appears all over the ipdlc-generated code, but only in conjunction with FatalError() here: if ((!(Read((&(image)), (&(__msg)), (&(__iter)))))) { FatalError("error deserializing (better message TODO)"); return MsgValueError; } ...at lines 382..385 in PImageContainerChild::OnMessageReceived().
Flags: needinfo?(roc)
I was unable to reproduce the crash with some pretty hard beating, but I was testing on beta. Do you have some "swing thoughts" that might help repro?
(In reply to Chris Jones [:cjones] [:warhammer] from comment #67) > > I was unable to reproduce the crash with some pretty hard beating, but I was > testing on beta. Do you have some "swing thoughts" that might help repro? I've basically been following the steps you and Inder laid out above: open camera, take a few pictures, cycle through the camera modes at 0.5 to 5Hz, try taking a few more photos, etc. Since I did a rebuild, I haven't seen the crash in comment 66 again, so I'm wondering if it's an artifact of a bad build. This is with 100+ mode cycles.
I see this issue fixed by the patch in bug 816817.
Whiteboard: [caf:blocking] → [caf:blocking][fix in bug 816817]
This is looking pretty good on inbound. Should be able to uplift to beta tonight.
Comment on attachment 685989 [details] [diff] [review] WIP of a (not too terribly gross) workaround Let's pretend this patch never existed :).
Attachment #685989 - Attachment is obsolete: true
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
NB: this landing in comment 73 is being blamed for a win32 tscroll and Paint regression on aurora, but that's literally impossible. This code doesn't even compile on win32, let alone run there. This isn't the patch you're looking for ;).
I'm experiencing something similar with B2G 1.0.0-prerelease 18.0 2013011007021. If I keep switching back and forth between video and camera mode, I eventually hit a black screen and the phone reboots. It usually reproduces after 5 or so switches back and forth. Is this the same bug? Should I reopen, file a new bug? Thanks
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #75) > I'm experiencing something similar with B2G 1.0.0-prerelease 18.0 > 2013011007021. If I keep switching back and forth between video and camera > mode, I eventually hit a black screen and the phone reboots. It usually > reproduces after 5 or so switches back and forth. Is this the same bug? > Should I reopen, file a new bug? > > Thanks I reproduced Anthony's crash/application exit, verbatim, and the stack trace I got from DDMS was linked to bug 827833. My incident was https://crash-stats.mozilla.com/report/index/2bac1e9d-9278-4815-ac59-8d44b2130112
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #75) > I'm experiencing something similar with B2G 1.0.0-prerelease 18.0 > 2013011007021. If I keep switching back and forth between video and camera > mode, I eventually hit a black screen and the phone reboots. It usually > reproduces after 5 or so switches back and forth. Is this the same bug? > Should I reopen, file a new bug? > > Thanks I'd file a new bug and link it as a dependency on this bug. There was a lot of changes that happened during the work week, so something might have regressed.
Blocks: 830008
Device: Unagi Build Identifier: 20130113070202 Update channel: nightly b2g18 (pvt server) unagi.zip 13-Jan-2013 08:06 100M https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/ Git commit info: 2013-11-13 15:15:51 Verified fixed!
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: