1167535 - [Nexus 5] fanotify EOVERFLOW error for 4g up video file size in GonkDiskSpaceWatcher().

Status: RESOLVED FIXED

(Core :: Hardware Abstraction Layer (HAL), defect)

Comment 1

[Hermes Cheng (inactive after July 27, 2015)]

* STR:
1. restart phone
2. unlock phone, and launch Camera at Homescreen
3. start record with flashlight on
4. keep recording more than 30 minutes

* Actual result:
After step 4, the phone is really hot, and then b2g crashes. After reboot, when I try to launch video and audio app, the b2g would crash. However, gallery can be launched.

* Crash Report:
https://crash-stats.mozilla.com/report/index/0bebbb46-999d-42c0-8dd0-25bdf2150522

Below is the list. Its content seems similar.
root@hammerhead:/data/b2g/mozilla/Crash Reports/submitted # ll
-rw-r----- root     root           49 2015-05-21 04:41 bp-0bebbb46-999d-42c0-8dd0-25bdf2150522.txt
-rw-r----- root     root           49 2015-05-21 05:10 bp-1243542b-e726-40b2-b429-a12ba2150522.txt
-rw-r----- root     root           49 2015-05-21 05:17 bp-36c9b0b5-c321-4f50-8240-8486b2150522.txt
-rw-r----- root     root           49 2015-05-21 05:19 bp-57f12ce4-ec0a-4173-8b0e-6e0252150522.txt
-rw-r----- root     root           49 2015-05-21 05:18 bp-6875bb3f-55d2-476d-97d3-b4ee52150522.txt
-rw-r----- root     root           49 2015-05-21 05:20 bp-90298975-0ec6-46f0-a74e-25bb12150522.txt
-rw-r----- root     root           49 2015-05-21 04:48 bp-cba1de2b-1bf0-4698-b4fa-a12bc2150522.txt
-rw-r----- root     root           49 2015-05-21 04:41 bp-d643f430-1a20-4823-9db1-c32402150522.txt
-rw-r----- root     root           49 2015-05-21 05:20 bp-fe3a292f-6d31-4acd-8b13-af6432150522.txt


* Test Env:
Build ID               20150521002508
Gaia Revision          bc42fbc12d622bffd7e8afcb8d56f8a1d9773c60
Gaia Date              2015-05-20 22:32:56
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/6e4eaf59efda
Gecko Version          37.0
Device Name            hammerhead
Firmware(Release)      5.1
Firmware(Incremental)  eng.cltbld.20150521.040628
Firmware Date          Thu May 21 04:06:43 EDT 2015
Bootloader             HHZ12f

Comment 2

[Hermes Cheng (inactive after July 27, 2015)]

Attachment: crash logcat

Comment 3

[Hermes Cheng (inactive after July 27, 2015)]

@Josh, please help here.

Comment 4

[Eric Chang [:ericcc] [:echang]]

Let me try this on flame-kk

Comment 5

[Hermes Cheng (inactive after July 27, 2015)]

@Eric, we might also need to test this without flashlight on. Could you ask Marigold for help?

Comment 6

[Eric Chang [:ericcc] [:echang]]

Hi Peter, Could you also check this on FLAME-KK with master, from my test on v2.2, video recording is not even working, I need to create a new bug after I confirm that again. Thanks.

Comment 7

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Here is the kernel log.
The VID_0001.3gp is the video with 30min recording content.

<36>[   50.217784] type=1400 audit(1432285370.500:46): avc: denied { search } for pid=531 comm=53747265616D5472616E73202331 name="/" dev="fuse" ino=0 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=dir
<36>[   50.218458] type=1400 audit(1432285370.500:47): avc: denied { read } for pid=531 comm=53747265616D5472616E73202331 name="0" dev="fuse" ino=1 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=dir
<36>[   50.218740] type=1400 audit(1432285370.500:48): avc: denied { open } for pid=531 comm=53747265616D5472616E73202331 name="0" dev="fuse" ino=1 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=dir
<36>[   50.223230] type=1400 audit(1432285370.500:49): avc: denied { getattr } for pid=531 comm=53747265616D5472616E73202331 path="/mnt/shell/emulated/0/DCIM/100MZLLA/VID_0001.3gp" dev="fuse" ino=5 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=file
<36>[   50.288781] type=1400 audit(1432285370.570:50): avc: denied { read } for pid=3115 comm=426C6F62204F70656E6572 name="VID_0001.3gp" dev="fuse" ino=5 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=file
<36>[   50.289212] type=1400 audit(1432285370.570:51): avc: denied { open } for pid=3115 comm=426C6F62204F70656E6572 name="VID_0001.3gp" dev="fuse" ino=5 scontext=u:r:init:s0 tcontext=u:object_r:fuse:s0 tclass=file
<36>[   50.385785] type=1400 audit(1432285370.670:52): avc: denied { transfer } for pid=732 comm="Binder_1" scontext=u:r:mediaserver:s0 tcontext=u:r:init:s0 tclass=binder
<36>[   50.386797] type=1400 audit(1432285370.670:53): avc: denied { call } for pid=3118 comm=4D65646961506C7E6261636B202332 scontext=u:r:init:s0 tcontext=u:r:init:s0 tclass=binder
<36>[   50.387200] type=1400 audit(1432285370.670:54): avc: denied { transfer } for pid=3118 comm=4D65646961506C7E6261636B202332 scontext=u:r:init:s0 tcontext=u:r:init:s0 tclass=binder

Comment 8

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Attachment: fix fanotify 4g size limit. v1

Add O_LARGEFILE to support 4g up size for 32bit os.

Comment 9

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Comment on attachment 8609546 [details] [diff] [review]
fix fanotify 4g size limit. v1

When device keeps recording more than 30 minutes, the video file will reach 4G size.
We don't handle the EOVERFLOW error in GonkDiskSpaceWatcher::OnFileCanReadWithoutBlocking(), then we get crash at [1].

This patch adds O_LARGEFILE flag to support 4G up file size
[1]
https://hg.mozilla.org/mozilla-central/annotate/3e737d30f842/hal/gonk/GonkDiskSpaceWatcher.cpp#l261

Comment 10

[Jerry Shih[:jerry] (UTC+8) (inactive)]

try link:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=208a0c57038d

Comment 11

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Please land attachment 8609546 [details] [diff] [review] to m-c.
try link:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=208a0c57038d

Comment 12

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Comment on attachment 8609546 [details] [diff] [review]
fix fanotify 4g size limit. v1

[Approval Request Comment]
Bug caused by (feature/regressing bug #):
Video recording size exceed 4G.
User impact if declined:
When we have a large video file, B2G always crash when we click video app. 
Testing completed:
Test manually at m-c and v2.2 with 4G file. I don't get crash in video app.
Risk to taking this patch (and alternatives if risky):
Low. This patch try to support 4g up file at 32bit system.
String or UUID changes made by this patch:
None.

Comment 13

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Should we also have this patch in b2g34 and b2g32? I don't know how to delete the 4G up file without video app unless I use adb command to kill it. If I try to open video app, b2g will crash.

Comment 14

[Eric Chang [:ericcc] [:echang]]

* NEXUS-5-L V2.2
** Reproducible on NEXUS-5-L, flash is not a factor here. 
1. restart phone
2. unlock phone, and launch Camera at Homescreen
4. keep recording more than 30 minutes

* FLAME-KK V2.2, V3.0
** Not reproducible on FLAME-KK, but I am trying to record for longer time to make the clip larger in file size, resolution & bitrate of flame is lower than NEXUS

Video clip by Flame-KK V2.2 - res:1280x720, bitrate: 38.8Kbps, file size 1.9GB for 45 min recording.
Video clip by NEXUS-5-L V2.2 - res:1920x1080, bitrate: 16.6 Mbps,

Comment 15

[Jerry Shih[:jerry] (UTC+8) (inactive)]

I think you should try to record for 90min. Thus, we can get a 4g up video file.

Comment 16

[Eric Chang [:ericcc] [:echang]]

Attachment: 2015-05-23-20-56-43.png

FLAME pops a warning says that my storage is full when the file is around 3.8GB(60min), but actually there is still 40% room left in my SD card, I need a gaia or gecko tweak to make it more than 4GB.

Comment 17

[Josh Cheng [:josh]]

Waiting patch land on master for b2g37 approval

Comment 18

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Josh, is this v2.2 blocking?

Comment 19

[Josh Cheng [:josh]]

(In reply to Jerry Shih[:jerry] (UTC+8) from comment #18)
> Josh, is this v2.2 blocking?

Hi Jerry,
I make this as 2.2+ since this will lead to device crash which is not acceptable.
Thanks

Comment 20

[helen（leave from Mozilla）]

Attachment: v2.2.mp4

I tried to do regression window-wanted on N5-L v2.2, but I found that this bug also exists on the latest nightly build which is newer than the oldest b2g-inboud build. Below is my test result using the nightly build:

Results: After recording more than 30 minutes, the device crashes. After rebooting it, when launch video and audio app, the device would crash but gallery can be launched.

See attachment: v2.2.mp4
Reproduce rate: 5/5

Device: Nexus 5 v2.2(Fail)
Build ID               20150224162516
Gaia Revision          ca64f2fe145909f31af266b1730874051ba76c78
Gaia Date              2015-02-24 22:06:53
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/16804008c29f
Gecko Version          37.0
Device Name            hammerhead
Firmware(Release)      5.0
Firmware(Incremental)  eng.cltbld.20150224.201149
Firmware Date          Tue Feb 24 20:12:04 EST 2015
Bootloader             HHZ12f

Comment 21

[Josh Cheng [:josh]]

(In reply to helen from comment #20)
> Created attachment 8610084 [details]
> v2.2.mp4
> 
> I tried to do regression window-wanted on N5-L v2.2, but I found that this
> bug also exists on the latest nightly build which is newer than the oldest
> b2g-inboud build. Below is my test result using the nightly build:
> 
Helen,
I think you mean:
"I can reproduce the issue on latest nightly build on N5-L v2.2"
Please be aware of your wording you use for your statement. Do not mention irrelevant information like "regression window-wanted..." 


> Results: After recording more than 30 minutes, the device crashes. After
> rebooting it, when launch video and audio app, the device would crash but
> gallery can be launched.
> 
> See attachment: v2.2.mp4
> Reproduce rate: 5/5
> 
> Device: Nexus 5 v2.2(Fail)
> Build ID               20150224162516
> Gaia Revision          ca64f2fe145909f31af266b1730874051ba76c78
> Gaia Date              2015-02-24 22:06:53
> Gecko Revision        
> https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/16804008c29f
> Gecko Version          37.0
> Device Name            hammerhead
> Firmware(Release)      5.0
> Firmware(Incremental)  eng.cltbld.20150224.201149
> Firmware Date          Tue Feb 24 20:12:04 EST 2015
> Bootloader             HHZ12f

Comment 22

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/847a0f04a73c

Comment 23

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/847a0f04a73c

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/7f2324ea3dde

Comment 25

[Hermes Cheng (inactive after July 27, 2015)]

Currently, there is no crash now but got some message mentioned at bug 1174626.
Still try to figure out the definition of this msg, so let's wait until bug 1174626 is resolved.

Comment 26

[Hayden Huang [:haydenh]]

It's limited in media/libstagefright/MPEG4Writer.cpp

static const int64_t kMax32BitFileSize = 0x00ffffffffLL; // 2^32-1 : max FAT32
                                                         // filesystem file size
                                                         // used by most SD cards

NI Munro to see if we can do a dynamic file size limit based on the storage file system.

Comment 27

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

We need to detect the storage type (internal or external SD card) and set the max file size accordingly.
I will try to implement this mechanism.

Comment 28

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Current design limits the maximum recording clip to 4GB, so it will stop recording after around 30 minutes. What's the expected behavior? Remove the limitation and only stop the recording when the whole storage is full? Or increase the maximum size to a larger number like 8GB?

Comment 29

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Munro Chiang [:mchiang] from comment #28)
> Current design limits the maximum recording clip to 4GB, so it will stop
> recording after around 30 minutes. What's the expected behavior? Remove the
> limitation and only stop the recording when the whole storage is full? Or
> increase the maximum size to a larger number like 8GB?

Currently, the message is "You have run out of media storage space" which might let user misunderstand the actual meaning. Either changing the notification message or changing the behavior to stop the recording when storage is full are both acceptable for me.

Besides, how did we define 4GB as the maximum size? Does it fit to a newer device?

I think we need UX to define clear behavior.
@Harly, do you know who can help here?

Comment 30

[Harly Hsu[:harly]]

Yes, I do think the message is misleading to the user, and probably show something like "You have reached the maximum file size for a video clip" would make more sense. I am wondering is the 4GB limitation because of sdcard file system uses FAT32? Also needinfo Rob as he might have more insight to the camera stuff.

Comment 31

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

4GB file size is limitation of FAT32.
If the file is stored at ext4 partition, like nexus 5 and Aries, the max file size 16 TiB.

Comment 32

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

4GB file size is limitation of FAT32.
If the file is stored at ext4 partition, like the internal storage of nexus 5 and Aries, the max file size can reach 16 TiB.
If there is a way for camera app to query current file system, then camera app can set the max file size to the whole free space when ext4 partition is used.

Or we can keep current design and just modify the misleading message. However, just be noticed that 4GB can only afford 30 minutes 1080p recording or 10 minutes 4K recording.

Comment 34

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

Create bug 1184484 to demand more information (file system type) from devicestorage API.

Comment 35

[Pi Wei Cheng [:piwei] (inactive)]

Excluding this bug in our queries because we don't have a Nexus 5.