1191760 - music app doesn't load all music

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Music, defect)

Description

[Natalia Martinez-Winter]

User-Agent: Mozilla/5.0 (Mobile; rv:42.0) Gecko/42.0 Firefox/42.0

I inserted a 4GB SD card with a dozen albums,  only 2 show up in the music app. There's NO way to refresh the list (even killing the app and open again doesn't refresh or removing and inserting the card again). I inserted the same card in my zte open C ( firedox os 1.3) and it works fine, all songs are loaded. Looks like a huge regression.

Comment 1

[Mahendra Potharaju [:mahe]]

[Blocking Requested - why for this release]: Broken Functionality

Comment 2

[Hema Koka [:hema]]

Please investigate and route for fix within current music devs

Comment 3

[Jim Porter (:squib)]

What version are you on now? What format are the files? Are you sure they're valid files? Do you see anything in the logcat? Does the "scanning" spinner in the top left disappear or does it stay around forever?

Comment 4

[Hubert Figuiere [:hub]]

I could have had a look at this bug this week while I was onsite :-/

Comment 5

[Hubert Figuiere [:hub]]

At that point if we can get a perfect image of the SD card so we can test here, it would really help.

Comment 6

[Hubert Figuiere [:hub]]

Natalia, do you think you can get an "image" of your SD card sent made available for us (Music team) to test? (in confidence of course) We really need to figure out what's going on.

Thanks,

Comment 7

[Natalia Martinez-Winter]

mmh... I have no idea how to do that, maybe if you explain, I can try to do that.
but why would the same SD card work perfectly on my Fx0, my ZTE Open C and NOT on my Sony ??? that really seems strange to me. all the files are MP3 (and the same SD card obviously works perfectly well on any of my Android devices !)

Comment 8

[Hubert Figuiere [:hub]]

(In reply to Natalia Martinez-Winter from comment #7)
> mmh... I have no idea how to do that, maybe if you explain, I can try to do
> that.

I think at that point we should ask one of our friendly engineers in Paris :-)

> but why would the same SD card work perfectly on my Fx0, my ZTE Open C and
> NOT on my Sony ??? that really seems strange to me. 

Do they all run the same version of the OS?

> all the files are MP3
> (and the same SD card obviously works perfectly well on any of my Android
> devices !)

The problem isn't the SD card, but its content (some OS tend to leave invisible files scattered through the disk) and the fact that we have put a lot of work on the metadata parsing in the Music app that might lead to some unwanted side effect (as usual when you change code). We really want to address that, but right now we are flying blind - this would help a lot.

Comment 9

[Natalia Martinez-Winter]

Open C : 1.3
Fx0 :  2.0
Sony: 2.5
(so I see 2.5 as a regression)

Comment 10

[Hubert Figuiere [:hub]]

Indeed it is a regression. I have no doubts about it.

Comment 11

[Pi Wei Cheng [:piwei] (inactive)]

I was NOT able to reproduce this issue on Aries. I inserted an SD card with 28 songs. All of the songs are loaded into Music app and can be played.

Leaving qawanted tag for someone else to attempt.

Device: Aries (RC4 and OTA'ed to dogfood-latest)
BuildID: 20150812135535
Gaia: 6fef72357971934c8774578044ea7a442be3a75d
Gecko: b7efed3a2640fba681d48846c3b2ab80474c6f77
Gonk: 2916e2368074b5383c80bf5a0fba3fc83ba310bd
Version: 43.0a1 (2.5 Master) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Comment 12

[Jim Porter (:squib)]

I doubt anyone will be able to reproduce this without the exact files that are causing the problem. We have fairly-thorough unit tests to ensure that this won't happen, but audio files can be broken in a lot of different ways, and we might not be catching all possible errors.

Comment 13

[Pi Wei Cheng [:piwei] (inactive)]

One of our testers in the lab were seeing this issue on Aries but they couldn't get a consistent repro. Putting steps-wanted to find some possible steps or at least get some logs.

Comment 14

[John Thomas [:Johnt]]

I have some STR with about 8/10 repro.

1. Open Browser.
2. Navigate to http://www.mp3juices.cc/
3. Download a song.

Result: Mp3 will not always be displayed in Music folder. 

Note: Originally I thought this was a 100% repro, however after about 45 minutes the newest songs downloaded all appeared at once when re-entering the music library. (I believe no restart occurred.)

I was also seeing that songs downloaded from an email were also not appearing in music library as well. Not sure if this is specific to mp3s only but previously when I downloaded some .ogg and .wav files from Email and browser they seemed to of showed up right away. 

Attaching a logcat.

Comment 15

[John Thomas [:Johnt]]

Attachment: logcat_20150814_1217.txt

Comment 16

[Peter Bylenga [:PBylenga]]

Switching stepswanted to qawanted to confirm steps in comment 14 and find a window.

Comment 17

[Hubert Figuiere [:hub]]

It is not a regression window that we need. It is the test case that can reproduce. We did heavy changes to the metadata parser so regressions of that nature are not surprising.

Comment 18

[Jim Porter (:squib)]

Yeah, if you can post a song file that reproduces the issue to this bug (or email it to me directly), I can fix it.

Comment 19

[Duane Dixon [:ddixon]]

(In reply to Jim Porter (:squib) from comment #18)
> Yeah, if you can post a song file that reproduces the issue to this bug (or
> email it to me directly), I can fix it.

Hi Jim, 

I used the STR from Comment 14 and saw the same results by downloading an .mp3 file from (https://www.youtube.com/watch?v=6BqxIqPDG0c) 

Repro rate: 3/3

Actual Results: User is given an error that the file can't be opened or the format is not compatible with the device.  The .mp3 file did not show up in the music library unless unless the user closes and reopens the Music app. 

Environmental Variables:

Device: Flame Master
BuildID: 20150817073034 (nightly, full flash)
Gaia: 60489c1ff8c5d1633fc4837d4f8019623d4e1940
Gecko: a6eeb28458fd2652e12e57334f046b7776d75bb4
Gonk: c4779d6da0f85894b1f78f0351b43f2949e8decd
Version: 43.0a1 (Master) 
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Comment 20

[Jim Porter (:squib)]

(In reply to Duane Dixon [:ddixon] from comment #19)
> Actual Results: User is given an error that the file can't be opened or the
> format is not compatible with the device.  The .mp3 file did not show up in
> the music library unless unless the user closes and reopens the Music app. 

That's bug 1193498, which is an issue with the open activity. I don't think it applies here. I tried downloading that file and putting it on my SD card and everything works just fine for me.

Comment 21

[Justin D'Arcangelo [:justindarc]]

I'm wondering if this may have been caused by Bug 1190656 which landed on 8/11. Would it be possible for you to re-test this on master?

Comment 22

[GitHub Autolander Bot]

Attachment: [gaia][landed] jimporter:music-bulletproof-metadata > mozilla-b2g:master

Comment 23

[Jim Porter (:squib)]

Comment on attachment 8648966 [details] [review]
[gaia][landed] jimporter:music-bulletproof-metadata > mozilla-b2g:master

This patch is purely speculative, but it tries to bulletproof our metadata parsers a little more. The only behavioral changes I made are:

1) MP3s with invalid ID3v2 headers will fall back to ID3v1 instead of being rejected outright.
2) FLAC files with album art no longer suffer from a race condition when loading the album art parser.

Comment 24

[Duane Dixon [:ddixon]]

Thanks for the help, Jim.  

I've retested this bug based on Comment 1 findings.  

Description: 
Loading multiple MP3 files onto the SD card in a Flame device may result in only part of those files being imported.  I copy and pasted files in the Music folder while the phone was plugged into my computer via USB.  After detaching the phone and opening the Music app then going to "Recently Added", it shows that only 2 of the 4 files have successfully loaded onto the SD card. 

Repro Steps: 
1. Connect the Flame device to a computer. 
2. Copy/paste multiple (4) MP3 files from computer and into "Music" folder on the Flames SD card.
3. Disconnect device from computer and launch Music app. 
4. Go to Playlist>"Recently added". 

Actual Results: All MP3 files are NOT successfully imported to a Flame's SD card when they are added to the Music folder. 
 
Repro Frequency: 5/10 

Note: I'm unsure if the length of the MP3 file has an impact on the repro rate, however, one of my test files was 60 minutes.  The other three were about 4 minutes each.  

Environmental Variables:
Device: Flame 2.5 (shallow flash)
BuildID: 20150817060738
Gaia: 60489c1ff8c5d1633fc4837d4f8019623d4e1940
Gecko: a6eeb28458fd2652e12e57334f046b7776d75bb4
Version: 43.0a1 (Master) 
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Comment 25

[Jim Porter (:squib)]

Can you post the files somewhere so I can take a look at them? It's possible there's something weird in the files that's causing us problems.

Comment 26

[Hubert Figuiere [:hub]]

Comment on attachment 8648966 [details] [review]
[gaia][landed] jimporter:music-bulletproof-metadata > mozilla-b2g:master

LGTM

Comment 27

[Jim Porter (:squib)]

Landed the speculative fixes: https://github.com/mozilla-b2g/gaia/commit/b14e44816535a9e1e607f67d678e4f18d944085d

Comment 28

[Hema Koka [:hema]]

Duane: Without the actual files to reproduce this issue it is tough to put in fixes. Nevertheless, could you please test with the speculative patch from Jim. 

Since this is not reproducible easily and the issue seems to be the specific set of files on DUT, this may not be a release blocker. Lets see if the patch helps in anyway for those who are able to reproduce and revisit blocking decision then. Putting this back in the nom queue.

Comment 29

[Duane Dixon [:ddixon]]

(In reply to Jim Porter (:squib) from comment #27)
> Landed the speculative fixes:
> https://github.com/mozilla-b2g/gaia/commit/
> b14e44816535a9e1e607f67d678e4f18d944085d

Jim, 

I have downloaded your Gaia repository via Github in order to test your fix.  However, I cannot find the the patch name in the "Branch: master" of the repository.  Thus, I'm unable to install the correct Gaia to test your fix.  Do you know where I can find the particular patch and what the name of it is?  Thanks!

Comment 30

[Jim Porter (:squib)]

It's in gaia/master. The link you quoted contains the patch.

Comment 31

[Duane Dixon [:ddixon]]

I was unable to reproduce this issue on the latest Flame 2.5 (full flash, nightly) build.  All of the music files I added into the phone's SD card imported correctly when viewed in the Music app.  It appears that the patch listed in Comment 27 has fixed this issue. 

Repro Attempts: 0/10

Device: Flame Master
Build ID: 20150827083721
Gaia: d784c81961d82cbe9e111405468c590a8345856c
Gecko: b33eae31bd7188024b54228e0c0345800a65e595
Gonk: c4779d6da0f85894b1f78f0351b43f2949e8decd
Version: 43.0a1 (Master)
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Comment 32

[Hubert Figuiere [:hub]]

Let's wait on Natalia to be able to test it with her SDcard. Once this lands in the OTA (which should be the next OTA for Aries)

Comment 33

[Hema Koka [:hema]]

Removing from nom. Natalia, please test with the patch and we can land this if it works fine for you.

Thanks
hema

Comment 34

[Jim Porter (:squib)]

The patch has landed. Perhaps I should have filed a separate bug for the speculative fix...

Comment 35

[Hubert Figuiere [:hub]]

Bug 1200401 has some similarities. But this time I'm get an image of the SDcard so we can test this....

Comment 36

[Reece]

I am also bumping into this issue every single time I shallow_flash 2.5 and launch the music app. Unfortunately I can only add that I have a 4 gigabyte SD card that is almost full, I have over 400 songs and all of them I have ran through iTunes to add album art and stuff. The music app loads less than 15% of my music and then stops attempting to load the rest. 

I am using the latest (as of the 5th of September) shallow_flashed version of 2.5 so it hasn't yet been fixed by a patch.

Comment 37

[Jim Porter (:squib)]

As said above, there's really nothing we can do without sample files. While it would be best to get a full image of the SD card, if you can provide a sample file that produces this issue, that would help a lot. One way to do this would be to start with an album at a time and see what the first album that causes a failure is (preferably while resetting the profile in between each try, but that might not be strictly necessary).

While I've occasionally run into issues with my music (I have about 11,000 songs and rotate what's on my phone periodically), all of the ones I've encountered have been fixed. It's entirely possible there are still bugs, but we *need* the offending files or this bug will end up being resolved as incomplete.

Comment 38

[Jim Porter (:squib)]

Oh, if nothing else, a logcat would at least give us some hints as to what's happening.

Comment 39

[Reece]

If you let me know of a place where I can do it I'm more than happy to send a full copy of my SD card to you (as the problem never fails to occur). I'm not quite sure how to send a logcat, sorry I'm a noob.
I run a shallow_flashed version of 2.5 on my Flame with the v18d_nightly_v4 base image and I always make sure to run the latest update.

Comment 40

[Reece]

Just an update, I shallow_flashed the latest 2.5 last night and the problem still occurs. A miniscule amount of music loads into the music app for me every time. Perhaps it's a shallow_flash only problem?

Comment 41

[Jim Porter (:squib)]

Try wiping your profile out (or at least resetting the music database). It's possible that the files have already been marked as invalid and won't be rescanned.

Comment 42

[Reece]

I think I already have, I actually went into recovery and wiped data and cache. The problem still persists.

Comment 43

[Jim Porter (:squib)]

Then you could try what I suggested in comment 37 to narrow down which songs are causing your problems. If you can narrow it down to just a few of them, it would be easier for me to take a look at the files in question.

A logcat would also be helpful.

Comment 44

[Reece]

Narrowing down the songs will take me a while, I have for example one album that is a soundtrack with 110 songs in it and that's just one album. Honestly about 80% of my music library fails to load every single time even after resets. I don't know how to do a logcat I'm a bugzilla noob, I'm just trying to help. 
I'm running V18D_nightly_v4 base image, I'm running the most up to date shallow_flash Flame v2.5 version (manually as I'm having OTA update issues). 

I'll give any info I know how to, I'll send any files I need to send just let me know how.

Comment 45

[Jim Porter (:squib)]

Chances are good that there's a single file (or a single album) that's causing the scanning process to stall. If you just remove half the songs and try, you should be able to narrow it down pretty fast. Assuming it's just a single file causing the problem, you'd only need to try 8 or 9 times starting with a set of 400 songs.

To get a logcat, just run `adb logcat > my_log.txt` from a command prompt. For more info, see [1] and [2].

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Debugging/On-device_console_logging#Using_logcat
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Debugging/Developer_settings

Comment 46

[Reece]

Okay, I'll try narrow it down for you and I'll post my logcat here soon. Thanks for the explanation!

Comment 47

[Reece]

I've attempted to narrow it down but it seems to happen regardless of what albums are on my phone. It stalls anyway and refuses to load anymore. Maybe it is down to the format of my songs?

I run all of my songs through iTunes on OS X to add tagging and album art work to my music. They are all MP3 files at 256kbps. Could you attempt to load a song onto your phone in this specific format to see if stalls the loading process for you?

By typing adb logcat I got long, seemingly neverending terminal text and this caught my eye:

"W/Music   ( 1465): Content JS WARN: MediaDB: error parsing metadata for /sdcard1/Music/Bleed From Within/Uprising/._09 Our Divide.mp3 : Unplayable music file 
W/Music   ( 1465):     at metadataError (app://music.gaiamobile.org/gaia_build_defer_index.js:363:27)
W/Music   ( 1465): [JavaScript Warning: "Media resource blob:app://music.gaiamobile.org/4e6db650-fbb6-448a-8b65-13dc95efa134 could not be decoded." {file: "app://music.gaiamobile.org/index.html" line: 0}]"

I've pasted just one here so that this message doesn't get too long but this has appeared in the terminal for every single song (even ones that actually do display in the music app)

I have a logcat but it's extremely long, am I okay to post it here as a response to this or should I upload the txt as an attachment?

Comment 48

[Hubert Figuiere [:hub]]

I thought we did blacklist these ._ files?

Comment 49

[Jim Porter (:squib)]

No. See bug 1186146.

Comment 50

[Reece]

I've had a read, so should I go through every album and delete these extra ._ files?
This is a pretty big problem for everyone on OS X if the OS doesn't read these, there should be some obvious documentation on the mozilla flame flash page

Comment 51

[Reece]

So I used a utility called "BlueHarvest" to fix it and I can confirm after wiping the ._ files from the "Music" folder on my SD card every single one of my songs loads onto the Flame (and they were read really fast too!).

How can this be fixed for the end user? 2.5 is shaping up to be a fantastic release and it will suck if OS X users don't get to enjoy it because they have to use third party tools to delete hidden files.

Comment 52

[Jim Porter (:squib)]

Well, we're going to ignore those files in bug 1186146, which will resolve the issue, but there's still a bug here: it should be impossible for a bad file to stall the scanning process. If you can send me a few sample ._ files, I'll take a look at them and write a test to ensure that we don't get confused by them.

Ideally, you can just zip a few of them up and post them to the bug, but if you'd rather send it to me directly, just use the email address for my Bugzilla account.

Thanks for investigating this on your end!

Comment 53

[Reece]

I've sent a zip named "Bug 1191760 Problematic Music Files" to the email you've used for Bugzilla, hopefully they're the files you need.

No problem, thanks for baring with me I'm still learning how to use Bugzilla.

Comment 54

[Jim Porter (:squib)]

Here's my current theory: there's something peculiar about a subset of OS X resource forks that will cause our checkPlayability() function to stall because the Audio element fires neither "error" nor "canplay" events. I'm guessing there's another event we need to listen for, but I don't know what one. If I can get a resource fork file that reproduces the error, I should be able to figure it out.

Comment 55

[Hubert Figuiere [:hub]]

(In reply to Jim Porter (:squib) from comment #54)
> Here's my current theory: there's something peculiar about a subset of OS X
> resource forks that will cause our checkPlayability() function to stall
> because the Audio element fires neither "error" nor "canplay" events. I'm
> guessing there's another event we need to listen for, but I don't know what
> one. If I can get a resource fork file that reproduces the error, I should
> be able to figure it out.

I emailed you a resource file that I know fails on Flame.

Comment 56

[GitHub Autolander Bot]

Attachment: [gaia] jimporter:music-check-playability > mozilla-b2g:master

Comment 57

[Jim Porter (:squib)]

I still need to test with the files from Reece, but the above patch fixes the issue for the resource file :hub sent me. Hopefully we can get this resolved soon. Thanks everyone for the detective work!

Comment 58

[Hubert Figuiere [:hub]]

Attached PR fixes it for the music side of bug 1200401

Comment 59

[Jim Porter (:squib)]

Comment on attachment 8660029 [details] [review]
[gaia] jimporter:music-check-playability > mozilla-b2g:master

I've confirmed that this patch works with the files Reece sent me, although I think those specific resource files would have been skipped before this patch (they're <128 bytes). In any case, this should hopefully fix the music app for people. I can write tests if you think it's important, but I think they'd need to be integration tests, since what we'd really be testing is the events that get fired from an <audio> element.

Comment 60

[Jim Porter (:squib)]

One interesting tidbit is that this is marked as a regression, which means our checkPlayability() function used to work. I guess that means something changed in Gecko to break us.

Comment 61

[GitHub Autolander Bot]

Attachment: [gaia] jimporter:music-nga-bulletproof > mozilla-b2g:master

Comment 62

[Jim Porter (:squib)]

Comment on attachment 8660111 [details] [review]
[gaia] jimporter:music-nga-bulletproof > mozilla-b2g:master

This is just the first patch on this bug, but applied to NGA.

Comment 63

[Hubert Figuiere [:hub]]

Comment on attachment 8660029 [details] [review]
[gaia] jimporter:music-check-playability > mozilla-b2g:master

works good. ship it !

Comment 64

[Hubert Figuiere [:hub]]

Comment on attachment 8660111 [details] [review]
[gaia] jimporter:music-nga-bulletproof > mozilla-b2g:master

ship it !

Comment 65

[Jim Porter (:squib)]

Landed: https://github.com/mozilla-b2g/gaia/commit/51a18f7af598db2554979c7f7422b3ec09854304
        https://github.com/mozilla-b2g/gaia/commit/2a7461a2c839280440239214552d3c2cf79b5e2d

Comment 66

[Ralph Giles (:rillian)]

Confirmed. The music scan no longer hangs on my aries device with today's gaia.

Comment 67

[Hubert Figuiere [:hub]]

Natalia, let us know if it works with the next update on the Sony device. (no idea when it will be out)

Comment 69

[Hubert Figuiere [:hub]]

Natalia directly reported that the SDcard was tested on a Flame with up to date code, and it works. So we fixed it. Excellent !

Comment 71

[qingchenchris0527]

I have some STR with about 8/10 repro.

1. Open Browser.
2. Navigate to http://www.freemusicdownloads.world/
3. Download a song.

Comment 72

[Descargar musica Gratis]

No puedo reproducir musica

Comment 73

[Tony Lawal]

Well, i do have some STR with about 8/10 repro.

1. open browser
navigate to http://mixtube.org
3. download a song

Comment 74

[Sarah Joie]

(In reply to Tony Lawal from comment #73)
> Well, i do have some STR with about 8/10 repro.
> 
> 1. open browser
> navigate to https://www.spriee.com 
> 3. download a song

First I will like to confirm that The music scan no longer hangs on my aries device with today's gaia. Thanks for your help and info.

Comment 76

[Stacy Benny]

(In reply to qingchenchris0527 from comment #71)

Well, I have some STR with about 8/10 repro.

1. Open Browser.
2. Navigate to https://smarthomeradar.com/
3. Download a song.