1202310 - [TaskCluster] Failed to run xpcshell tests: zip not available

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[John Dai[:jdai]]

Please see bug 1200928 comment #1.

Comment 1

[John Dai[:jdai]]

Attachment: WIP, patch, v1

I use re-symlink busybox unzip command to avoid zip not available issue.

Comment 2

[Edgar Chen [:edgar] (OOO ~ 04/02)]

We fix an adb hang issue on emulator recently [1], let see if this bug still happen: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d3fe883a315&group_state=expanded

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1207039

Comment 3

[Edgar Chen [:edgar] (OOO ~ 04/02)]

(In reply to Edgar Chen [:edgar][:echen] from comment #2)
> We fix an adb hang issue on emulator recently [1], let see if this bug still
> happen:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=3d3fe883a315&group_state=expanded
> 
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1207039

I still saw this error, and seems only happen on emulator-x86-kk.

Comment 4

[Edgar Chen [:edgar] (OOO ~ 04/02)]

Attachment: emulator_kk_kmsg.log

Finally, I reproduced this error with emulator-kk locally. I found some clues in kernel log: `/system/` was remounted to read-only filesystem again.

<6>[   21.530000] EXT4-fs (mtdblock0): re-mounted. Opts: (null)
// adb remount

<3>[   21.860000] qtaguid: iface_stat: stat_update() lo not found
<3>[   21.860000] qtaguid: iface_stat: stat_update() lo not found
<3>[   21.860000] qtaguid: iface_stat: stat_update() lo not found
<3>[   21.860000] qtaguid: iface_stat: stat_update() lo not found
<3>[   21.860000] qtaguid: iface_stat: stat_update() lo not found
<2>[   26.550000] EXT4-fs error (device mtdblock0): ext4_valid_block_bitmap:325: comm flush-31:0: Invalid block bitmap - block_group = 1, block = 32809
<3>[   26.550000] Aborting journal on device mtdblock0-8.
<2>[   26.550000] EXT4-fs (mtdblock0): Remounting filesystem read-only
// `/system/` was remounted to read-only filesystem again.

Comment 5

[Edgar Chen [:edgar] (OOO ~ 04/02)]

I triggered a try which dumps the kernel log [1][2], I saw the exactly same error on the failed test.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a44547d4ca9e&group_state=expanded
[2] https://public-artifacts.taskcluster.net/TSZMMmfLRnWwyyk6D2olpg/0/public/test_info//emulator-5554_kmsg.log

Comment 6

[Edgar Chen [:edgar] (OOO ~ 04/02)]

Looks like something related to kernel EXT4 filesystem, there are some new ext4 patches in upstream that are not in our code base [2], I can try to cherry-pick them to see if they help.

[1] https://android.googlesource.com/kernel/goldfish/+log/android-goldfish-3.4/fs/ext4
[2] https://github.com/mozilla-b2g/kernel_goldfish/commits/b2g-goldfish-3.4/fs/ext4

Comment 7

[Edgar Chen [:edgar] (OOO ~ 04/02)]

(In reply to Edgar Chen [:edgar][:echen] from comment #6)
> Looks like something related to kernel EXT4 filesystem, there are some new
> ext4 patches in upstream that are not in our code base [2], I can try to
> cherry-pick them to see if they help.

No luck, still got this error even using kernel 3.4.67 [1].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=55862a4e1607&group_state=expanded

> 
> [1]
> https://android.googlesource.com/kernel/goldfish/+log/android-goldfish-3.4/
> fs/ext4
> [2]
> https://github.com/mozilla-b2g/kernel_goldfish/commits/b2g-goldfish-3.4/fs/
> ext4

I also tried following things, but didn't helps,
* mount /system as sync
* increase the partition size
* upgrade adb to 1.0.32
* delay installing the busybox

Comment 8

[Samael Wang [:freesamael] (away for now)]

An easier way to try this:
> adb shell  dd if=/dev/urandom of=/system/urandom bs=1048576 count=100

Interestingly if it fails it fails very quickly right after boot, while if the first run passes, I can repeat up to 100 times without seeing and error.

Comment 9

[Samael Wang [:freesamael] (away for now)]

Attachment: kernel-verbose.patch

Comment 10

[Samael Wang [:freesamael] (away for now)]

Attachment: emulator-x86-kk.log

Comment 11

[Samael Wang [:freesamael] (away for now)]

Attachment: android-x86-kk.log

Comment 12

[Samael Wang [:freesamael] (away for now)]

So I added some log in the kernel (see kernel-verbose.patch), and it turns out the problem is not a random issue. It happens on every boot (see emulator-x86-kk.log):

> [    0.241351] EXT4-fs warning (device mtdblock1): ext4_valid_block_bitmap:328: bad inode table block = 19, offset = 19
> [    0.241537] EXT4-fs warning (device mtdblock1): ext4_valid_block_bitmap:328: bad inode table block = 32787, offset = 19

In fact even stock Android has similar log. I tried AOSP's emulator / kernel with stock Android 4.4 (see android-x86-kk.log):

> [    0.450464] EXT4-fs warning (device mtdblock1): ext4_valid_block_bitmap:328: bad inode table block = 32787, offset = 19
> [    1.307986] warning: `rild' uses 32-bit capabilities (legacy support in use)
> [    2.391338] EXT4-fs warning (device mtdblock1): ext4_valid_block_bitmap:328: bad inode table block = 19, offset = 19

So if you remount system image before the error, it fails. Simple workaround: do not remount the image until the system fully boots.

Comment 13

[John Dai[:jdai]]

Attachment: [device_generic_goldfish] pull request 38

Hi Thomas, 

This problem occurs in emulator-kk and emulator-x86-kk, so we can set system partition mount to read-write by default which can avoid remounting timing issue. Would you mind giving me some feedback of this patch? Thanks.

Try link:
https://treeherder.allizom.org/#/jobs?repo=try&revision=b63130a0bcc9&exclusion_profile=false

Comment 14

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Comment on attachment 8684853 [details] [review]
[device_generic_goldfish] pull request 38

We certainly don't want this change on a phone, but for the emulator it's absolutely OK. Thanks to all of you for fixing all these emulator bugs.

Comment 15

[John Dai[:jdai]]

Comment on attachment 8684853 [details] [review]
[device_generic_goldfish] pull request 38

Hi Thomas,

Could you help to review this patch? Thanks a lot.

Comment 16

[Carsten Book [:Tomcat]]

https://github.com/mozilla-b2g/device_generic_goldfish/commit/58800ecb50e4e41cfb0a36cb43c82b73fb3612e5

Comment 18

[Pete Moore [:pmoore][:pete]]

Moving closed bugs across to new Bugzilla product "TaskCluster".