835214 - Firefox OS cannot start up (black screen): failed to open AUTO_VOLUME_CONTROL

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Gene Lian (I already quit Mozilla)]

After building the flashing the m-c with changeset: 120017:63685729b771. The OS cannot start up (black screen)! Looking at the logcat, it keeps showing me the following message:

E/AudioHardwareMSM76XXA(  439): failed to open AUTO_VOLUME_CONTROL /system/etc/AutoVolumeControl.txt: No such file or directory (2)
E/QualcommCamera(  439): Qint android::get_number_of_cameras(): E
I/QualcommCameraHardware(  439): getCameraInfo: IN
I/QualcommCameraHardware(  439): getCameraInfo: loading libqcamera at 0xb000ee28
V/QualcommCameraHardware(  439):  Storing the current target type as 3 
I/QualcommCameraHardware(  439): getCameraInfo: numOfCameras = 1
I/QualcommCameraHardware(  439): Camera sensor 0 info:
I/QualcommCameraHardware(  439): camera_id: 0
I/QualcommCameraHardware(  439): modes_supported: 5
I/QualcommCameraHardware(  439): position: 0
I/QualcommCameraHardware(  439): sensor_mount_angle: 90
V/QualcommCameraHardware(  439): getCameraInfo: dlclose(libqcamera)
I/QualcommCameraHardware(  439): getCameraInfo: OUT
W/AudioFlinger(  439): Thread AudioOut_1 cannot connect to the power manager service
W/AudioHardwareMSM76XXA(  439): rpc_snd_set_device(6, 1, 1)

Comment 1

[Dave Hylands [:dhylands]]

Try doing the following:

adb shell
stop b2g
/system/bin/b2g.sh

I've started seeing the following sometimes under the emulator. I don't know if this is related to you problem or not:

root@android:/ # /system/bin/b2g.sh
XPCOMGlueLoad error for file /system/b2g/libxpcom.so:
Cannot load library: link_image[1936]:   189 could not load needed library 'libxul.so' for 'libxpcom.so' (load_segments[915]: 189 failed to map segment from 'libxul.so' @ 0x406e9000 (0x00000000). p_vaddr=0x00000000 p_offset=0x00070b70)
Couldn't load XPCOM.

Comment 2

[Gene Lian (I already quit Mozilla)]

(In reply to Dave Hylands [:dhylands] from comment #1)
> Try doing the following:
> 
> adb shell
> stop b2g
> /system/bin/b2g.sh
> 
> I've started seeing the following sometimes under the emulator. I don't know
> if this is related to you problem or not:
> 
> root@android:/ # /system/bin/b2g.sh
> XPCOMGlueLoad error for file /system/b2g/libxpcom.so:
> Cannot load library: link_image[1936]:   189 could not load needed library
> 'libxul.so' for 'libxpcom.so' (load_segments[915]: 189 failed to map segment
> from 'libxul.so' @ 0x406e9000 (0x00000000). p_vaddr=0x00000000
> p_offset=0x00070b70)
> Couldn't load XPCOM.

Exactly. The symptom is the same.

Comment 3

[Dave Hylands [:dhylands]]

I tried adding:

ac_add_options --disable-elf-hack

to gonk-misc/default-gecko-config, and I haven't seen the problem since, although this might be a red herring as I've only rebuilt a couple of times since as well.

Prior to that, doing a clobber build helps. I'm pretty sure I've the problem appear and go away with no code changes, just a rebuild.

Comment 4

[Gene Lian (I already quit Mozilla)]

CC'ing some gonk members. Looks like some drivers are missing in the system.

Comment 5

[Vincent Liu[:vliu]]

I can still see below error messages in a successful boot up.

E/AudioHardwareMSM76XXA(  118): failed to open AUTO_VOLUME_CONTROL /system/etc/AutoVolumeControl.txt: No such file or directory (2)

W/AudioFlinger(  118): Thread AudioOut_1 cannot connect to the power manager service

I don't think they are factors to block the OS booting up.

Comment 6

[Jonathan Griffin (:jgriffin)]

I've seen this problem as well, and confirm that it can appear and disappear with no code changes, just rebuilds.

Comment 7

[Dave Hylands [:dhylands]]

I saw it again tonight.

I've captured my borked objdir-gecko and out directories (all I did was rename them and rebuild without absolutely no other changes) to get good ones (so I'm going to save the good ones too).

Comment 8

[Dave Hylands [:dhylands]]

I took a look at the readelf -S output of the two libxul.so files found in the out/target/product/unagi/system/b2g drectory.  (so these were both processed by elfhack)

Good:
  [25] .bss              NOBITS          01406ef0 11e7ef0 02f02c 00  WA  0   0 16
  [26] .comment          PROGBITS        00000000 11e7ef0 000012 01  MS  0   0  1
  [27] .note.gnu.gold-ve NOTE            00000000 11e7f04 000018 00      0   0  4
  [28] .ARM.attributes   ARM_ATTRIBUTES  00000000 11e7f1c 000032 00      0   0  1
  [29] .shstrtab         STRTAB          00000000 11e7f4e 000166 00      0   0  1

Borked:
  [25] .bss              NOBITS          01406ef0 11e7eec 02f02c 00  WA  0   0 16
  [26] .comment          PROGBITS        00000000 11e7eec 000012 01  MS  0   0  1
  [27] .note.gnu.gold-ve NOTE            00000000 11e7f00 000018 00      0   0  4
  [28] .ARM.attributes   ARM_ATTRIBUTES  00000000 11e7f18 000032 00      0   0  1
  [29] .shstrtab         STRTAB          00000000 11e7f4a 000144 00      0   0  1

I find it interesting that the broked one has 16-byte alignment, but the Offset is not a multiple of 16.

I then compared the libraries from objdir-gecko/toolkit/library/libxul.so and the only differences were in the size of one of the .debug_line sections

Good:
  [23] .bss              NOBITS          01406ef0 1406ef0 02f02c 00  WA  0   0 16
  [24] .debug_abbrev     PROGBITS        00000000 1406eec 8b7701 00      0   0  1
  [25] .debug_info       PROGBITS        00000000 1cbe5ed 177527ad 00      0   0  1
  [26] .debug_line       PROGBITS        00000000 19410d9a 18af1ef 00      0   0  1
  [27] .debug_pubnames   PROGBITS        00000000 1acbff89 75aeb1 00      0   0  1
  [28] .debug_str        PROGBITS        00000000 1b41ae3a 1820fce 01  MS  0   0  1
  [29] .comment          PROGBITS        00000000 1cc3be08 000012 01  MS  0   0  1
  [30] .debug_frame      PROGBITS        00000000 1cc3be1c 4bead4 00      0   0  4

Borked:
  [23] .bss              NOBITS          01406ef0 1406ef0 02f02c 00  WA  0   0 16
  [24] .debug_abbrev     PROGBITS        00000000 1406eec 8b7701 00      0   0  1
  [25] .debug_info       PROGBITS        00000000 1cbe5ed 177527ad 00      0   0  1
  [26] .debug_line       PROGBITS        00000000 19410d9a 18af1f0 00      0   0  1
  [27] .debug_pubnames   PROGBITS        00000000 1acbff8a 75aeb1 00      0   0  1
  [28] .debug_str        PROGBITS        00000000 1b41ae3b 1820fce 01  MS  0   0  1
  [29] .comment          PROGBITS        00000000 1cc3be09 000012 01  MS  0   0  1
  [30] .debug_frame      PROGBITS        00000000 1cc3be1c 4bead4 00      0   0  4

Differences in section [26]. 27-29 got shifted by a byte due to this difference.

It seems odd to me that the .bss section in both source libraries are the same, but the bad one got modified.

Comment 9

[Mike Hommey [:glandium]]

The dynamic linker never looks at ELF sections, and .bss offset shouldn't matter to any tool that looks at them. What does readelf -l look like for these files?

Comment 10

[Dave Hylands [:dhylands]]

readelf -l reports:

Good:
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00000034 0x00000034 0x00140 0x00140 R   0x4
  LOAD           0x000000 0x00000000 0x00000000 0x71530 0x71530 R E 0x1000
  LOAD           0x072000 0x00072000 0x00072000 0x21e000 0x21e000     0
  LOAD           0x071a44 0x00290a44 0x00290a44 0xfb4afc 0xfb4afc R E 0x1000
  LOAD           0x1027000 0x01246000 0x01246000 0x1c0eec 0x1eff1c RW  0x1000

Borked:
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00000034 0x00000034 0x00140 0x00140 R   0x4
  LOAD           0x000000 0x00000000 0x00000000 0x71530 0x71530 R E 0x1000
  LOAD           0x071530 0x00000000 0x00072000 0x00000 0x00000     0
  LOAD           0x071a44 0x00290a44 0x00290a44 0xfb4afc 0xfb4afc R E 0x1000
  LOAD           0x1027000 0x01246000 0x01246000 0x1c0eec 0x1eff1c RW  0x1000

with this line:
  LOAD           0x071530 0x00000000 0x00072000 0x00000 0x00000     0

being the only line that was different between the two.

Comment 11

[Dave Hylands [:dhylands]]

If I push the borked libxul.so to the phone with the good image on it and try to start b2g I see the following error:

root@android:/ # /system/bin/b2g.sh
XPCOMGlueLoad error for file /system/b2g/libxpcom.so:
Cannot load library: link_image[1936]:   481 could not load needed library 'libxul.so' for 'libxpcom.so' (load_segments[915]: 481 failed to map segment from 'libxul.so' @ 0x406ff000 (0x00000000). p_vaddr=0x00000000 p_offset=0x00071530)
Couldn't load XPCOM.

Comment 12

[Mike Hommey [:glandium]]

So, this looks like a bug with the code added in bug 822584. Could you attach both the good and borked libxul.so *before* they are elfhacked? (you can strip them to make them smaller)

Comment 13

[Dave Hylands [:dhylands]]

The libxul.so's were too big to add as attachments.

So I put them here:
http://people.mozilla.org/~dhylands/bug-835214/libxul-stripped-good.so
http://people.mozilla.org/~dhylands/bug-835214/libxul-stripped-borked.so

These were taken from the objdir-gecko/toolkit/libary directory, so these are the pre-elfhacked versions.

If I push libxul-stripped-borked.so to the phone, it boots up fine.

Comment 14

[Dave Hylands [:dhylands]]

For completeness, here are the libxul.so files from objdir-gecko/dist/b2g (I think that these are the packaged versions)

http://people.mozilla.org/~dhylands/bug-835214/libxul-dist-good.so
http://people.mozilla.org/~dhylands/bug-835214/libxul-dist-borked.so

If I push the libxul-dist-borked.so to the phone, then I get an error similar to the one in comment 11.

Comment 15

[Mike Hommey [:glandium]]

This is a mozilla-central problem only, isn't it?

Comment 16

[Gene Lian (I already quit Mozilla)]

(In reply to Mike Hommey [:glandium] from comment #15)
> This is a mozilla-central problem only, isn't it?

Yes. The b2g18 looks fine.

Comment 17

[Mike Hommey [:glandium]]

This is a regression due to bug 834228.

Comment 18

[Dave Hylands [:dhylands]]

(In reply to Mike Hommey [:glandium] from comment #15)
> This is a mozilla-central problem only, isn't it?

I was seeing it on m-i, so yes

Comment 19

[Mike Hommey [:glandium]]

Workaround until a fix is landed: rm -rf objdir-gecko/dist/b2g before building.

Comment 20

[Mike Hommey [:glandium]]

Attachment: Avoid re-stripping executables already stripped.

Also rework UnifiedExecutableFile so that it leaves stripping to ExecutableFile.

Comment 21

[Dave Hylands [:dhylands]]

Looking back, the times that I saw the problem was when I only modified JS files and rebuilt (which is in agreement with what I'm seeing in your patch).

Comment 22

[Mike Hommey [:glandium]]

FWIW, the core problem is strip breaks elfhacked binaries.

Comment 23

[Mike Hommey [:glandium]]

https://hg.mozilla.org/mozilla-central/rev/ba515e203813

(rebased on top of bug 836218)

Comment 24

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Only on MC, not on 1.0.1.  Not sure about v1train/1.1 atm.

since it breaks elfhacked binaries... it sounds pretty major.  Should we ask to uplift this down to 1.0.0?

Comment 25

[Mike Hommey [:glandium]]

The new packager has never been backported, so the problem that is due to the new packager is limited to m-c and now aurora. If the problem occurs on other branches, then it will need an ad-hoc fix.