991915 - about:memory reports do not work on Tarako devices flashed via pvtbuild

Status: RESOLVED INVALID

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: aboutmem.zip

1. flash the pvtbuild for tarako
2. make sure Tarako devices are connected to the computer (adb devices)
3. run MOZ_IGNORE_NUWA_PROCESS=1 ./get_about_memory.py from a B2G git repo/tools

Expected: browser launches with the proper memory report
Actual: Invalid memory report(s): non-sentence other description

Note:
1. taipei builds seems to work for the Mac but not for linux.
2. Qanalysts found this issue.
3. I think the gonk layer for the pvtbuilds might be the issue:
about-memory-0 is from a pvtbuild (full flash via images)
about-memory-1 is from a taipei build (full flash via images)
about-memory-2 is from a taipei fullflash and pvtbuild shallow flash (gecko gaia)

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Hi Thomas and Chris, this looks like a difference between the gonk layers in two builds.  Is there anything we can do to fix the disparity between gonk layers?  the difference of the gonk seems to affect this bug and bug 991408.

Comment 2

[Chris AtLee [:catlee]]

We're just building whatever is specified by https://github.com/mozilla-b2g/b2g-manifest/blob/v1.3t/tarako.xml

is there a difference between the manifests we're using?

are there any other special binary blobs we should be including?

Comment 3

[Joe Cheng [:jcheng] (please needinfo)]

hi? Alan as well

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Apparently there's a hack that allows for b2g-info to work on the production builds that's on the taipei builds.  ( See comment 11 in bug 991408 )

Having said that the engineering builds do not show this information either in today's build.

Gaia      acd18bbd94ebfa534e252a24a75a0617e4b5d5ae
Gecko     https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/91a1b54da4a6
BuildID   20140404004001
Version   28.1
ro.build.version.incremental=eng.cltbld.20140404.045138
ro.build.date=Fri Apr  4 04:51:46 EDT 2014
Tarako

Comment 5

[Alan Huang]

As I remember, bug 874065 already fixed memory report cannot work on un-rooted device bug. Unless there's something change after that, but I don't have tarako in hand right now.

Comment 6

[thomas tsai]

Is PVT build "user build" or "userdebug" build?

Comment 7

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: memory-reports

I tried many times with today PVT user build. 
MOZ_IGNORE_NUWA_PROCESS=1 ./tools/get_about_memory.py works properly. Please see the attach report.

But I found that sometime it will wait and stop here
Pulling GC/CC logs...
Got 4/6 files.

I can ignore it with
MOZ_IGNORE_NUWA_PROCESS=1 ./tools/get_about_memory.py --no-gc-cc-log

Comment 8

[Joe Cheng [:jcheng] (please needinfo)]

triage: base on comment 7, it seems to work on latest build, minus for now

Comment 9

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

It does not work completely on the latest build.  I don't think this should be 1.3T- yet.

Comment 10

[Joe Cheng [:jcheng] (please needinfo)]

triage: 1.3T+ to make sure PVT build works

Comment 11

[Joe Cheng [:jcheng] (please needinfo)]

triage: to kli for now. thanks

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think part of it is that we need root access.  The Taipei builds already have root access. 

adb root
MOZ_IGNORE_NUWA_PROCESS=1 ./get_about_memory.py

Will allow us to have access to the about_memory report on the Mac OS.
Having said that the memory report is still broken on the linux side.
We need the linux side to work as QAnalysts do not have macs.

Comment 13

[thomas tsai]

Hi Naoki: can you confirm if this issue can be repro in your side after comment7 and comment12.

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Even with gaining root, we cannot get the gc-cc log:

ubuntu@ubuntu-VirtualBox:~/Projects/B2G_emulator/tools$ MOZ_IGNORE_NUWA_PROCESS=1 ./get_about_memory.py
Got 3/3 files.
Pulled files into about-memory-0.

I just tried to open the memory report in Firefox.  If that didn't
work for some reason, or if you want to open this report at a later
time, open the following URL in a Firefox nightly build:

  about:memory?file=/home/ubuntu/Projects/B2G_emulator/tools/about-memory-0/memory-reports

Pulling GC/CC logs...
Got 6/6 files.
Pulled files into about-memory-0.
Compressing logs...

Even when opening with: MOZ_IGNORE_NUWA_PROCESS=1 ./tools/get_about_memory.py --no-gc-cc-log

I still get :
Invalid memory report(s): non-sentence other description

on the Ubuntu machine.  I am not sure why it works on Mac but does not work on ubuntu as of yet.

Comment 15

[Kai-Zhen Li [:seinlin][:kli]]

I try today PVT build, the report is generated correctly. Firefox is not started automatically, but I can open the report by copy this URL to Firefox 29.

about:memory?file=/home/kaizhen/Project/tarako/about-memory-11/memory-reports

---
$ MOZ_IGNORE_NUWA_PROCESS=1 tools/get_about_memory.py
Got 3/3 files.
Pulled files into about-memory-11.

I just tried to open the memory report in Firefox.  If that didn't
work for some reason, or if you want to open this report at a later
time, open the following URL in a Firefox nightly build:

  about:memory?file=/home/kaizhen/Project/tarako/about-memory-11/memory-reports

Pulling GC/CC logs...
Got 6/6 files.
Pulled files into about-memory-11.
Compressing logs...

Comment 16

[thomas tsai]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #14)
> 
> on the Ubuntu machine.  I am not sure why it works on Mac but does not work
> on ubuntu as of yet.

Is MAC still the problem?

Comment 17

[thomas tsai]

Hi Naoki and Chris: Is this still a problem not solved?

Comment 18

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This is a Ubuntu problem (not a mac).
What is strange is that it works for the build on the Taipei server; not the pvtbuild for ubuntu.

The pvtbuild does work on the mac for whatever reason.

At this point I think we can "backlog" this instead of making this a shipping blocker.  We do need this working so that our team in seattle can get about:memory logs.

Comment 19

[Eric Rahm [:erahm]]

Naoki, can you provide a full log of what was output on the command line? Additionally what version of python is installed on the Ubuntu machine?

Also it's not clear to me what the actual issue is. Are the logs being pulled (a directory created, files in it), but just not auto-launched in your Ubuntu VM or are they actually corrupt and you cannot load them manually?
 
I was able to load the pvtbuild log from your original attachment (about-memory-0) in Fx30 on my local machine so I don't think the logs are actually corrupt, but it's possible that the linux version of Fx is having issues parsing the output.

RE: stalling out when retrieving cc/gc logs - I've seen this happen when a process is LMK/OOM killed while we were retrieving the report. We could probably make the script handle that better (it will timeout eventually) by checking that the process is still available.

Comment 20

[Jason Smith [:jsmith]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #18)
> This is a Ubuntu problem (not a mac).
> What is strange is that it works for the build on the Taipei server; not the
> pvtbuild for ubuntu.
> 
> The pvtbuild does work on the mac for whatever reason.
> 
> At this point I think we can "backlog" this instead of making this a
> shipping blocker.  We do need this working so that our team in seattle can
> get about:memory logs.

I don't think I agree with that rationale. Linux is a primary operating system that developers use for development. I wouldn't assume that all developers have access to a Mac, so this is still a development blocker for those don't have access to a Mac.

Comment 21

[Eric Rahm [:erahm]]

tl;dr - Have your testers update to the latest version of Firefox.

I checked things out in my Ubuntu VM and was able to reproduce nhirata's initial issue. Please note this is not an get_about_memory.py issue (the logs are generated and retrieved), not a tarako issue, and not really a Linux issue either. It's an out-of-date browser issue. 

From what I can tell Firefox 24 is shipped with the stock Ubuntu 13.10 release, this version is unable to parse our current memory logs. After doing a software update today in my VM I got updated to Firefox 29 and was able to open the log.

Comment 22

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Ok.  That's embarassingly lame of me.  Thanks for taking the time to look at this.

I've updated and I am seeing the same thing.

Comment 23

[Nicholas Nethercote [inactive]]

> Expected: browser launches with the proper memory report
> Actual: Invalid memory report(s): non-sentence other description

Please CC me if you have any more bugs like that where memory report files fail to load. Thanks!