Closed Bug 948774 Opened 11 years ago Closed 10 years ago

B2G get_about_memory.py script doesn't work

Categories

(Firefox OS Graveyard :: General, defect)

Product:
Firefox OS Graveyard
Component:
General
Platform:
All
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(blocking-b2g:1.3+, firefox27 wontfix, firefox28 fixed, firefox29 fixed, b2g-v1.3 fixed, b2g-v1.3T fixed, b2g-v1.4 fixed)

Status:
RESOLVED FIXED
Milestone:
1.3 C2/1.4 S2(17jan)
Project Flags:
blocking-b2g 1.3+
Tracking Flags:
Tracking Status
firefox27 --- wontfix
firefox28 --- fixed
firefox29 --- fixed
b2g-v1.3 --- fixed
b2g-v1.3T --- fixed
b2g-v1.4 --- fixed

People

(Reporter: askeing, Assigned: khuey)

References

Details

(Keywords: regression, Whiteboard: [MemShrink:P1])

Attachments

(2 files)

Gecko changes
4.72 KB, patch
fabrice
: review+
Details | Diff | Splinter Review
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/306
43 bytes, text/x-github-pull-request
fabrice
: review+
Details | Review 
### ENV
Buri
Gaia      c952e2756c03eceb4de6a3eba15651741a62f9e8
Gecko     http://hg.mozilla.org/mozilla-central/rev/df82be9d89a5
BuildID   20131210040206
Version   29.0a1
ro.build.version.incremental=eng.archermind.20131114.105818

### STR
0. flash pvt build's gaia/gecko on Buri
1. reboot
2. run tools/get_about_memory.py

### ACTURAL
Console Output:
$ ./get_about_memory.py -o 
Got 4/5 files.We've waited 120s but the only relevant files we see are
  /data/local/tmp/memory-reports/memory-report-1386730452-490.json.gz
  /data/local/tmp/memory-reports/memory-report-1386730452-405.json.gz
  /data/local/tmp/memory-reports/memory-report-1386730452-441.json.gz
  /data/local/tmp/memory-reports/memory-report-1386730452-136.json.gz
We expected 5 but see only 4 files.  Giving up...
Traceback (most recent call last):
  File "./get_about_memory.py", line 289, in <module>
    get_and_show_info(args)
  File "./get_about_memory.py", line 182, in get_and_show_info
    (out_dir, merged_reports_path, dmd_files) = get_dumps(args)
  File "./get_about_memory.py", line 179, in get_dumps
    return utils.run_and_delete_dir_on_exception(do_work, out_dir)
  File "/home/askeing/workspace/hamachi/tools/include/device_utils.py", line 147, in run_and_delete_dir_on_exception
    return fun()
  File "./get_about_memory.py", line 164, in do_work
    optional_outfiles_prefixes=['dmd-'])
  File "/home/askeing/workspace/hamachi/tools/include/device_utils.py", line 228, in notify_and_pull_files
    _wait_for_remote_files(outfiles_prefixes, num_expected_files, old_files)
  File "/home/askeing/workspace/hamachi/tools/include/device_utils.py", line 310, in _wait_for_remote_files
    raise Exception("Unable to pull some files.")
Exception: Unable to pull some files.
It seems like no memory-report of PID 345 (Nuwa).
I can reproduce this issue on other Buri with 20131210040206 build.

$ adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      136   1     183440 70736 ffffffff 40026604 S /system/b2g/b2g
(Nuwa)           root      345   136   51232  19208 ffffffff 40112604 S /system/b2g/plugin-container
Communications   app_405   405   345   81400  34116 ffffffff 40112604 S /system/b2g/plugin-container
Homescreen       app_441   441   136   80300  33156 ffffffff 400f0604 S /system/b2g/plugin-container
(Preallocated a  root      490   345   60440  19836 ffffffff 40112604 S /system/b2g/plugin-container

$ adb shell ls /data/local/tmp/memory-reports
memory-report-1386730452-136.json.gz
memory-report-1386730452-405.json.gz
memory-report-1386730452-441.json.gz
memory-report-1386730452-490.json.gz
Confirmed on hamachi too since nuwa landed.
Try to dump the "num_expected_files" from include/device_utils.py
  # num_expected_files = len(outfiles_prefixes) * (1 + len(child_pids))
  outfiles_prefixes  ['memory-report-'] , len  1
  child_pids  [345, 405, 441, 490] , len  4
  num_expected_files  5
Askeing - Can you get a regression range for this? That will help confirm if this is indeed caused by nuwa's landing.
Flags: needinfo?(fyen)
Keywords: regression, regressionwindow-wanted
### Latest success build:
Buri
Gaia      1d45d1dc3201059d5c8f2efdeb92c04576d8e161
Gecko     http://hg.mozilla.org/mozilla-central/rev/9f12a9fab080
BuildID   20131209053402
Version   28.0a1
ro.build.version.incremental=eng.archermind.20131114.105818

### First failed build:
Buri
Gaia      c952e2756c03eceb4de6a3eba15651741a62f9e8
Gecko     http://hg.mozilla.org/mozilla-central/rev/df82be9d89a5
BuildID   20131210040206
Version   29.0a1
ro.build.version.incremental=eng.archermind.20131114.105818
Flags: needinfo?(fyen)
The regression range here confirms this was caused by Nuwa.
Blocks: Nuwa
Yeah, the Nuwa process is frozen, so it won't process the memory report signal.

We just need to exempt it from the iteration.
blocking-b2g: --- → 1.3+
Whiteboard: [MemShrink]
I'll look into this tomorrow.
Assignee: nobody → khuey
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #8)
> We just need to exempt it from the iteration.

As I know from QA yesterday, they planned to make something like a dashboard, which collects what we get from get_about_memory on each daily PVT build. The purpose is helping everyone to see the trend of memory usage.

If we exempt Nuwa, is it possible that we won't aware bugs like bug 921659 in early stage?
(In reply to Alan Huang [:ahuang] from comment #10)
> (In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #8)
> > We just need to exempt it from the iteration.
> 
> As I know from QA yesterday, they planned to make something like a
> dashboard, which collects what we get from get_about_memory on each daily
> PVT build. The purpose is helping everyone to see the trend of memory usage.
> 
> If we exempt Nuwa, is it possible that we won't aware bugs like bug 921659
> in early stage?

Yeah, that's a fair point.  Maybe we can try forking the Nuwa process, immediately measuring the memory usage of the new fork, and then kill that process.  I need to think about this a bit.
> Maybe we can try forking the Nuwa process,
> immediately measuring the memory usage of the new fork, and then kill that
> process.  I need to think about this a bit.

Bug 945973 will help here.
Yeah, it would.  Although ideally we'd still have visibility into the process itself.
hi Kyle, wonder if there are further update on this bug? intending to set up dashboard on datazilla requiring this to work. thanks
Flags: needinfo?(khuey)
And it looks like something else broke the about memory report even with Nuwa disabled. Adding dependency.
Depends on: 951567
Askeing, that occurs if the device ends up sleeping and losing adb connection.  Make sure that your device stays awake and has adb connection to the computer.

Having said that, I get : 
Invalid memory report(s): non-sentence other description: mem/processes/process(\init, pid=1)/other-files/init/[r-xp], /init [r-xp]

When trying to load my results in about:memory.
> Invalid memory report(s): non-sentence other description:

What Gecko version do you have?  That check has been removed as of a couple of days ago on mozilla-central.
Kyle, I think doing comment #8 is ok to unblock re-enabling nuwa if you don't have time for the more complex patch.
Yeah, I think I will land a patch that does comment 8 tomorrow.  Getting real reporting working will take more work.
Flags: needinfo?(khuey)
And I didn't manage to get that done because I couldn't get b2g to build today.  Sigh.
I'll keep working on the "real" Gecko side fix to get reporting for the Nuwa process, but we will want the B2G tools change regardless I think.
Comment on attachment 8350827 [details] [diff] [review]
Gecko changes

Review of attachment 8350827 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with nit fixed

::: dom/ipc/ContentParent.cpp
@@ +1860,5 @@
>                                        nsDependentString(aData)))
>              return NS_ERROR_NOT_AVAILABLE;
>      }
>      else if (!strcmp(aTopic, "child-memory-reporter-request")) {
> +#ifdef MOZ_IS_NUWA_PROCESS

you want MOZ_NUWA_PROCESS here
Attachment #8350827 - Flags: review?(fabrice) → review+
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #22)
> Created attachment 8350829 [details] [review]
> Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/306

I haven't tested but I'm worried that this will break get_about_memory.py on non-nuwa builds. And I don't think that we have branches for this repo.
(In reply to Fabrice Desré [:fabrice] from comment #25)
> (In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #22)
> > Created attachment 8350829 [details] [review]
> > Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/306
> 
> I haven't tested but I'm worried that this will break get_about_memory.py on
> non-nuwa builds. And I don't think that we have branches for this repo.

Yeah, it will.  Sigh.
Kyle, let's turn on the nuwa mode in device_utils.py if the MOZ_NUWA_PROCESS environment variable is set. That let us backward compatible while easy to turn on.
Whiteboard: [MemShrink] → [MemShrink][tarako]
Fabrice/Kyle, when you return from the holidays, do you mind updating this bug? it seems like this bug is close to landing. thanks
Flags: needinfo?(fabrice)
Attachment #8350829 - Flags: review?(fabrice) → review-
Flags: needinfo?(fabrice)
Whiteboard: [MemShrink][tarako] → [MemShrink:P1][tarako]
pinging to see if this is still being worked on. Thanks
Flags: needinfo?(khuey)
Thanks Joe.
Attachment #8350827 - Attachment is obsolete: true
Flags: needinfo?(khuey)
Comment on attachment 8350829 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/306

I've updated the pull request.  Now we just have an environment variable to ignore the Nuwa process (expect N-1 files).  I think we should take this.  In bug 946407 we're rewriting reporting to all go through the parent process and produce one file, and after we do this the script won't care about Nuwa at all and we can remove this hack.
Attachment #8350829 - Flags: review- → review?(fabrice)
Attachment #8350829 - Flags: review?(fabrice) → review+
Let's call this fixed.  I'll file another bug on what remains.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #33)
> Let's call this fixed.  I'll file another bug on what remains.

Well, we still need to land the gecko part that was r+ with a nit!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8350827 [details] [diff] [review]
Gecko changes

d'oh
Attachment #8350827 - Attachment is obsolete: false
You forgot to fix the MOZ_IS_NUWA_PROCESS nit...
(In reply to Fabrice Desré [:fabrice] from comment #36)
> You forgot to fix the MOZ_IS_NUWA_PROCESS nit...

scratch that, I thought you pushed it.
I fixed what I checked in.  All I did above was unobsolete the patch.

https://hg.mozilla.org/integration/b2g-inbound/rev/166e06815cfd
https://hg.mozilla.org/mozilla-central/rev/166e06815cfd
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → 1.3 C2/1.4 S2(17jan)
status-b2g-v1.3T: fixed in tarako branch, remove [tarako] in whiteboard
Whiteboard: [MemShrink:P1][tarako] → [MemShrink:P1]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: